Skip to content

Commit df8a4cd

Browse files
hackwaanurag
andauthored
update readme (#43)
Signed-off-by: anurag <anurag@amd.com> Co-authored-by: anurag <anurag@amd.com>
1 parent a697ff5 commit df8a4cd

1 file changed

Lines changed: 79 additions & 19 deletions

File tree

‎README.md‎

Lines changed: 79 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,25 +8,26 @@ The tool has to be run inside the debug target.  
88

99
#### Feature Summary
1010

11-
Device Support: Phx, Stx, Telluride, npu3
11+
Device Support: Phx, Stx, Telluride (and its t20/t10 variants)
1212

1313
##### Standalone mode
1414
1. AIE Status, UC Status
15-
2. Use AIE Breakpoints, Single Step at overlay level
15+
2. AIE Breakpoints, Single Step at overlay level
1616
3. Peano design lst dump
17-
4. Function calltree and function list for all reloadables
17+
4. Calltrees and functions in reloadables
1818
5. Read/Write registers, memory
1919

2020
##### VAIML Batch Debug
2121
1. L1, L2, L3 Dump for each Layer in design for mismatch analysis
22-
2. Multi stamp, batch support, X2 support (TG support upcoming)
22+
2. Multi stamp, batch, X2 and Templated Graph (TG) support
2323
3. Status at Hang and at each layer
2424
4. Error Detection
2525

2626
##### VAIML Interactive Debug
2727
1. GDB Like Interface to step through and inspect layers
28-
2. Design Inspection
29-
3. Advanced mode with all Standalone functionality and per layer inspection
28+
2. Design Inspection, including a layer report that needs no hardware
29+
3. Subfunction call tree for TG layer kernels
30+
4. Advanced mode with all Standalone functionality and per layer inspection
3031

3132
##### XRT CoreDump Analysis
3233
All read features of Standalone mode like status, read_registers
@@ -69,7 +70,7 @@ sudo modprobe amdxdna timeout_in_sec=0
6970

7071
Status Dump provides snapshot of AIE Array State in case of a hang.
7172
MLDebugger's Standalone mode (-s) is used for this purpose
72-
Supported devices : "phx", "stx", "telluride"
73+
Supported devices : "phx", "stx", "telluride", "t20", "t10"
7374
Default overlay is 4x4
7475

7576
1. Start Application and wait for it to hang. Additionally, ENABLE_ML_DEBUG can be used to keep the hw_ctx alive
@@ -101,8 +102,8 @@ This feature can find source code lines where controller is stuck in case of a h
101102
1. Add initial Halt to VAIML Application. Find out location of initial halt elfs:
102103
```
103104
python3 -c "import mldebug;from pathlib import Path; print(Path(mldebug.__file__).parent)"
104-
# Under this directory, look for "initial_halt_elfs/telluride" directory.
105-
NOTE: For git users, the halts can be found here: MLDebug/ext/initial_halt_elfs/telluride
105+
# Under this directory, look for "bin/initial_halt_elfs/telluride" directory.
106+
NOTE: For git users, the halts can be found here: MLDebug/src/mldebug/bin/initial_halt_elfs/telluride
106107
```
107108
2. The initial halt elfs are of format <stamps>x<row>x<col>. Copy the elf that matches overlay to the design directory (for example: 4x4, will use 1x4x4)
108109
3. Add following content to xrt.ini:
@@ -135,10 +136,55 @@ mldebug -v <top level vaiml_design_folder>
135136
mldebug  -b .\resnet18\buffer_info.json -a .\resnet18\Work -l dumps -o 4x4
136137
```
137138

139+
### Layer Report
140+
141+
Lists every layer the debugger can step through.
142+
143+
```
144+
# Write to a file
145+
mldebug -v ./resnet18 --dump_layers layers.txt
146+
# Same report from inside interactive mode
147+
> layers layers.txt
148+
149+
ID #ITERS #STAMPS #MLADF LAYER_NAME
150+
|-- KERNEL
151+
--------------------------------------------------------------
152+
1 1 1 1 /Gather
153+
|-- mllib_graphs::concat_adf_wrapper<bfloat16>
154+
```
155+
156+
### Inspecting a TG Layer's Kernel
157+
158+
A TG layer's kernel fuses several ops behind one wrapper, so the debugger can
159+
only break on it as a whole. The interactive "i"/info() command breaks that
160+
wrapper down into its subfunctions, with the PCs to break on:
161+
162+
```
163+
> i
164+
Current Layer: 58, Current Iteration: 1, #Iterations: 7
165+
Stopped at Start of Kernel(s):
166+
Stamp 0: elf: 8, kernels: mllib_graphs::add_layernorm_adf_wrapper<bfloat16>, ...
167+
168+
FUNCTION START_PC END_PC LOCK_REL SIZE STACK
169+
-----------------------------------------------------------------------------------------
170+
mllib_graphs::add_layernorm_adf_wrapper 0x001210 0x0013ba 0x00138e 458 64
171+
|-- setup_add_layernorm_1pass_params 0x000c10 0x000c90 - 148 0
172+
|-- add_layernorm_1pass 0x000cb0 0x0011d0 - 1362 0
173+
```
174+
175+
SIZE and STACK come from the Chess map file, so they read as "-" on Peano
176+
designs, whose call tree is recovered from the disassembly instead.
177+
138178
### User Options
139179

140180
```
141-
usage: mldebug [-h] [-b <file>] [-a <file>] [-d DEV] [-o <cxr>] [-i] [-l <dir>] [-v <path>] [-s] [-l3] [-f [<flag1>, <flag2> ...]]
181+
usage: mldebug [-h] [-b <file>] [-a <file>] [-c COREDUMP_FILE]
182+
[--dump-aie-status <output_file_name>]
183+
[--dump_layers [<output_file_name>]]
184+
[-d {phx,stx,telluride,npu3,t20,t10}] [-o <cxr>] [-i]
185+
[-l <dir>] [-v <path>] [-x2 <path>] [-s]
186+
[--load_script <filename>] [-l3] [--peano]
187+
[-f [<flag1> <flag2> ...]]
142188
143189
AIE Debug for VAIML Requirements:
144190
1. Active Hardware Context (xrt-smi in PATH)
@@ -155,22 +201,37 @@ options:
155201
Use -d flag to specify device.
156202
--dump-aie-status <output_file_name>
157203
Write AIE status to a file and exit.
158-
-d DEV, --device DEV AIE device [phx,stx,telluride]. Default: telluride
204+
--dump_layers [<output_file_name>]
205+
Write a text report of the design's layers and exit.
206+
Needs no hardware.
207+
-d {phx,stx,telluride,npu3,t20,t10}, --device {phx,stx,telluride,npu3,t20,t10}
208+
Specify device if it can't be detected from aie_dir.
159209
-o <cxr>, --overlay <cxr>
160210
Overlay used by design. Default: 4x4
161211
-i, --interactive Launch in Interactive Mode. Default: Batch
162212
-l <dir>, --output_dir <dir>
163-
Directory to store memory and status dumps. Default: layer_dump
213+
Directory to store memory and status dumps.
214+
Default : layer_dump
164215
-v <path>, --vaiml_folder_path <path>
165216
Specify the VAIML top level folder path.
217+
This overrides aie_dir and buffer_info.
218+
-x2 <path>, --x2_folder_path <path>
219+
Specify the X2 top level folder path.
220+
This overrides aie_dir and buffer_info.
166221
-s, --aie_only Standalone AIE debug. Work dir can be optionally specified.
222+
--load_script <filename>
223+
Execute a Python script in the advanced shell namespace.
167224
-l3, --l3 Dumps L3 buffers during the execution
168-
-f [<flag1>, <flag2> ...], --run_flags [<flag1>, <flag2> ...]
169-
Specify one or more runtime flags for batch mode:
225+
--peano Enable support for peano.
226+
With -v flag, peano support is autodetected.
227+
-f [<flag1> <flag2> ...], --run_flags [<flag1> <flag2> ...]
228+
Specify one or more runtime flags:
170229
skip_dump : Do not dump memory
171230
l2_ifm_dump : Dump only L2 IFM buffers
231+
text_dump : Dump in text format
172232
skip_iter : Skip iterations in batch mode when possible
173-
dump_temps : Write intermediate (.lst) files to disk
233+
skip_iter2 : skip_iter using lcp lock.(Telluride only)
234+
multistamp : Enable N Stamp/Batch mode
174235
```
175236

176237
### FAQ
@@ -179,7 +240,7 @@ options:
179240
1. Any AIE cores in ERROR_HALT
180241
2. All AIE cores in LOCK_STALL
181242
3. Any AIE cores in Reset
182-
4. DM_ADDRESS_OUT_OF_RANGE_CORE is present in events
243+
4. DM_ADDRESS_OUT_OF_RANGE_CORE is present in event list - points to bad memory access in kernel
183244

184245
#### Mismatch Scenarios
185246

@@ -188,9 +249,8 @@ options:
188249
2. Error bits in SR1/SR2 registers are set
189250

190251
#### FAQ
191-
1. Hang in conv2d_maxpool: Is caused due to an aiecompiler optimization that cause race condition sometimes. Fix is to rerun aiecompiler with "--multi-layer-lcp-acquire-lock" switch
192-
2. No L3 dumps for TG layers. This is due to L3 placements and work in progress.
193-
3. FlexmlRT doesn't halt or env variables can't be set. Use correct command for the shell:
252+
1. Debugger Hang in conv2d_maxpool: Is caused due to an aiecompiler optimization that cause race condition sometimes. Fix is to rerun aiecompiler with "--multi-layer-lcp-acquire-lock" switch
253+
2. FlexmlRT doesn't halt or env variables can't be set. Use correct command for the shell:
194254
```
195255
# Powershell
196256
$Env:ENABLE_ML_DEBUG=1

0 commit comments

Comments
 (0)