python3 ~/sim/astra-sim/extern/graph_frontend/param/train/compute/python/build/lib/param_bench/train/compute/python/tools/trace_link.py --et-file ./pytorch_et0.json --kineto-file ./kineto_trace_0.json --exact-match --annotation 'enumerate(DataLoader)#_MultiProcessingDataLoaderIter.__next__'
[2024-03-19 08:44:21,007] execution_trace.py:441 [INFO]: Iteration node ids list = [1, 2216, 24992, 43225, 61321, 79426, 97551, 115473, 133356, 151330, 169160, 187095, 205044, 222820, 240719, 258762, 276644, 294685, 312664, 330426, 348528, 366397, 384192, 402211, 420042, 438014, 455971, 473796, 491710, 509663, 527590, 545569, 563452, 581313, 599257, 617204, 635038, 653018, 670987, 688774, 706819, 724711, 742619, 760647, 778522, 796362, 814376, 832289, 850186, 868155, 886031, 903970, 922059, 939831, 957709, 975604, 993383, 1011351, 1029385, 1047274, 1065277, 1083188, 1101020, 1119059, 1137034, 1154930, 1172953, 1190866, 1208781, 1226717, 1244599, 1262451, 1280351, 1298218, 1316039, 1333961, 1351727, 1369657, 1387584, 1405377, 1423294, 1441239, 1459051, 1476974, 1494877, 1512689, 1530617, 1548550, 1566508, 1584505, 1602407, 1620294, 1638240, 1656171, 1674091, 1692104, 1710069, 1727971, 1745854, 1763599, 1781537, 1799552, 1817433, 1835357, 1853315, 1871241, 1889234, 1907100, 1924916, 1942829, 1960750, 1978610, 1996572, 2014462, 2032349, 2050281, 2068129, 2086085, 2104043, 2122001, 2139913, 2157763, 2175708, 2193640, 2211586, 2229573, 2247456, 2265399, 2283326, 2301269, 2319315, 2337145, 2355062, 2372970, 2390800, 2408827, 2426699, 2444776, 2462690, 2480552, 2498594, 2516523, 2534325, 2552341, 2570246, 2588170, 2606230, 2623963, 2641960, 2660003, 2677732, 2695721, 2713769, 2731570, 2749459, 2767443, 2785239, 2803315, 2821242, 2839064, 2857060, 2875030, 2892845, 2910959, 2928860, 2946651, 2964657, 2982412, 3000358, 3018399, 3036264, 3054200, 3072144, 3089919, 3107858, 3125706, 3143568, 3161515, 3179388, 3197226, 3215159, 3233033, 3250869, 3268808, 3286787, 3304756, 3322784, 3340609, 3358568, 3376624, 3394464, 3412407, 3430452, 3448347, 3466274, 3484100, 3501964, 3519975, 3537923, 3555846, 3573816, 3591713, 3609611, 3627567, 3645477, 3663444, 3681484, 3699271, 3717188, 3735100, 3752888, 3770905, 3788951, 3806776, 3824750, 3842662, 3860501, 3878434, 3896335, 3914229, 3932164, 3950077, 3968058, 3985971, 4003828, 4021652, 4039554, 4057421, 4075268, 4093196, 4111000, 4128993, 4147054, 4164893, 4182943, 4201012, 4218959, 4236977, 4254862, 4272845, 4290882, 4308716, 4326641, 4344631, 4362496, 4380530, 4398424, 4416329, 4434292, 4452184, 4470078, 4487992, 4505864, 4523657, 4541664, 4559545, 4577465]
[2024-03-19 08:44:21,008] trace_link.py:294 [INFO]: Execution trace has 256 > 1 iterations.
[2024-03-19 08:44:21,008] execution_trace.py:682 [INFO]: Copying nodes for iter 2 for ids in the range [24992, 43225)
Traceback (most recent call last):
File "/Users/sim/astra-sim/extern/graph_frontend/param/train/compute/python/build/lib/param_bench/train/compute/python/tools/trace_link.py", line 895, in <module>
main() # pragma: no cover
File "/Users/sim/astra-sim/extern/graph_frontend/param/train/compute/python/build/lib/param_bench/train/compute/python/tools/trace_link.py", line 868, in main
) = trace_analysis(args.et_file, args.kineto_file, args.annotation)
File "/Users/sim/astra-sim/extern/graph_frontend/param/train/compute/python/build/lib/param_bench/train/compute/python/tools/trace_link.py", line 297, in trace_analysis
et_ = et.clone_one_iteration(trim_iter)
File "/opt/homebrew/anaconda3/envs/astra-sim-new/lib/python3.8/site-packages/parambench_train_compute-1.0.0+git.1710448259-py3.8.egg/param_bench/train/compute/python/tools/execution_trace.py", line 704, in clone_one_iteration
assert len(thread_nodes) > 0
AssertionError
Hello,
I get the following error:
could someone please tell me what is going on?