[BUG] Timeout when running task

[Alital]

(Explorer pid=599, ip=10.0.25.9) ERROR 11-22 15:37:46 [scheduler.py:93] Timeout when running task of batch 8 at runner 159 at attempt 1: Task(workflow=<class 'trinity.common.workflows.customized_math_workflows.MathBoxedWorkflow'>, repeat_times=16, format_args=FormatConfig(prompt_type=<PromptType.MESSAGES: 'messages'>, prompt_key='problem', response_key='answer', system_prompt_key=None, system_prompt=None, messages_key='message', tools_key='tools', image_key=None, video_key=None, reply_prefix=None, workflow_key='', reward_fn_key='', chosen_key='chosen', rejected_key='rejected', enable_concatenated_multi_turn=False, chat_template=None), rollout_args=GenerationConfig(temperature=1.0, top_p=1.0, top_k=-1, logprobs=0, max_tokens=16384, n=1), workflow_args={'with_think': True}, reward_fn_args={}, is_eval=False, reward_fn=None, raw_task={'problem': 'For any positive integer $n$, define a function $f$ by \[f(n)=2n+1-2^{\lfloor\log_2n\rfloor+1}.\] Let $f^m$ denote the function $f$ applied $m$ times. Determine the number of integers $n$ between $1$ and $65535$ inclusive such that $f^n(n)=f^{2015}(2015)."', 'answer': '8008'}, batch_id=8, task_id=85, index={'index': 981, 'taskset_id': 0})
(Explorer pid=599, ip=10.0.25.9) ERROR 11-22 15:37:46 [scheduler.py:93] Timeout when running task of batch 8 at runner 177 at attempt 1: Task(workflow=<class 'trinity.common.workflows.customized_math_workflows.MathBoxedWorkflow'>, repeat_times=16, format_args=FormatConfig(prompt_type=<PromptType.MESSAGES: 'messages'>, prompt_key='problem', response_key='answer', system_prompt_key=None, system_prompt=None, messages_key='message', tools_key='tools', image_key=None, video_key=None, reply_prefix=None, workflow_key='', reward_fn_key='', chosen_key='chosen', rejected_key='rejected', enable_concatenated_multi_turn=False, chat_template=None), rollout_args=GenerationConfig(temperature=1.0, top_p=1.0, top_k=-1, logprobs=0, max_tokens=16384, n=1), workflow_args={'with_think': True}, reward_fn_args={}, is_eval=False, reward_fn=None, raw_task={'problem': '设数列 $\left\{x_{n}\right\}$ 满足\n$x_{1}=1, x_{n+1}=4 x_{n}+\left[\sqrt{11} x_{n}\right]$ 。\n求 $x_{2012}$ 的个位数字.', 'answer': '3'}, batch_id=8, task_id=103, index={'index': 999, 'taskset_id': 0})

[Alital]

每次训练到第8个batch就会报错类似错误，然后训练就终止了，请帮忙看看这个错误是什么原因导致的

[pan-x-c]

上述报错是因为 Workflow 运行超时导致的。可以检查 yaml 配置文件中的 explorer.max_timeout，该值决定了每个 workflow 的最大等待时间（默认为 1800 s），如果一个 workflow 运行耗时超过该时间就会出现上述报错。

另外报错中提到使用的 workflow 为 MathBoxedWorkflow，该 workflow 出现超时有两种可能

1. 训练不稳定导致模型输出了超长内容，使得 workflow 运行过慢发生超时
2. 模型输出了乱码内容，导致 reward 函数解析超时

建议可以用 debug 模式 利用上述报错信息中提到的那条数据来调试检查定位原因

[Alital]

好的，感谢回复~~~