mix chord算法复现问题

[23557-l]

您好。我最近用您的框架复现qwen2.5-math-7b-base在luffy数据集的性能。
yaml如下：

project: "mix_chord"
name: "mix_chord_math_qwen2.5-math_luffy"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,/r-contentsecurity/share/datas_yl/yanlong/trinity_ckpt}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  kl_loss_fn_args:
    kl_coef: 0.00
  sample_strategy_args:
    expert_data_ratio: 0.111
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.5 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.1 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.28
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 576 # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 8
    ngpus_trainer: 8
    train_batch_size_expert: 64
    train_batch_size_usual: 512 # 32 batchsize * 8 repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/r-contentsecurity/share/datas_yl/yanlong/checkpoints/Qwen2.5-Math-7B}
  max_response_tokens: 3072
  max_model_len: 4096
cluster:
  node_num: 1
  gpu_per_node: 16
buffer:
  total_epochs: 1
  batch_size: 64
  train_batch_size: 576
  explorer_input:
    taskset:
      name: math_aligned
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /mnt/nas/yanlong/data-important/ContinualRL/datasets/data/chord/luffy/rl}
      split: 'train'
      format:
        prompt_key: 'question'
        response_key: 'answer'
        system_prompt: "Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: \"<think>\n {thoughts} </think>\n\". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After \"</think>\n,\" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \\boxed{} for closed-form results like multiple choices or mathematical solutions."
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
    eval_tasksets:
    - name: AIME2024
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /mnt/nas/yanlong/datasets/aime_2024}  # e.g. path to AIME2024
      split: 'test'
      repeat_times: 8
      format:
        prompt_key: 'Problem'
        response_key: 'Answer'
        system_prompt: "Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: \"<think>\n {thoughts} </think>\n\". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After \"</think>\n,\" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \\boxed{} for closed-form results like multiple choices or mathematical solutions."
      rollout_args:
        temperature: 1.0
        top_p: 0.7
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:////mnt/nas/yanlong/data-important/cache/mix_chord_math_qwen2.5-math.db'
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 1
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH, /mnt/nas/yanlong/data-important/ContinualRL/datasets/data/chord/luffy/sft}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 10
  runner_per_model: 8
  rollout_model:
    engine_num: 8
    tensor_parallel_size: 1
    enable_prefix_caching: true
    enforce_eager: false
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 25600
        ulysses_sequence_parallel_size: 2
        optim:
          lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size}
monitor:
  monitor_type: wandb

但是感觉效果不是很理想。training reward和aime eval accuracy如下：

请问是哪里的设置有问题吗

[23557-l]

感觉一般用qwen2.5-math-base训完aime2024都能到30分以上吧

[23557-l]

我用qwen2.5-7b-instruct也训了一次，配置跟qwen2.5-math-base一样。训练reward和aime24 accuracy如下：

[23557-l]

不知道是不是batch size和learning rate还得再调一调？

[garyzhang99]

对应的两个曲线看上去都是直接训崩了，看了一眼config怀疑最可能导致训练崩掉的是response length的问题？3072作为max token设置可能有些小，而SFT数据集里面就应该已经有超过4000token的样本了吧，这个地方可能会导致一些BUG。
这边能设置ppo_max_token_len_per_gpu: 25600都没有OOM，说明response length应该开到20K都是没问题的。

& 可以试一下mu直接设置为0直接进行pure RL会不会训崩，看看是哪里的问题。

[garyzhang99]

另外额外可以提及的一些点：

1. 当时CHORD的设计实验主要是在Instruct模型上面做的，是想解决Instruct模型上用不同Pattern的Response去SFT会导致发生偏移和Performance Shift的问题。
   对应的方法在Base模型上其实不一定是最优的（如果是在Base模型上训练的话，给low prob token更高的权重会效果更好），可以对权重函数再做调整。

2. 以及MATH领域不同于Tooluse这种，MATH是一个进行SFT就实际上会很强的领域，如果SFT数据的数量和质量足够，光SFT就能有特别好的Performance了（比如Openthougths SFT数据集就能把qwen2.5-7b-instruct的AIME24打到70）。LUFFY的SFT数据集应该对应有20K个SFT样本，其实用这个数量的SFT样本在Instruct模型上多调一调SFT就已经足够把模型的Pattern完全掰过来：我们之前尝试用20K样本在qwen2.5-7b-instruct上训练（1e-5 lr, bs 64, 5 epoch）能有接近25的Accuracy，然后后续加RL也能明显超过30，其实这个performance会明显强于以RL为主的方法。
   如果你这边是在尝试SFT/RL结合的相关research，也可以去试一试MATH以外不同Domain～

[23557-l]

谢谢回复。我用qwen2.5-7b-instruct训练配置是：

project: "mix_chord"
name: "mix_chord_math_qwen2-5_luffy"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,/r-contentsecurity/share/datas_yl/yanlong/trinity_ckpt}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  kl_loss_fn_args:
    kl_coef: 0.00
  sample_strategy_args:
    expert_data_ratio: 0.111
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.5 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.1 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.28
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 576 # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 8
    ngpus_trainer: 8
    train_batch_size_expert: 64
    train_batch_size_usual: 512 # 32 batchsize * 8 repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/r-contentsecurity/share/checkpoints/opensources/Qwen2.5-7B-Instruct}
  max_response_tokens: 10240
  max_model_len: 11264
cluster:
  node_num: 1
  gpu_per_node: 16
buffer:
  total_epochs: 1
  batch_size: 64
  train_batch_size: 576
  explorer_input:
    taskset:
      name: math_aligned
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /mnt/nas/yanlong/data-important/ContinualRL/datasets/data/chord/luffy/rl}
      split: 'train'
      format:
        prompt_key: 'question'
        response_key: 'answer'
        system_prompt: "Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: \"<think>\n {thoughts} </think>\n\". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After \"</think>\n,\" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \\boxed{} for closed-form results like multiple choices or mathematical solutions."
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
    eval_tasksets:
    - name: AIME2024
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /mnt/nas/yanlong/datasets/aime_2024}  # e.g. path to AIME2024
      split: 'test'
      repeat_times: 8
      format:
        prompt_key: 'Problem'
        response_key: 'Answer'
        system_prompt: "Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: \"<think>\n {thoughts} </think>\n\". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After \"</think>\n,\" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \\boxed{} for closed-form results like multiple choices or mathematical solutions."
      rollout_args:
        temperature: 1.0
        top_p: 0.7
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:////mnt/nas/yanlong/data-important/cache/mix_chord_math_qwen2.5.db'
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 1
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH, /mnt/nas/yanlong/data-important/ContinualRL/datasets/data/chord/luffy/sft}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 10
  runner_per_model: 8
  rollout_model:
    engine_num: 8
    tensor_parallel_size: 1
    enable_prefix_caching: true
    enforce_eager: false
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 25600
        ulysses_sequence_parallel_size: 2
        optim:
          lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size}
monitor:
  monitor_type: wandb

max token设置比较大。但也出现训崩的情况

[garyzhang99]

好的，那有没有response length或者grad norm的曲线可以看看
或者pure RL看有没有提升，会不会训崩？

[23557-l]

qwen2.5-7b-instruct的：

[23557-l]

pure rl我可以也试下

[23557-l]

另外，

model:
  max_response_tokens: 10240
  max_model_len: 10752

请问下这里面的max_response_tokens影响sft数据吗？看上去只影响rl rollout？

[garyzhang99]

另外，

model:
  max_response_tokens: 10240
  max_model_len: 10752

请问下这里面的max_response_tokens影响sft数据吗？看上去只影响rl rollout？

这个是model的参数，对SFT和RL都会影响，我们SFT数据和RL都是转换成experience的形式训练。

&现在max_prompt_len是自动根据max_model_len - max_response_tokens去计算。为了方式题目被一半就截断可以把max_model_len 和 max_response_tokens的差值也设置大一些，比如到2K。

[23557-l]

另外额外可以提及的一些点：

1. 当时CHORD的设计实验主要是在Instruct模型上面做的，是想解决Instruct模型上用不同Pattern的Response去SFT会导致发生偏移和Performance Shift的问题。
   对应的方法在Base模型上其实不一定是最优的（如果是在Base模型上训练的话，给low prob token更高的权重会效果更好），可以对权重函数再做调整。
2. 以及MATH领域不同于Tooluse这种，MATH是一个进行SFT就实际上会很强的领域，如果SFT数据的数量和质量足够，光SFT就能有特别好的Performance了（比如Openthougths SFT数据集就能把qwen2.5-7b-instruct的AIME24打到70）。LUFFY的SFT数据集应该对应有20K个SFT样本，其实用这个数量的SFT样本在Instruct模型上多调一调SFT就已经足够把模型的Pattern完全掰过来：我们之前尝试用20K样本在qwen2.5-7b-instruct上训练（1e-5 lr, bs 64, 5 epoch）能有接近25的Accuracy，然后后续加RL也能明显超过30，其实这个performance会明显强于以RL为主的方法。
   如果你这边是在尝试SFT/RL结合的相关research，也可以去试一试MATH以外不同Domain～

我打算先按照您这边说的试下纯用luffy数据去sft

mode: train
project: "sft"
name: "sft_qwen2_5"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,/r-contentsecurity/share/datas_yl/yanlong/trinity_ckpt}
algorithm:
  algorithm_type: sft
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/r-contentsecurity/share/checkpoints/opensources/Qwen2.5-7B-Instruct}
  max_response_tokens: 20480
  max_model_len: 21054
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 5
  train_batch_size: 64
  trainer_input:
    experience_buffer:
      name: sft_data
      storage_type: file
      path: ${oc.env:TRINITY_SFT_DATASET_PATH,/mnt/nas/yanlong/data-important/ContinualRL/datasets/data/chord/luffy/sft}
      split: 'train'
      format:
        prompt_type: messages
        messages_key: 'messages'
  explorer_input:
    eval_tasksets:
    - name: AIME2024
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /mnt/nas/yanlong/datasets/aime_2024}  # e.g. path to AIME2024
      split: 'test'
      repeat_times: 8
      format:
        prompt_key: 'Problem'
        response_key: 'Answer'
        system_prompt: "Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: \"<think>\n {thoughts} </think>\n\". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After \"</think>\n,\" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \\boxed{} for closed-form results like multiple choices or mathematical solutions."
      rollout_args:
        temperature: 1.0
        top_p: 0.7
synchronizer:
  sync_method: 'checkpoint'
  sync_interval: 10
explorer:
  eval_interval: 100
trainer:
  save_interval: 100
  trainer_config:
    actor_rollout_ref:
      model:
        use_remove_padding: true
      actor:
        use_dynamic_bsz: true
        ppo_max_token_len_per_gpu: 22000
        ulysses_sequence_parallel_size: 1
        optim:
          lr: 1e-5
      ref:
        log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
        log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
        ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size}

请您可以看下这个sft配置是否ok吗

[23557-l]

然后我用的trinity-rft应该是九月底发布的版本。0.3.1.dev0. 不知道是否会因为版本较老而受影响？

[garyzhang99]

请您可以看下这个sft配置是否ok吗

看上去OK的

九月底发布的版本。0.3.1.dev0.

如果可以，最好更新下版本。之前的版本后面确实也发现并且修了几个BUG🤦

[23557-l]

谢谢。请问可以加您的微信吗？如果不方便在github评论区发您的微信号的话，请问可以把您的微信号发到我的邮箱后我来添加您吗？我的邮箱是 2149849060@qq.com