Skip to content

Fix bug in wan self forcing xfuser #498

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bubbliiiing wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix bug in wan self forcing xfuser #498

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ea85c96
Fix bug in wan self forcing xfuser
bubbliiiing Jun 11, 2026
c7291b0
Updare Readme and fix get_scheduler bug in distill
bubbliiiing Jun 17, 2026
a250947
Updare Readme and training code
bubbliiiing Jun 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```

**Absolute Path**:
Expand All @@ -147,7 +147,7 @@ If your data uses absolute paths, configure in the training script:

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
Expand Down Expand Up @@ -223,8 +223,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 512 |
| `--video_sample_size` | Maximum video training resolution | 512 |
Expand Down
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN_CONTROL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
If your data uses relative paths, configure in the training script:

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```

**Absolute Path**:
Expand All @@ -155,7 +155,7 @@ If your data uses absolute paths, configure in the training script:

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
Expand Down Expand Up @@ -231,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-Control` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 4 |
| `--image_sample_size` | Maximum image training resolution | 512 |
| `--video_sample_size` | Maximum video training resolution | 512 |
Expand Down
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN_CONTROL_zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
如果数据的路径为相对路径，则在训练脚本中设置：

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```

**绝对路径**：
Expand All @@ -155,7 +155,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
Expand Down Expand Up @@ -231,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-Control` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 4 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
Expand Down
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN_LORA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```

**Absolute Path**:
Expand All @@ -147,7 +147,7 @@ If your data uses absolute paths, configure in the training script:

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, absolute paths are recommended.
Expand Down Expand Up @@ -225,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size | 1 |
| `--image_sample_size` | Maximum training resolution for images | 512 |
| `--video_sample_size` | Maximum training resolution for videos | 512 |
Expand Down
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN_LORA_zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```

**绝对路径**：
Expand All @@ -147,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
Expand Down Expand Up @@ -225,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
Expand Down
10 changes: 5 additions & 5 deletions scripts/cogvideox_fun/README_TRAIN_zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```

**绝对路径**：
Expand All @@ -147,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
Expand Down Expand Up @@ -223,8 +223,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
Expand Down
5 changes: 4 additions & 1 deletion scripts/cogvideox_fun/train_reward_lora.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,10 @@
from accelerate.logging import get_logger
from accelerate.state import AcceleratorState
from accelerate.utils import ProjectConfiguration, set_seed
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
from diffusers import CogVideoXDPMScheduler, DDIMScheduler
from diffusers.optimization import get_scheduler
from diffusers.utils import check_min_version, deprecate, is_wandb_available
Expand Down
18 changes: 9 additions & 9 deletions scripts/ernie_image/README_TRAIN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```

**Absolute Paths**:
Expand All @@ -139,7 +139,7 @@ If your data uses absolute paths, configure the training script as follows:

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
Expand Down Expand Up @@ -208,8 +208,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/ERNIE-Image` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
Expand Down Expand Up @@ -267,8 +267,8 @@ You can configure validation parameters to periodically generate test images dur

```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
Expand Down Expand Up @@ -306,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR

```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
Expand Down
18 changes: 9 additions & 9 deletions scripts/ernie_image/README_TRAIN_zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```

**绝对路径**：
Expand All @@ -139,7 +139,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```

> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
Expand Down Expand Up @@ -208,8 +208,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/ERNIE-Image` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率,代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
Expand Down Expand Up @@ -267,8 +267,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con

```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
Expand Down Expand Up @@ -306,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR

```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
Expand Down
10 changes: 5 additions & 5 deletions scripts/fantasytalking/README_TRAIN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ The `metadata.json` for FantasyTalking is slightly different from the normal JSO
If your data uses relative paths, configure the training script as follows:

```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```

**Absolute Paths**:
Expand All @@ -159,7 +159,7 @@ If your data uses absolute paths, configure the training script as follows:

```bash
export DATASET_NAME=""
export DATASET_META_NAME="/path/to/your/metadata.json"
export DATASET_META_NAME="/path/to/your/metadata_add_width_height.json"
```

> 💡 **Tip**: If your dataset is small and stored locally, use relative paths. If your dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
Expand Down Expand Up @@ -249,8 +249,8 @@ Here is a detailed explanation of the key parameters in the training script:
| `config_path` | Model configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-I2V-14B-720P` |
| `pretrained_audio_model_name_or_path` | Audio encoder path | `None` (automatically uses $MODEL_NAME/audio_encoder) |
| `train_data_dir` | Training dataset directory | `datasets/internal_datasets/` |
| `train_data_meta` | Training dataset metadata file | `datasets/internal_datasets/metadata.json` |
| `train_data_dir` | Training dataset directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `train_data_meta` | Training dataset metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `video_sample_size` | Video sample size (maximum resolution) | `512` |
| `token_sample_size` | Token sample size | `512` |
| `video_sample_stride` | Video sample stride | `1` |
Expand Down
Loading