When using for image-to-video , if prompt optimization or calling the official video effects does not meet your custom requirements for specific actions, effects, or styles , use model fine-tuning .
Applicable scope
- Applicable region: The features described in this document are available only in the Singapore region. You must use an API key created in this region.
- Fine-tuning method supported: Efficient SFT-LoRA fine-tuning.
-
Models supported for fine-tuning:
- Image-to-video (first-frame-based): wan2.7-i2v, wan2.6-i2v, wan2.5-i2v-preview, wan2.2-i2v-flash.
- Image-to-video (first-and-last-frame-based): wan2.2-kf2v-flash.
How to fine-tune a model
- Image-to-video (first-frame-based)
- Image-to-video (first-and-last-frame-based)
Fine-tuning goal: Train a LoRA model for the money rain effect.Expected result: Input a first-frame image. No prompt is needed. The model generates a video with the money rain effect.
Input first-frame image | Output video (before fine-tuning)You cannot generate a consistent money rain effect every time using prompts. Motion is not controllable. | Output video (after fine-tuning)The fine-tuned model reproduces the specific money rain effect from the training dataset without prompts. |
Step 1: Upload your dataset
Upload your local dataset (in .zip format) to Alibaba Cloud Model Studio. Retrieve the file ID (id).
Sample training dataset: See the training set for format details.
- Image-to-video (first-frame-based): wan-i2v-training-dataset.zip.
- Image-to-video (first-and-last-frame-based): wan-kf2v-training-dataset.zip.
This example uses the first-frame-based image-to-video model. Only upload the training set. The system automatically splits part of it as the validation set.Uploading a dataset takes several minutes. Exact time depends on file size.
id. This is the unique identifier for your uploaded dataset.
Step 2: Fine-tune the model
Step 2.1 Create a fine-tuning job
Start training using the file ID from Step 1. Request example Replace<replace with training dataset file ID> fully with the id from the previous step.
| Field | Type | Required | Description | Recommended value |
|---|---|---|---|---|
| batch_size | int | Yes | Batch size. The number of data samples sent to the model for training at once.This parameter is a per-instance configuration. We recommend using the default value for each model. Do not adjust unless necessary.
The actual number of instances running for a training job is determined by platform scheduling. The Global Step output in training logs may differ from estimated results, but this does not change the total amount of training data or affect the final model performance. | Model-dependent |
| n_epochs | int | Yes | Training epochs. steps = n_epochs x ceiling(dataset size / batch_size). Total steps should be >= 800.Example: dataset has 5 samples, batch_size=4, steps per epoch = ceiling(5/4) = 2, minimum n_epochs = 800/2 = 400. The recommended number of training epochs adjusts automatically based on data volume. Less data requires more epochs to learn sufficiently; more data means each epoch contains more samples, so fewer epochs are needed. 50 epochs is mainly suitable for very small datasets of about 2 samples; when data volume reaches 50-60 videos, training approximately 3000-5000 steps is typically recommended. | 50 |
| learning_rate | float | Yes | Learning rate. Controls the magnitude of model weight updates. Too high may degrade the model; too low may result in negligible changes. | 2e-5 |
| eval_epochs | int | Yes | Evaluation interval. Value must be >= n_epochs/10. Specifies how many epochs between each evaluation and checkpoint save. | 20 |
| max_pixels | int | Yes | Maximum resolution for training videos (total pixels = width x height). The system only scales videos that exceed this value.
| Model-dependent |
| split | float | No | Training set split ratio. Value range: (0,1). Only takes effect when validation_datasets is not specified. | 0.9 |
| max_split_val_dataset_sample | int | No | Maximum number of validation samples from auto-split. Validation count = min(total x (1 - split), this value). | 5 |
| save_total_limit | int | No | Maximum number of checkpoints to keep. The system only retains the last N checkpoints. | 10 |
| lora_rank | int | No | LoRA low-rank matrix dimension. Value must be 2n (16/32/64). | 32 |
| lora_alpha | int | No | LoRA weight scaling coefficient. Value must be 2n (16/32/64). | 32 |
output:
job_id: Job ID. Use it to check progress.finetuned_output: Name of the new fine-tuned model. Use this name for deployment.status: Training status. After creating the job, the initial status is PENDING. Training has not started yet.
Step 2.2 Check the fine-tuning job status
Use thejob_id from Step 2.1 to check progress. Poll this endpoint until status becomes SUCCEEDED.
This fine-tuning job takes several hours. Exact time depends on the model. Wait patiently.
<replace with fine-tuning job job_id> in the URL fully with the value of job_id.
output field:
status: When this value becomes SUCCEEDED, training is complete. You can deploy the model.usage: Total tokens used for training. Used for billing.
Step 3: Deploy the fine-tuned model
Step 3.1 Deploy the model as an online service
After the fine-tuning job status is SUCCEEDED, deploy the model as an online service. Request example Replace<replace with model name model_name> fully with the value of finetuned_output from creating the fine-tuning job.
output:
deployed_model: Name of the deployed model. Use it to check deployment status and call the model.status: Deployment status. After deploying the fine-tuned model, the initial status is PENDING. Deployment has not started yet.
Step 3.2 Check deployment status
Poll this endpoint untilstatus becomes RUNNING.
Deploying this fine-tuned model takes about 5–10 minutes.
<replace with deployed_model> fully with the value of deployed_model from Step 3.1.
Response example
Check these two parameters in the output field:
status: When the status becomes RUNNING, the model is deployed successfully. You can start calling it.deployed_model: Name of the deployed model.
Step 4: Call the model to generate videos
After successful model deployment (when the deployment status is RUNNING ), you can start calling the model.
Step 4.1: Create a video generation task and get the task_id
Step 4.1: Create a video generation task and get the task_id
Request exampleReplace
<replace with deployed_model name> with the deployed_model value output from the previous step.- Image-to-video (first-frame-based)
- Image-to-video (first-and-last-frame-based)
Expected result: Input a first-frame image. No prompt is needed. The model automatically generates a video with the "money rain effect" based on the image.
Response exampleCopy and save the Input parameter description
Wan2.7 model call
Wan2.7 model call
Wan2.6/Wan2.5/Wan2.2 model call
Wan2.6/Wan2.5/Wan2.2 model call
task_id for querying results in the next step.When calling a fine-tuned LoRA model, Wan2.7 input parameter usage is mostly the same as Wan2.7 - image-to-video. For Wan2.6/Wan2.5/Wan2.2, refer to Wan - image-to-video - first frame API reference (2.1-2.6).The table below lists only LoRA-specific parameter usage or restrictions. For general parameters not listed here (such as
duration), refer to the API documentation.Field | Type | Required | Description | Example value |
|---|---|---|---|---|
model | string | Yes | Model name. You must use a fine-tuned model that is successfully deployed and has a RUNNING status. | xxxx-ft-202511111122-xxxx |
input.prompt | string | No | Text prompt. Whether this parameter takes effect depends on the aigc_config.use_input_prompt setting:
| - |
parameters.resolution | string | No | Resolution of the generated video. wan2.2 and wan2.5 models: 480P, 720P. Default is 720P. wan2.6 models: 720P, 1080P. Default is 720P. wan2.7 models: 720P, 1080P. Default is 1080P. | 720P |
parameters.prompt_extend | boolean | No | Enable prompt rewriting. When calling a fine-tuned LoRA model, disable this. Set it to false. | false |
Step 4.2: Query results using the task_id
Step 4.2: Query results using the task_id
Poll the task status using the Response example
task_id until task_status becomes SUCCEEDED. Then get the video URL.Request example
Replace 86ecf553-d340-4e21-xxxxxxxxx with the real task_id.
The video URL is valid for 24 hours. Download the video promptly.
Build a custom dataset
In addition to using the sample datasets in this guide, you can build your own dataset for fine-tuning.
Your dataset must include a training set (required) and may include a validation set (optional, supports automatic splitting from the training set). Package all files into a .zip file. Use only English letters, numbers, underscores, or hyphens in filenames.
Dataset format
Training set: Required
- Image-to-video (first-frame-based)
- Image-to-video (first-and-last-frame-based)
The training set includes first-frame images, training videos, and an annotation file (data.jsonl).
- Sample training set: wan-i2v-training-dataset.zip.
- ZIP directory structure:
- Annotation file (data.jsonl): Each line is one training sample. It must be a JSON object. Structure:
Validation set: Optional
- Image-to-video (first-frame-based)
- Image-to-video (first-and-last-frame-based)
The validation set includes first-frame images and an annotation file (data.jsonl). Videos are not required. At each evaluation point, the training job automatically calls the model service to generate preview videos using the validation images and prompts.
- Sample validation set: wan-i2v-valid-dataset.zip.
- ZIP directory structure:
- Annotation file (data.jsonl): Each line is one validation sample. It must be a JSON object. Structure:
Data scale and limits
- Data volume: Provide at least 10 samples. More training data yields better results. We recommend 20–100 samples for stable performance.
- ZIP archive: When uploading via API, total size ≤ 1 GB.
-
Training image requirements:
- Formats: BMP, JPEG, PNG, WEBP.
- Resolution ≤ 4096×4096.
- No hard limit on individual file size (system pre-processes automatically).
-
Training video requirements:
- Formats: MP4, MOV.
- Resolution ≤ 4096×4096.
- No hard limit on individual file size (system pre-processes automatically).
- Duration per video: wan2.2 models 2–5 seconds recommended; wan2.5 models 2–10 seconds recommended; wan2.6 and wan2.7 models 2–10 seconds recommended.
Data collection and cleaning
1. Define the fine-tuning scenario
Wanxiang supports fine-tuning for image-to-video in these scenarios:- Fixed video effects: Teach the model a specific visual change, such as a carousel or magic costume change.
- Fixed character actions: Improve the model’s ability to reproduce specific body movements, such as dance moves or martial arts techniques.
- Fixed camera motion: Replicate complex cinematography, such as push, pull, pan, tilt, or orbit shots.
2. Collect raw assets
- AI generation and filtering: Use the Wanxiang foundation model to batch-generate videos. Manually select high-quality samples that best match your target effect. This is the most common method.
- Real-world filming: If you need high realism for interactive scenes (e.g., hugging, shaking hands), use real footage.
- 3D software rendering: For effects or abstract animations requiring precise control, use 3D software (e.g., Blender, Cinema 4D).
3. Clean the data
Dimension | Positive Requirements | Negative example |
|---|---|---|
Consistency | Core features must be highly uniform. Example: For training “360-degree rotation”, all videos must rotate clockwise at roughly the same speed. | Mixed directions. The dataset contains both clockwise and counterclockwise rotations. The model does not know which direction to learn. |
Diversity | More variety in subjects and scenes is better. Cover different subjects (men, women, children, animals, buildings) and compositions (close-ups, wide shots, high angles, low angles). Also vary resolution and aspect ratio. | Single scene or subject. All videos show “a person in red clothes rotating in front of a white wall”. The model may mistakenly treat “red clothes” and “white wall” as part of the effect. It fails when clothes change. |
Balance | Data types should be balanced. If multiple styles exist, their counts should be roughly equal. | Severe imbalance. 90% are portrait videos, 10% are landscape videos. The model may perform poorly on landscape videos. |
Purity | Clean and clear visuals. Use original assets without interference. | Interfering elements. Videos contain subtitles, logos, watermarks, obvious black bars, or noise. The model may learn the watermark as part of the effect. |
Duration | Asset duration ≤ target duration. If you want 5-second videos, crop assets to 4–5 seconds. | Too long assets. You want 5-second videos but feed the model 8-second assets. This leads to incomplete action learning and a choppy feel. |
Video annotation: Write prompts for videos
In the dataset annotation file (data.jsonl), each video has a corresponding prompt. The prompt describes the video content. Prompt quality directly determines what the model learns.
| Prompt exampleThe video begins with a young woman standing in front of a brick wall covered with ivy. She has long, smooth reddish-brown hair, wearing a white sleeveless dress, a shiny silver necklace, and a smile on her face. The background is a brick wall covered with green vines, appearing rustic and natural. Then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green patterns) pour down like a torrential rain, densely hitting and surrounding her. The bills continue to fall, she stretches her arms upward, neck slightly tilted back, expression surprised, completely immersed in this wild money rain. |
Prompt writing formula
Prompt = [Subject description] + [Background description] + [Trigger word] + [Motion description]
Prompt element | Description | Guidance | Example |
|---|---|---|---|
Subject description | Describe people or objects present in the frame | Required | The video begins with a young woman... |
Background description | Describe the environment where the subject is located | Required | The background is a brick wall covered with green vines... |
Trigger word | A rare, meaningless word | Recommended | s86b5p or m01aa |
Motion description | Describe the dynamic changes during the effect in detail | Recommended | Countless huge-sized US dollar bills (beige background/dark green patterns) pour down like a torrential rain... |
About trigger words
About trigger words
- What is a trigger word? It acts as a "visual anchor". Many complex motions (e.g., a unique dance path or light effect) are hard to describe in text. A trigger word forces the model to generate that exact visual effect when it sees the word.
- Why use it? Fine-tuning builds a mapping between text and video features. The trigger word binds a hard-to-describe effect to a unique word so the model can lock onto the target.
-
If we have a trigger word, why describe motion in detail?
They serve different roles and work better together.
- Motion description explains what happens in the scene. It tells the model basic physical actions and logic. Motion descriptions are usually consistent across samples.
- Trigger word explains what the action specifically looks like. It represents unique changes and features that words cannot describe.
How to write good prompts
Follow consistency rules for effect descriptions
For all samples containing the same effect, keep the motion description part as consistent as possible. Apply this rule to both training and validation sets.-
Purpose: When the model sees
s86b5pfollowed by the same fixed description and always sees money rain, it learns that s86b5p = money rain visual effect. -
Example: Whether the subject is a “young woman” or a “man in a suit”, the prompt ends the same way for the money rain effect: “...then the s86b5p money rain effect begins, countless US dollar bills pour down like a torrential rain...”
Sample type
Prompt content (Note consistency in underlined parts)
Training sample 1
The video begins with a young woman standing in front of a brick wall... (environment description omitted)...
Then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green patterns) pour down like a torrential rain, densely hitting and surrounding her. The bills continue to fall, she stretches her arms upward, neck slightly tilted back, expression surprised, completely immersed in this wild money rain.Training sample 2
The video begins with a man in a suit in a high-end restaurant... (environment description omitted)...
Then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green patterns) pour down like a torrential rain, densely hitting and surrounding him. The bills continue to fall, he stretches his arms upward, neck slightly tilted back, expression surprised, completely immersed in this wild money rain.Validation sample 1
The video begins with a young child standing in front of a cityscape... (environment description omitted)...
Then the s86b5p money rain effect begins, countless huge-sized US dollar bills (beige background/dark green patterns) pour down like a torrential rain, densely hitting and surrounding him. The bills continue to fall while the camera slowly zooms in, he stretches his arms upward, neck slightly tilted back, expression surprised, completely immersed in this wild money rain.
Use AI to generate prompts
To get high-quality prompts, use multimodal large language models such as Qwen-VL to assist.-
Use AI to generate initial descriptions
-
Free-form brainstorming (find inspiration): If you’re unsure how to describe the effect, let AI explore freely.
- Send “
Describe the video content in detail” and observe the model’s output. - Focus on terms the model uses for motion trajectory (e.g., “pour down like a torrential rain”, “camera slowly zooms in”). These terms can be used for later optimization.
- Send “
-
Fixed-format prompting (standardize output): Once you have a rough idea, design a fixed format to guide AI to generate prompts that match your template.
Example code
See visual understanding for API call details.
-
Free-form brainstorming (find inspiration): If you’re unsure how to describe the effect, let AI explore freely.
-
Extract effect templates
- Run the process multiple times on samples with the same effect. Identify high-frequency, accurate phrases used to describe the effect. Extract a generic “effect description”.
- Copy and paste this standardized effect description into all samples for that effect.
- Keep unique “subject” and “background” descriptions for each sample. Replace only the “effect description” part with the unified template.
- Manual review AI may hallucinate or make detection errors. As a final step, perform a manual check. For example, confirm that the descriptions of the entity and background match the actual scene.
Evaluate the model using a validation set
Specify a validation set
A fine-tuning job requires a training set. A validation set is optional. You can choose to let the system split automatically or upload manually. Specify as follows:
Method 1: No validation set uploaded (system splits automatically)
When you create a fine-tuning job, if you do not upload a separate validation set, meaning thevalidation_file_ids parameter is not passed, the system automatically splits a portion of the training set to use as the validation set based on the following two hyperparameters:
split: Proportion of training data to split. For example, 0.9 means 90% for training and 10% for validation.max_split_val_dataset_sample: Maximum number of samples for the auto-split validation set.
dataset size × (1 - split) and max_split_val_dataset_sample.
-
Example: Suppose you upload only a training set with 100 samples, split=0.9 (10% for validation), and max_split_val_dataset_sample=5.
- Theoretical split: 100 × 10% = 10 samples.
- Actual split: min(10, 5)=5. So the system uses only 5 samples for the validation set.
Method 2: Upload a validation set manually (using validation_file_ids)
If you prefer to evaluate checkpoints using your own prepared data instead of relying on random system splits, upload a custom validation set. Note: Once you choose manual upload, the system ignores the automatic split rules above and validates only with your uploaded data.Steps: Upload a validation set manually
Steps: Upload a validation set manually
- Prepare the validation set: Package validation data into a standalone
.zipfile. See validation set format. - Upload the validation set: Call the upload dataset API to upload this
.zipfile. Get a dedicated file ID. - Specify the validation set when creating the job: When calling the create fine-tuning job API, put this file ID in the
validation_file_idsparameter.
Select the best checkpoint for deployment
During training, the system saves periodic “snapshots” (checkpoints). By default, the system outputs the final checkpoint as the fine-tuned model. But intermediate checkpoints may outperform the final version. You can pick the best one for deployment.
The system runs checkpoints on the validation set and generates preview videos at intervals defined by the hyperparameter eval_epochs.
- Evaluation: Judge by watching the preview videos directly.
- Selection criteria: Pick the checkpoint with the best visual quality and no motion distortion.
Procedure
Step 1: View preview results for checkpoints
Step 1: View preview results for checkpoints
Step 1.1 List validated checkpoints
This API returns only checkpoints that passed validation and successfully generated preview videos. Failed checkpoints are not listed.Request example<replace_with_fine-tuning_job_id>: Replace fully with thejob_idfrom the create fine-tuning job API.
Step 1.2 View validation results for a checkpoint
Select one checkpoint from the list above (e.g., “checkpoint-160”) and view its video result.Request example<replace_with_fine-tuning_job_id>: Replace fully with thejob_idfrom creating the fine-tuning job.<replace_with_selected_checkpoint>: Replace fully with the checkpoint value, e.g., “checkpoint-160”.
video_path. It is valid for 24 hours. Download and review promptly. Repeat this step to compare multiple checkpoints and find the best one.Step 2: Export the checkpoint and get the model name for deployment
Step 2: Export the checkpoint and get the model name for deployment
Step 2.1 Export the model
Assume “checkpoint-160” gives the best result. Now export it.Request example<replace_with_fine_tuning_job_id>: Replace fully with thejob_idfrom creating the fine-tuning job.<replace_with_checkpoint_to_export>: Replace fully with the checkpoint value, e.g., “checkpoint-160”.<replace_with_exported_model_name_for_console_display>: Replace fully with a custom model name for console display, e.g., “wan2.5-checkpoint-160”. This name must be globally unique. Duplicate names are not allowed. For parameter guidance, see export checkpoint.
output=true means the export request was created successfully.Step 2.2 Query the new model name after deployment
Query all checkpoint statuses to confirm export completion and get the unique model name (model_name) for deployment.Request example<replace_with_fine-tuning_job_id>: Replace fully with thejob_idfrom creating the fine-tuning job.
status becomes SUCCEEDED, export succeeded. The model_name field is the new model name for deployment and calls.Step 3: Deploy and call the model
Step 3: Deploy and call the model
After successfully exporting the checkpoint and getting the
model_name, follow these steps:- Model deployment: In the input parameter
model_name, enter the specific value obtained after export. - Model call: Follow the API documentation to call the deployed model.
Go live
In production, if the initial trained model performs poorly (e.g., distorted visuals, weak effects, inaccurate motion), tune along these dimensions:
1. Check data and prompts
- Data consistency: Consistency is critical. Check for “bad samples” with opposite directions or vastly different styles.
- Sample count: Increase high-quality data to 20+ samples.
- Prompt: Ensure trigger words are meaningless, rare terms (e.g., s86b5p). Avoid common words (e.g., running) that cause interference.
-
n_epochs (training epochs)
- Default: 50. Use the default unless necessary. If adjusting, follow the rule: “Total training steps ≥ 800”.
-
Total steps formula:
steps = n_epochs × ceiling (training set size / batch_size). -
n_epochs minimum formula:
n_epochs = 800 / ceiling (dataset size / batch_size). -
Example: Assume a training set of 5 samples, using the Wan2.5 model (batch_size=4).
- Steps per epoch: 5 / 4 = 1.25, rounded up to 2. Total epochs: n_epochs = 800 / 2 = 400. This is the recommended minimum. You can increase it as needed.
-
batch_size (batch size)
- Batch size. The number of data samples sent to the model for training at once.
- This parameter is a per-instance configuration. We recommend using the default value for each model. Do not adjust unless necessary.
- wan2.7-i2v: Recommended value: 1.
- wan2.5-i2v-preview: Recommended value: 4.
- wan2.2-i2v-flash: Recommended value: 4.
- wan2.2-kf2v-flash: Recommended value: 4.
The actual number of instances running for a training job is determined by platform scheduling. The Global Step output in training logs may differ from estimated results, but this does not change the total amount of training data or affect the final model performance.
- learning_rate (learning rate): Use the default value (2e-5). Typically does not require modification.
Billing
-
Model training: Charged.
- Cost = Total training tokens × Unit price. See model training billing.
- After training, check the total tokens consumed in the
usagefield of the query fine-tuning job status API.
- Model deployment: Free.
-
Model calls: Charged.
- Billed according to the standard call price of the fine-tuned foundation model. See model pricing.
API reference
Video and image generation model fine-tuning API
FAQ
Q: How are training and validation set data volumes calculated?
A: The training set is required, the validation set is optional. Calculation methods are as follows:
-
When no validation set is provided: The uploaded training set is the "total dataset size". The system automatically splits part of the data for validation.
- Validation set size =
min(total dataset size × (1 − split), max_split_val_dataset_sample). For an example, see specify a validation set. - Training set size =
total dataset size − validation set size.
- Validation set size =
-
When a validation set is uploaded manually: The system no longer splits the validation set from the training data.
- Training set size = Uploaded training set data volume.
- Validation set size = Uploaded validation set data volume.
Q: How do I design a good trigger word?
A: Follow these rules:
- Use meaningless letter combinations, such as sksstyle, a8z2_bbb.
- Avoid common English words (e.g., beautiful, fire, dance). This prevents polluting the model's original understanding of these words.