RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild

This repository is the official model checkpoint of RealisDance-DiT. RealisDance-DiT is a structurally simple, empirically robust, and experimentally strong baseline model for controllable character animation in the wild.

Gallery

Here are several character animation generated by RealisDance-DiT. Note that the GIFs shown here have some degree of visual quality degradation. Please visit our project page for more original videos

Quick Start

1. Setup Repository and Environment

git clone https://github.com/theFoxofSky/RealisDance.git
cd RealisDance

conda create -n realisdance python=3.10
conda activate realisdance

pip install -r requirements.txt

# FA3 (Optional)
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout ea3ecea97a1393c092863330aff9a162bb5ce443  # very important, using other FA3 will yield bad results
cd hopper
python setup.py install

2. Quick Inference

• Inference with Demo sequences

python inference.py \
    --ref __assets__/demo/ref.png \
    --smpl __assets__/demo/smpl.mp4 \
    --hamer __assets__/demo/hamer.mp4 \
    --prompt "A blonde girl is doing somersaults on the grass. Behind the grass is a river, \
    and behind the river are trees and mountains. The girl is wearing black yoga pants and a black sports vest." \
    --save-dir ./output

Disclaimer

This project is released for academic use. We disclaim responsibility for user-generated content.

Contact Us

Jingkai Zhou: [email protected]

BibTeX

@article{zhou2025realisdance-dit,
  title={RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wild},
  author={Zhou, Jingkai and Wu, Yifan and Li, Shikai and Wei, Min and Fan, Chao and Chen, Weihua and Jiang, Wei and Wang, Fan},
  journal={arXiv preprint arXiv:2504.14977},
  year={2025}
}

@article{zhou2024realisdance,
  title={RealisDance: Equip controllable character animation with realistic hands},
  author={Zhou, Jingkai and Wang, Benzhi and Chen, Weihua and Bai, Jingqi and Li, Dongyang and Zhang, Aixi and Xu, Hao and Yang, Mingyang and Wang, Fan},
  journal={arXiv preprint arXiv:2409.06202},
  year={2024}
}

Acknowledgements

Thanks Shikai Li for condition paraperation and Chenjie Cao for pose align.