How do you train a LoRA?

Beginners
1

I’ve started creating AI models, but I’m currently stuck at the LoRA training stage; my videos are turning out blurry and unrealistic. I’d like to know how the pros handle this and what mistakes I might be making, as I haven’t been able to find any solid information in open sources.

2

Hmm… I’m not very familiar with T2V/I2V, but it seems there are quite a few different ways blur can show up here…:

I cannot really speak for how “the pros” do it, but I would not start by assuming that blurry output means you picked the wrong LoRA rank or learning rate.

With video LoRAs, “blurry and unrealistic” can come from several different layers of the pipeline: the base model, inference settings, the LoRA itself, the checkpoint you selected, video preprocessing/frame sampling, the VAE, or something specific to the model/trainer/runtime.

So the fastest route is usually to find the first boundary where the quality becomes bad, rather than changing many training parameters at once.

A useful first pass would be:

  1. Generate the same case with the base model and no LoRA.
  2. If the base is sharp, sweep the LoRA strength instead of immediately retraining it.
  3. Compare an early, middle, and late training checkpoint under exactly the same inference settings.
  4. Inspect what the trainer actually sees after preprocessing, not just the original MP4 files.

Those four checks already separate a surprising number of failure modes.

For example, Diffusers supports varying adapter weight with set_adapters() / LoRA scaling, so something roughly like

base only
LoRA 0.25
LoRA 0.50
LoRA 0.75
LoRA 1.00

with the same prompt / input image / seed / resolution / frame count / sampler settings is a much more informative experiment than immediately trying another rank.

If the blur increases smoothly with LoRA strength, that points in a different direction from a case where even the base model is blurry.

A small symptom map might be:

What you see I would look at first
Base model is already blurry inference settings / VAE / model / runtime
Base is sharp, LoRA makes it blurry LoRA checkpoint / scale / loading path / training
Early checkpoint is OK, later ones degrade training progression / overtraining / effective update strength
Individual frames are soft everywhere preprocessing / VAE / spatial quality / training
Still areas are sharp but moving areas smear FPS / frame sampling / motion distribution / temporal path
Training validation looks good, normal inference looks bad validation-vs-inference mismatch / runtime / LoRA loading
First frame is strange, later frames blur I2V conditioning / VAE / model-specific denoising path

That does not identify the root cause by itself, but it reduces the search space quickly.

A slightly more complete decision tree

I would probably debug it in approximately this order:

START
  |
  +-- Is the base model sharp with the same input/settings?
  |      |
  |      +-- NO
  |      |    -> Do not retrain the LoRA yet.
  |      |       Check inference / VAE / scheduler / CFG / resolution /
  |      |       frame count / runtime / base-model limitations.
  |      |
  |      +-- YES
  |           |
  |           +-- Does lowering LoRA strength reduce the blur?
  |                  |
  |                  +-- YES
  |                  |    -> Adapter/checkpoint/training becomes more likely.
  |                  |
  |                  +-- NO
  |                       -> Check preprocessing, VAE, load path,
  |                          runtime and model-specific behavior too.
  |
  +-- Was an early checkpoint sharper than a later one?
  |      |
  |      +-- YES
  |      |    -> Training progression is a strong clue.
  |      |
  |      +-- NO
  |           -> Look earlier in the pipeline.
  |
  +-- Does the trainer-visible / decoded training data already look wrong?
  |      |
  |      +-- YES
  |      |    -> Fix data/preprocessing before tuning LoRA parameters.
  |      |
  |      +-- NO
  |           -> Continue to adapter/inference/model-specific checks.
  |
  +-- Does trainer validation look good but external inference look bad?
         |
         +-- YES
         |    -> Match the two inference paths before retraining.
         |
         +-- NO
              -> Then start tuning training parameters one at a time.

The important part is that this is a branching problem, not really a single “best LoRA configuration” problem.

Video preprocessing is worth checking very early

One particularly easy mistake with video training is assuming:

“My source videos look good, therefore my training data looks good.”

The model usually does not see the source MP4 exactly as you see it.

Depending on the trainer, there may be:

  • scene splitting
  • frame extraction
  • FPS conversion
  • frame truncation
  • resizing
  • random/center cropping
  • aspect-ratio buckets
  • temporal buckets
  • VAE encoding
  • cached latents
  • caption preprocessing

before the training step even starts.

This is why I like the workflow in the LTX-Video-Trainer dataset preparation guide.

It can preprocess the dataset into cached latents and then decode those latents back into videos for verification with --decode-videos.

That is an excellent debugging idea even if you are not using LTX:

inspect the representation immediately before the model learns from it.

If that decoded/transformed version is already softer, badly cropped, temporally strange, or missing important motion, changing LoRA rank will not repair the real problem.

FPS handling is also more subtle than it sounds.

For example, musubi-tuner’s dataset documentation explicitly says that when source_fps is specified, frames can be skipped to reach the model FPS, and that this skipping does not consider the image content. It recommends checking the result with:

--debug_mode video

If source_fps is not specified there, all frames are used regardless of the original video’s frame rate.

So even something apparently simple like “I trained on 30 FPS video” does not fully describe the temporal data the model received.

Also check:

  • Are the clips dominated by camera motion rather than subject motion?
  • Do the source clips themselves contain motion blur?
  • Are there many near-duplicate clips?
  • Are captions describing the important motion as well as appearance?
  • Are long scenes being squeezed into a very small number of sampled frames?
  • Are important subjects repeatedly near crop boundaries?
  • Is the training resolution much smaller than the source resolution?
  • Are frames/resolutions being silently truncated to model-supported shapes?

For a beginner, I would treat viewing the actual transformed training samples as a standard step, not an advanced debugging trick.

Make training validation and normal inference comparable

Another thing I would check before retraining is whether the output you call “bad” is being generated under the same conditions as the trainer’s validation output.

Try to match as many of these as possible:

base model / revision
prompt
input image/video
seed
height / width
number of frames
frame rate
number of inference steps
scheduler / sampler
CFG / guidance settings
LoRA scale
LoRA loading method
precision / quantization
VAE configuration

This matters because a trainer’s built-in validation loop is not necessarily identical to the pipeline you later use in ComfyUI, another UI, or a standalone Diffusers script.

Hugging Face’s Finetrainers validation dataset format, for example, allows validation cases to explicitly specify things such as:

  • num_inference_steps
  • height
  • width
  • num_frames
  • frame_rate

That is a good habit in general: make a small fixed validation set and stop changing it while comparing checkpoints.

Something like:

validation/
  prompt_A + seed_A
  prompt_B + seed_B
  prompt_C + seed_C

and then compare:

base
checkpoint_early
checkpoint_middle
checkpoint_late

under identical inference settings.

The goal is not to produce a beautiful benchmark. It is just to stop three variables from changing at once.

There are also real examples where runtime integration mattered. For example, this Wan 2.2 SwarmUI issue reported LoRA generations that were blurry/noisy in one path while appearing sharp through another backend with similar settings.

That issue is not evidence that your problem has the same cause; it is just a useful reminder that:

blurry output after loading a LoRA does not automatically prove that LoRA training itself failed.

Check that the LoRA is actually attached where you think it is

If your trainer uses PEFT underneath, another cheap sanity check is to verify the adapter structure itself.

The current PEFT troubleshooting guide recommends checking the number of trainable parameters:

model.print_trainable_parameters()

and inspecting adapter layers with:

model.get_layer_status()

or inspecting the module hierarchy.

One subtle point from the PEFT documentation is that if none of your configured target_modules match, PEFT raises an error, but if some match and some do not, the unmatched entries can be silently skipped.

That means “the training ran successfully” does not necessarily prove that the adapter targeted exactly the layers you intended.

This becomes especially relevant with newer or unusual transformer architectures, or when copying a LoRA config from a different model family.

So before spending another long training run, I would verify:

expected target modules
        ==
actual adapter modules

and make sure the trainable parameter count is in the range you expected.

Trainer settings are not portable vocabulary

I would also be careful when copying recipes between trainers.

A setting with the same friendly name can have different semantics in:

  • Diffusers training examples
  • Finetrainers
  • musubi-tuner
  • diffusion-pipe
  • SimpleTuner
  • AI Toolkit
  • other GUI wrappers

Things such as:

repeats
epochs
steps
FPS
frame count
bucket
cache
resume
validation
LoRA alpha
network dimension
timestep range

are not necessarily interchangeable just because their labels look similar.

Even video extraction itself can differ substantially.

That is why I would start from a maintained example for your exact base model and task, get that baseline running, save the exact config and software versions, and only then change one thing.

Hugging Face’s Finetrainers is useful here because it has model-specific examples rather than presenting video LoRA as one universal recipe.

For example, there are separate model guides/examples for:

If your actual model has a maintained trainer from its authors or a well-maintained model-specific implementation, I would normally use that as the first baseline rather than translating settings from an unrelated LoRA tutorial.

A useful reproducibility habit is to keep something like:

base model + exact revision
trainer + commit/version
dataset config
training config
seed
checkpoint numbers
validation prompts
inference config

next to every run.

That sounds boring, but once you have three LoRA attempts it becomes much more useful than memory.

Only after those checks would I tune rank / LR / steps

Rank, learning rate, alpha, optimizer, timestep sampling and training duration absolutely can matter.

I just would not make them the first explanation for blur.

A good order is:

1. establish a known-working pipeline
2. inspect processed data
3. establish fixed validation
4. compare checkpoint progression
5. verify adapter structure
6. then tune one training variable at a time

For example:

Run A: baseline
Run B: lower learning rate
Run C: baseline LR, lower rank
Run D: baseline again, fewer steps

is much easier to interpret than:

new LR + new rank + new optimizer + new captions + different resolution

in one run.

There are model-specific published starting points, but they should stay model-specific.

For example, the Diffusers CogVideoX training guide gives concrete observations about rank, steps, dataset size and learning rate from its experiments, but the guide also explicitly notes that the testing was not exhaustive and recommends finding settings appropriate for your data.

That is probably the right way to treat most “recommended LoRA settings”:

a starting point for the same model/trainer regime, not a law of LoRA training.

The checkpoint sweep is especially useful.

If:

500 steps  -> sharp but weak concept
1000 steps -> good
2000 steps -> blurry / oversaturated / distorted

then you have learned much more than you would from only evaluating the final checkpoint.

At that point, reducing effective training strength becomes a much more plausible next experiment.

The exact model matters more than it may seem

Video models are not all variations of the same architecture.

For example, Wan 2.2 can use a two-stage denoising setup with separate high-noise and low-noise transformers.

The current Diffusers Wan documentation exposes:

transformer
transformer_2
boundary_ratio

and also documents loading LoRA weights into the second denoiser with:

load_into_transformer_2=True

So for Wan 2.2, questions such as:

Which denoiser was trained?
Which denoiser is receiving the LoRA at inference?
Where is the timestep/noise boundary?

can be more important than asking whether rank 32 or 64 is better.

Likewise, if you are using CogVideoX, I would follow the CogVideoX-specific training/inference path before importing Wan or image-LoRA recipes.

There is even an open CogVideoX1.5-5B-I2V report where the base/zero-shot output had normal color, while applying a trained LoRA produced an overexposed first frame followed by blurry frames.

That does not mean your case has the same cause—the issue is not a general explanation for LoRA blur—but it is a nice example of why the shape of the symptom matters.

“Blurry” is more useful if you can distinguish:

every frame is spatially soft
motion smears but static regions stay sharp
first frame is abnormal
quality progressively collapses over time
colors/contrast blow out
identity is learned but fine detail disappears

Those point toward different branches.

A practical beginner route through the LoRA ecosystem

If you are still learning LoRA itself, I would separate two things:

1. Learn the adapter mechanics

PEFT is a good reference for what LoRA is actually targeting and saving:

PEFT documentation

and especially:

PEFT troubleshooting

You do not need to understand every internal detail, but it helps to know what:

rank
alpha
target_modules
trainable parameters
adapter weights

actually refer to.

2. Learn diffusion LoRA mechanics

The Diffusers LoRA training guide is useful for seeing the basic adapter/training structure in a relatively transparent form.

It is not a universal video-LoRA recipe, but it gives context for what the trainer is doing.

3. Move to a trainer/example for your exact video model

Good entry points include:

The LTX dataset guide is particularly educational even outside LTX because it walks through:

scene splitting
-> captioning
-> preprocessing
-> latent caching
-> decoding for verification
-> training

which is a good mental model for video fine-tuning in general.

I would learn the workflow first, and only then collect model-specific “tips”.

That avoids the common trap where you know that someone online used:

rank = X
lr = Y
steps = Z

but not what their dataset, model, preprocessing or trainer actually did.

If you want a very small default workflow, mine would be:

A. Reproduce a maintained example for the exact model/task.

B. Save the exact trainer version + config.

C. Verify the processed training samples.

D. Make 2-4 fixed validation cases.

E. Save several checkpoints, not only the final one.

F. Compare base vs LoRA at several adapter strengths.

G. If quality degrades during training, then tune LR/rank/steps
   one variable at a time.

That should give you much more information per training run than randomly searching for “professional LoRA settings”.

And if you post the base model, whether this is T2V or I2V, the trainer/version, and whether the base-only output is sharp under the same inference settings, people can probably narrow the tree much further without needing your entire project configuration.