I fine-tuned qwen-image-edit-2509 with lora method and checked the result weight matrix and found it is very sparse, meanwhile the weight number is very small and nearby zero.
Here is the statistics of some selected layers:
Is this normal? I wonder it will affect the final effect.
Seems normal in this case?
Yes, what you are seeing (very small, near-zero LoRA ΔW values for Qwen-Image-Edit-2509) is normal and expected, and by itself it does not mean your fine-tune is ineffective.
Below is a detailed breakdown of why this happens, how to interpret your statistics, when it would indicate a bug, and how to verify that your LoRA is actually changing Qwen-Image-Edit’s behavior.
For a linear layer with base weight matrix (W_0), LoRA re-parametrizes the adapted weight as
[
W’ = W_0 + \Delta W,\quad \Delta W = \alpha , B A
]
where
Key properties:
The original LoRA paper and follow-up surveys explicitly assume that the optimal fine-tuning update (\Delta W) lies in a low-rank, low-energy subspace and can be well-approximated by this small BA term. (arXiv)
So by design:
That is exactly what you are observing.
From your screenshot, for several to_q projections you computed
delta = (lora_B @ lora_A).flatten()
and reported approximately:
mean ≈ 4.8e-08std ≈ 2.86e-04max |ΔW| ≈ 0.002"sparsity" ≈ 0.31% where sparsity = fraction with |ΔW| < 1e-6Important implications:
Dense, not sparse.
Your “sparsity” metric says only 0.31% of entries are almost zero (|ΔW| < 1e-6). That means > 99% of the elements are non-zero.
A truly sparse matrix (in the usual sense) would have the opposite pattern (e.g., 90–99% zero).
Small-magnitude, but non-trivial.
Those are small compared to typical transformer weights (often ~1e-2 to 1e-1 after pre-training), but they are not “numerical noise”. This is a normal scale for LoRA deltas: small relative to (W_0) yet non-zero.
Zero-mean, bell-shaped histogram is expected.
So the matrix is:
which matches exactly what LoRA theory suggests for (\Delta W). (cognativ.com)
Several independent sources point out that LoRA deltas should be small:
LoRA paper (Hu et al., 2021).
They show that the full fine-tune update (\Delta W) for large models can be well-approximated by a low-rank correction, which implicitly means (\Delta W) is low-energy (small norm) and does not drastically overwrite the base model.
“A Note on LoRA” (Fomenko et al., 2024).
This paper explicitly states that LoRA delta weights are small, and then studies serving overhead for swapping those small delta weights in and out. (arXiv)
Surveys and tutorials on LoRA / PEFT.
Empirical analyses of LoRA weight distributions.
Analyses on both LLMs and vision models consistently find LoRA-induced weights have significantly smaller standard deviation and spectral norms than full fine-tuning, yet still produce comparable downstream performance. (arXiv)
Intuitively:
So the fact that your ΔW entries lie roughly in [−0.002, 0.002] with std ~3×10−4 is not a red flag; it is close to what you want.
Yes. Small but structured deltas can have a large semantic impact because:
Real-world LoRAs for Qwen-Image-Edit show exactly this behavior:
Qwen/Qwen-Image-Edit and is advertised as significantly improving scene coherence, object positioning, and camera perspective for in-scene edits.lovis93/next-scene-qwen-image-lora-2509 similarly describe clear cinematic behavior changes at modest LoRA strengths (0.7–0.8). (huggingface.co)Those public Qwen-Image-Edit LoRAs were trained with the same kind of small ΔW updates you are seeing. The fact that they work in practice is strong evidence that small ΔW is enough.
So the key question is not “are the numbers big?” but “does the LoRA change outputs when applied?”.
There are known failure modes where LoRA really does “do nothing”. They look very different from your stats:
Exactly zero deltas (implementation bug).
lora_B remained all zeros after training; ΔW stayed exactly zero, so the model was identical to the base despite training. (Reddit)In those cases you’d see:
std == 0, max == 0 (or machine epsilon)That is not what your numbers show.
LoRA weights exist but are never applied.
In some Stable Diffusion / SD-WebUI / InvokeAI issues, LoRAs loaded but the inference pipeline didn’t actually add ΔW to the UNet weights, so outputs were identical to the base model. The LoRA files themselves were fine; the bug was in how they were wired into inference. (huggingface.co)
In that failure mode:
Your stats (non-zero std, non-zero max, only 0.3% under 1e-6) clearly indicate you are not in the “everything zero” pathological regime.
To be confident your Qwen-Image-Edit LoRA is doing what you want, run these checks:
You already did this for some to_q layers. Repeat for the other modules you trained (e.g., to_k, to_v, to_out, any MLP or image-side modules):
def describe_delta(lora_A, lora_B, name, atol=1e-6):
d = (lora_B @ lora_A).flatten()
mean = d.mean().item()
std = d.std().item()
maxv = d.abs().max().item()
sparsity = (d.abs() < atol).float().mean().item()
print(f"{name:25s} mean={mean:+.2e} std={std:.2e} "
f"max={maxv:.3e} sparsity={sparsity*100:.2f}%")
You want:
This is the decisive test.
You should see:
For reference, public Qwen-Image-Edit LoRAs (InScene, Next Scene, InStyle, etc.) recommend strengths around ~0.7–0.8 for good balance, and their authors show visually obvious differences vs base.
If you see no visible difference even at high scale, then you should investigate:
But the small ΔW per element is not itself the problem.
As a sanity check:
flymy-ai/qwen-image-edit-inscene-lora or lovis93/next-scene-qwen-image-lora-2509. (DataCamp)to_q and to_k matrices.If your numbers are in the same ballpark as these known-good LoRAs, that’s strong evidence everything is normal.
LoRA is explicitly built on the assumption that the update matrix (\Delta W) is small and low-rank; it is written as (\Delta W = B A) with small rank r and small magnitude entries. (AI Bites)
Your measured ΔW for Qwen-Image-Edit-2509 has:
This is exactly what the LoRA literature and real Qwen LoRAs show as normal behavior, not a sign of failure. (arXiv)
What actually matters is whether your LoRA changes outputs when applied (and how). That is best checked by toggling the adapter on/off and sweeping the LoRA strength.
You asked for links; here is a small, focused set (they should appear as clickable citations):
flymy-ai/qwen-image-edit-inscene-lora – model card showing before/after images for an in-scene LoRA trained on Qwen-Image-Edit. Good visual reference for what a working LoRA can achieve.FlyMyAI/flymyai-lora-trainer – open-source trainer specifically for LoRA on Qwen-Image, Qwen-Image-Edit, and FLUX; useful as a reference for configs and expected behavior. (GitHub)lovis93/next-scene-qwen-image-lora-2509 and related tutorials / spaces – show cinematic “next scene” behavior for Qwen-Image-Edit 2509 using only a LoRA adapter and typical strength settings. (huggingface.co)If you like, you can take one of those public Qwen-Image-Edit LoRAs, dump its lora_B @ lora_A stats next to yours, and you’ll have a very concrete, side-by-side confirmation that your “tiny, near-zero” ΔW is right where a healthy LoRA usually lives.
As John already mentioned, by default, LoRA B is initialized as a zero matrix, so at the start of the training, it is expected that dW is close to zero. You should monitor other metrics to see how well the training goes, possibly increase the learning rate to see results more quickly.
Depending on framework you use for LoRA training, you may also have other initialization options than zero init for B. E.g. there could be an option for orthogonal initialization, which might result in faster learning. Try out a few options.