Abstract
On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from 62.27% rightarrow 67.04% at 2B, 71.30% rightarrow 73.16% at 4B, and 72.51% rightarrow 76.26% at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.
Community
๐ Visual Contrastive Self-Distillation (VCSD)
Can on-policy self-distillation improve Vision-Language Models without privileged answers or auxiliary visual evidence? Yes.
The Core Idea: VCSD compares the same EMA teacher under two matched visual conditions, the original image and a content-erased control, while keeping the prompt and student-generated response prefix unchanged.
Pure Input Conditioning: The token-wise prediction difference reveals which candidates are specifically supported by the instance-level image content, turning input contrast into a self-distillation signal.
Contrast-Shaped Target: VCSD uses this visual contrast to sharpen the teacherโs original-image distribution within its plausible support, then distills the resulting full-distribution target into the student.
No Auxiliary Supervision: No external teacher, answer hints, reasoning traces, cropped or annotated visual evidence, or additional inference-time cost.
Consistent Gains: Across 6 Qwen3-VL/Qwen3.5 models and 7 vision-language benchmarks, VCSD outperforms matched OPSD on every model by +1.76 to +5.33 points. It also improves over the corresponding base models by up to +4.77 points, with Qwen3.5-9B reaching 79.24 average accuracy.
๐ Project and Code: https://joliang17.github.io/VisualCSD/
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- V-Zero: Answer-Label-Free On-Policy Distillation with Contrastive Evidence Gating for Fine-Grained Visual Reasoning (2026)
- Contrastive On-Policy Distillation (2026)
- Self-Distilled Policy Gradient (2026)
- Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization (2026)
- RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation (2026)
- DemoPSD: Disagreement-Modulated Policy Self-Distillation (2026)
- Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2607.21556 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
