Papers
arxiv:2607.21556

Visual Contrastive Self-Distillation

Published on Jul 23
ยท Submitted by
Yijun Liang
on Jul 24
Authors:
,
,
,
,
,

Abstract

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from 62.27% rightarrow 67.04% at 2B, 71.30% rightarrow 73.16% at 4B, and 72.51% rightarrow 76.26% at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

Community

Paper submitter

๐Ÿš€ Visual Contrastive Self-Distillation (VCSD)

Can on-policy self-distillation improve Vision-Language Models without privileged answers or auxiliary visual evidence? Yes.

  • The Core Idea: VCSD compares the same EMA teacher under two matched visual conditions, the original image and a content-erased control, while keeping the prompt and student-generated response prefix unchanged.

  • Pure Input Conditioning: The token-wise prediction difference reveals which candidates are specifically supported by the instance-level image content, turning input contrast into a self-distillation signal.

  • Contrast-Shaped Target: VCSD uses this visual contrast to sharpen the teacherโ€™s original-image distribution within its plausible support, then distills the resulting full-distribution target into the student.

  • No Auxiliary Supervision: No external teacher, answer hints, reasoning traces, cropped or annotated visual evidence, or additional inference-time cost.

  • Consistent Gains: Across 6 Qwen3-VL/Qwen3.5 models and 7 vision-language benchmarks, VCSD outperforms matched OPSD on every model by +1.76 to +5.33 points. It also improves over the corresponding base models by up to +4.77 points, with Qwen3.5-9B reaching 79.24 average accuracy.

๐ŸŒ Project and Code: https://joliang17.github.io/VisualCSD/

fig2_method

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2607.21556
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2607.21556 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2607.21556 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2607.21556 in a Space README.md to link it from this page.

Collections including this paper 2