I think this is an interesting result:
The part I find most interesting is not simply that a fixed logit bias can produce lexical steering, but that your causal controls suggest the support, rank ordering, and magnitudes are not trivially interchangeable. In particular, top1–50 dying while full200 works, the largest-coordinate deletion surviving, and the shuffled/equal controls dying makes this worth decomposing further rather than reducing it to “just boost some topic words.” Your current framing — lexical forcing rather than semantic transport, with low-rate / pilot-level evidence — also seems like the right conservative baseline to build from.
There is one separation I would probably make before adding many more concepts or models, though: separate the words used to score the effect from the coordinates being directly manipulated.
Your neighbor_probe.py already does essentially the clean version of this for the semantic test: SEM and UNR tokens are explicitly filtered to tokens not in top-K. I think applying the same contract to the K/rank and λ experiments would make the causal story much easier to interpret.
As a small sanity check, reconstructing the current FANTASY contrast with Qwen2-1.5B put evil at about contrast rank 136, so it becomes a directly nonzero coordinate almost exactly where the reported K≈150 window begins. I would not use that to explain the original published 5/30 run — my local T4/fp16 proxy did not reproduce the exact bf16 seed outcomes — but it is enough to show that the present evaluation and intervention can be coupled.
That seems more like an opportunity for a cheap control than a problem with the experiment.
My default route would be:
| Question |
Cheap control |
| Direct lexical efficacy vs effect on unmanipulated evaluation terms |
Report the current metric and a support-excluded metric |
| Does one directly scored coordinate matter? |
Zero evaluation-token coordinates in full200, then rerun the same paired seeds |
| Is the K window about support size or redistributed dose? |
Compare raw-K and current norm-matched K |
| Is the useful mass broadly distributed or concentrated in one rank region? |
full200 minus 1–50, 51–100, 101–150, 151–200 |
| Is the λ effect geometry or direct lexical dose? |
Clamp evaluation coordinates to zero in every λ vector before comparing λ |
So the first version could be almost entirely evaluation-side:
- keep your current “direct lexical hit” metric unchanged;
- add a second support-excluded hit metric that only counts evaluation terms whose relevant token coordinate is zero in the intervention;
- if that second signal survives, then spend compute on the K/rank/geometry decomposition.
For FANTASY, an especially cheap OFAT-style check would be to remove the evil coordinate from full200 and rerun the same seeds. In my reconstruction that removal barely changed the vector direction (cosine ≈ 0.998), so it is close to removing the measurement coupling while preserving almost all of the original vector.
The nice thing is that every branch is informative:
- support-excluded effect survives → much stronger evidence that the ranked structure is doing something beyond directly forcing the words used to score it;
- only the direct-hit metric survives → still a useful sparse ranked lexical-steering result, but the K onset should be interpreted more narrowly;
- mixed result → report two effects separately: direct lexical efficacy and support-excluded / indirect lexical effect.
I would do that before a larger model/concept sweep because it seems much cheaper and has higher information gain.
For the λ experiment I would use the same principle, but with a slightly different control. After projection, the vector becomes dense, so “only score terms outside top-K” is no longer sufficient. Instead, choose the evaluation token set first, then set those coordinates to zero in every λ vector before generation (optionally norm-match again afterward). If the λ<1 / λ=1 difference survives that, the geometry interpretation becomes considerably cleaner.
Why I would separate the evaluation vocabulary from the steering support first
There are really three different meanings of “held-out” here:
- a word was not intended as one of the construction anchors;
- the literal word does not occur in the target construction sentences;
- the token coordinate used to score that word is zero in the actual steering vector.
For the causal question, (3) is the important one.
The current rank_causality.py builds the steering vector in vocabulary coordinates and then scores generated text against HELD_OUT. Those two sets are not explicitly made disjoint.
Using the current Qwen2-1.5B revision and the same contrast construction, I get examples like:
| Concept |
evaluation term directly inside top-200 |
| FANTASY |
evil around rank 136 |
| SPACE |
orbit around rank 2; spaceship around rank 62 |
| PIRATE |
sail around rank 49 |
There is a second, different kind of overlap as well: some nominally held-out words occur literally in construction sentences (beast in FANTASY, astronaut in SPACE, several PIRATE terms). Literal construction overlap does not necessarily mean that the final token coordinate lands in top-K, so I would keep these two diagnostics separate.
A minimal audit could be something like:
order = perz.argsort(descending=True)
rank = torch.empty_like(order)
rank[order] = torch.arange(len(order))
active = set(order[:K].tolist())
for word in HELD_OUT.split():
ids = tok(" " + word, add_special_tokens=False).input_ids
if len(ids) == 1:
tid = int(ids[0])
print(
word,
"rank =", int(rank[tid]) + 1,
"directly_steered =", tid in active,
"delta =", float(dL[tid]),
)
Then I would retain both scores:
direct_hit:
current lexical metric
support_excluded_hit:
same metric, but excluding lexical items whose token
coordinate is directly nonzero in this intervention
That also fits nicely with what the repo already does in neighbor_probe.py, where the semantic and unrelated probe tokens are explicitly filtered to tid not in idxK.
So this would be extending an evaluation principle you are already using, rather than introducing a different methodology.
One caveat on my numbers: my local generation checks used a current Transformers stack on a T4/fp16 runtime, whereas the post reports bf16. I did not reproduce the exact published success-seed set, so I would not infer that the original successes were caused by evil. The useful observation is only that the construction permits direct intervention on evaluation coordinates and that the K boundary can coincide with that entry point.
Why I do not think the direct-hit observation explains the whole result
I would also avoid going too far in the opposite direction and reducing the experiment to a one-token artifact.
There is evidence against that simple explanation in your own controls.
For example, the shuffle/equal controls are important precisely because support membership alone does not appear sufficient. If the right lexical coordinates were all that mattered, then preserving the same rough support while destroying the rank↔magnitude correspondence should be much less damaging than what you observe.
Likewise, SPACE can contain very high-ranked directly scored coordinates without showing the same robust behavioral effect.
So I would currently separate these two statements:
Fairly strong:
Some evaluation coordinates can be directly manipulated, so the current transport metric does not isolate an effect beyond direct lexical forcing.
Not established:
Direct forcing of those evaluation coordinates explains the entire ranked-steering phenomenon.
The second statement would be too strong.
The shuffle/equal results leave room for something genuinely interesting about:
- which coordinates are selected;
- their relative magnitudes;
- how those biases interact with the model’s native next-token distribution;
- and possibly how enough related coordinates cooperate during sampling.
That is why I think the support-excluded control would strengthen the project regardless of which way it comes out.
There is a recent methodological parallel in Yuan & Vlachos, Evaluating the Semantic Specificity of Representation Steering in Language Models. Their task and intervention are different, but the useful idea is similar: explicitly separate a purported transferred/semantic effect from a simpler direct output bias that can produce the same benchmark success.
I would treat that as a control-design analogy, not evidence that the same mechanism is happening here.
K currently changes more than just the number of coordinates
One thing I would be careful about when interpreting the K window is the norm matching.
The retention code intentionally rescales the surviving top-K vector to the reference dose:
def rescale(v):
n = v.norm()
return v * (N_REF / n) if n > 1e-9 else v
That is a perfectly reasonable control for total L2 dose, but it means K is not a pure “number of coordinates” intervention.
As K becomes small:
- coordinates are removed;
- the surviving coordinates are amplified;
- the largest individual applied logit offsets can become much stronger;
- degeneration/repetition pressure can change;
- and if an evaluation token is already included, its own direct dose can change with K.
So I would compare two curves rather than choose one normalization:
raw-K:
truncate to top K, preserve original coordinate magnitudes
norm-matched-K:
truncate to top K, then restore reference L2 norm
If both show the same K window, the support-size interpretation gets stronger.
If they differ substantially, that is still useful: it means support size and dose concentration are different causal knobs.
After that, I think leave-one-band-out is cleaner than isolated norm-matched bands:
full200
full200 - ranks 1:50
full200 - ranks 51:100
full200 - ranks 101:150
full200 - ranks 151:200
Why I prefer this direction:
- it keeps most of the known-working vector intact;
- it tests necessity rather than asking a small fragment to work alone;
- it avoids concentrating the full reference norm into a narrow band;
- and it directly distinguishes “broadly distributed dependence” from “one particular mid-rank region is doing most of the work.”
Possible interpretations:
several band removals each hurt
-> broad distributed dependence becomes more plausible
one band removal dominates
-> narrow the claim to that rank region
raw-K and norm-matched-K disagree
-> the K effect partly depends on dose redistribution
nothing survives support-excluded scoring
-> the original K curve mainly characterizes direct ranked lexical forcing
None of these outcomes makes the experiment uninteresting; they just identify a different mechanism.
Bias-before-top-p: candidate entry vs reweighting
There is another useful split that may explain why the same fixed logit bias behaves differently across prompts/concepts.
Because the steering offset is added before nucleus truncation, it can do two different things:
-
candidate entry
move a token from outside the base-model top_p nucleus into the candidate set;
-
within-candidate reweighting
increase the probability of a token that was already plausible enough to be in the base nucleus.
Those are mechanically different effects.
In a small proxy run I saw boosted coordinates enter the post-bias nucleus, but freezing the base-model nucleus did not eliminate every direct lexical hit. So I would not assume either mechanism explains everything.
A cheap diagnostic is just logging, per decoding step:
base nucleus membership
steered nucleus membership
number of boosted coordinates entering/leaving
total probability mass of entering boosted coordinates
rank/probability of each evaluation token:
before bias
after bias
after top-p
Then a frozen-candidate control can distinguish:
base logits
-> determine nucleus membership once for this step
steered logits
-> reweight only inside that fixed nucleus
If the effect survives, it does not require candidate entry.
If it disappears, pushing new lexical candidates through the top-p boundary is probably important.
If it partially survives, both mechanisms contribute.
This is not the same algorithm, but Contrastive Decoding is a useful conceptual neighbor because it explicitly separates a contrastive preference signal from a plausibility constraint. I think the same vocabulary is helpful here: preference/bias and candidate plausibility are separate knobs.
More broadly, decoding-time logit control has a long lineage — e.g. FUDGE and DExperts — so I would probably position the distinctive question here as the causal structure of this particular fixed contrast vector, rather than logit control itself.
For the λ experiment, I would clamp the evaluation coordinates
I think the λ experiment needs a slightly different form of the same control.
In mechanism_matrix.py, the top-K vector is decomposed using
P = W(W^T W)^{-1}W^T
with W = model.lm_head.weight.
Since W has shape vocabulary × hidden, this projector maps a vocabulary-logit vector into the column space / image of the unembedding.
So I would probably call it image(W) / col(W) in the writeup, or at least define row(W) as project shorthand, because the mathematical row space of a vocabulary × hidden matrix lives in hidden-dimensional space rather than vocabulary-logit space.
More importantly, projection makes the vector dense.
That means the λ sweep changes several things at once:
- image-vs-orthogonal-residual fraction;
- direction relative to the reference vector;
- sparsity/density;
- individual lexical-coordinate doses;
- including the coordinates used to evaluate the effect.
So a denser λ grid alone would not isolate the geometry question.
I would first do:
eval_ids = {...} # fixed before looking at lambda outcomes
for lam in lambdas:
v = make_lambda_vector(lam)
for tid in eval_ids:
v[tid] = 0.0
# optional, if constant dose remains the intended control
v = rescale(v)
run(v)
Then report both:
unclamped lambda curve
evaluation-coordinate-clamped lambda curve
That gives a very clean branch:
-
λ boundary survives clamping
→ substantially stronger evidence for a geometry / representability effect;
-
λ boundary disappears
→ direct lexical dose on the scored vocabulary was part of the endpoint difference;
-
partial survival
→ both are involved.
In my proxy diagnostics, the direct offset on evil changed substantially across λ and was much smaller at the pure projected endpoint. Again, I would not use that to reinterpret your original runs by itself, but it is exactly the kind of coupled variable that clamping can remove very cheaply.
So I would phrase the current result as something like:
the pure projected endpoint is behaviorally different, but the present interpolation does not yet isolate representability from coordinate-wise lexical-dose changes.
That preserves the observation without asking it to support more than the current design separates.
A few small implementation / reproducibility notes
These seem secondary to the evaluation separation above, but they are cheap to clean up.
1. Random-coordinate deletion
In the current rank_causality.py, the random indices are sampled from the whole vocabulary:
ridx = torch.randperm(V, generator=gi)[:50]
and full_minus_rd then does:
v = torch.zeros(V)
v[top_idx[:K]] = perz[top_idx[:K]]
v[int(ridx[0])] = 0.0
Because v is sparse, a random vocabulary coordinate will almost always already be zero.
I would sample the deletion from the active support instead:
active = top_idx[:K]
j = torch.randint(
0,
len(active),
(1,),
generator=gi,
).item()
delete_tid = int(active[j])
v[delete_tid] = 0.0
and repeat several draws.
The largest-coordinate deletion result remains useful independently: it already says the single largest coordinate is not individually necessary. This just makes the random comparator test the intended question.
2. DYN_PREFIX
I may be misunderstanding the intended definition, but the current dynamic_contrast.py branch labeled dyn_prefix appears to recompute:
anchor = torch.stack([logits_of(s) for s in tgt]).mean(0)
p = perz_of(anchor)
inside the decoding loop without incorporating the generated live prefix into those target strings.
So I would interpret the current result narrowly as:
this implemented recomputation does not improve over static.
I would not yet use it as evidence that a genuinely prefix-conditioned contrast cannot help.
If true live-prefix adaptation is still interesting, it would be worth defining the intended counterfactual explicitly first, because there are several non-equivalent possibilities.
3. Cache/runtime contract
The custom loops also use past_key_values directly. Cache APIs have changed over Transformers versions; current Hugging Face documentation describes DynamicCache as the default growing cache.
For future reproducibility I would log at least:
model revision
transformers version
torch version
dtype
GPU
cache type
and in manual decoding loops make the caller contract explicit:
out = model(
input_ids=ids,
past_key_values=past,
use_cache=True,
)
past = out.past_key_values
I would treat this as reproducibility hygiene, not evidence that the existing reported runs used the wrong cache semantics; that depends on the exact runtime version.
4. Output degeneration
The current coherence filter checks:
- identical-token run < 6;
- type/token ratio > 0.6.
That can still allow short phrase loops such as:
... summoned evil summoned evil summoned evil ...
because no single token repeats six times consecutively.
A very cheap post-hoc addition would be:
repeated-bigram fraction
repeated-trigram fraction
duplication / repeated-span ratio
No generation changes are needed.
This lets “steering efficacy” and “output quality” remain separate measurements rather than allowing a lexical success caused by a loop to count exactly like a fluent success.
5. Paired seeds
Since conditions reuse the same seed IDs, the per-seed outcomes also naturally form paired data.
An exact McNemar test / paired permutation on discordant seeds would be a useful complement to independent Fisher tests.
I would keep this low priority, though. The post already describes the main K-window results as suggestive rather than significant, and fixing the causal/evaluation separation seems much more important than changing the statistical test.
How I would summarize the current causal picture
At the moment I would separate the evidence roughly like this.
Seems fairly well supported
- The working intervention is primarily lexical, not semantic-neighbor transport.
- The largest single coordinate is not necessary.
- Very small top-K retention does not reproduce the working behavior under the current normalization.
- Arbitrary support / shuffled magnitude controls do not reproduce the same result in the tested cells.
- The effect is prompt/concept dependent and low-rate.
Still entangled
- “K≈150–300 is necessary because the mechanism is broadly distributed.”
- “The K onset is independent of directly scoring tokens that enter the support around that K.”
- “λ=1 fails because the vector became fully representable by the unembedding geometry.”
- “Dynamic prefix recomputation does not help.”
- The random-coordinate-deletion comparison.
Cheap path to a stronger claim
A. Separate direct-hit and support-excluded metrics
|
+-- strict effect disappears
| -> characterize direct ranked lexical steering
|
+-- strict effect survives
|
B. raw-K vs norm-matched-K
|
C. leave-one-band-out
|
D. evaluation-coordinate-clamped lambda
If A survives through D, I think the phrase distributed ranked structure becomes much easier to defend causally.
If A fails, there is still a clean result:
a target-neutral contrast can identify a sparse/ranked vocabulary bias that causes lexical topic forcing, with efficacy depending on which coordinates and magnitudes are retained.
That is narrower, but still useful and much easier for another person to reproduce and extend.
And a mixed outcome may actually be the most informative one: a direct lexical component plus a weaker support-excluded component, each with its own K/geometry behavior.
The reason I like this route is that it mostly reuses the existing pipeline. It does not require jumping immediately to 7B models, dozens of concepts, or a new benchmark before the mechanism inside the current 1.5B experiment is separated.
Overall, I think the ranked result is worth continuing. I would just make the next round less about adding more cells and more about making each cell answer one question:
- evaluation support vs steering support;
- support size vs coordinate dose;
- candidate entry vs reweighting;
- geometry vs direct lexical offset.
If the signal survives those separations, the resulting story will be considerably stronger; if part of it disappears, the remaining lexical-steering result becomes more precisely characterized rather than lost.