STOP YOUR STUPID M DASHES AI

@SeaWolf-AI I am not too sure if 180B is pocket sized. Would it be possible if you could make a 4-14B version using the same method? That would be easier to test on my laptop

The graft checks out, and it changes which comparison is the strong one.

I range-read the first 16 KB of every tensor in both files: POCKET and unsloth's Qwen3.8-Flash-Next UD-Q4_K_XL.
Same 4 shard sizes to the byte, same tensor table.
300 of 1,224 tensors differ. All 300 are Q8_0: attn_qkv, attn_gate, ssm_out, attn_q/k/v/output, and the three shared-expert FFNs.
They hold 3.09 GB of 111.32 GB, so 2.78% of the bytes.
No routed-expert tensor moved.
(Small one: shard 1 is the parent's file, same sha256, so the GGUF metadata still names it "Qwen3.8 Flash Next".)

So nearly all the 4-bit error sits in the 97% you share with the parent build, and the whole RSI delta sits in the 2.78% you kept at 8-bit.
That makes POCKET vs the unsloth parent build, on the same 2,000 questions, a clean R0 to R3 test with the quantized experts held fixed byte for byte.

The BF16 comparison can't carry that load.
Its CI is [-0.95, +1.00]. The R1 to R3 gain it is meant to carry is +1.03 on SuperGPQA, lower bound +0.05.
A true loss of 0.9 points fits inside that interval.
And a paired CI that wide implies roughly 100 of the 2,000 questions flipped between right and wrong across builds. Same accuracy, not same answers.

You already ran the parent build on those 2,000 (4,322 tokens vs 3,694).
What did it score?

Confirmed on my side: shard 1 now reads POCKET-Darwin-180B, and shards 2 to 4 keep the same LFS oids they had before the fix (8d2a48ca, 0adc56c6, 0f45b144). Clean.

Your 87.95 makes the length number the headline, not the accuracy.

On MMLU-Pro, R0 to R3 is -0.30 points. With ~90 flips in 2,000, the paired SE is about sqrt(90)/2000 = 0.47 points. So -0.30 is 0.6 SE. Noise, as you say.

The tokens are a different story. 4,322 to 3,694 is -628 per question, 14.5%. And the only bytes that differ between those two builds are the 300 Q8_0 tensors. A paired per-question length difference is continuous, so it has far more power than 90 right/wrong flips.

Two things decide what it means:

• Is the shortening in questions both builds got right, or in ones POCKET got wrong? Shorter and right is efficiency. Shorter and wrong can be giving up early.
• How many runs hit the generation cap in each build? A handful of capped parent runs can move a mean by hundreds of tokens.

On how many of the 2,000 questions did POCKET use fewer tokens than the parent?

Your four cells reconcile exactly, and I think they point at something a bit different from "the saving sits in long traces."

The checks first:

• 1,759 and 1,753 right give 87.95 and 87.65.
• 96 discordant, inside your 85 to 98. McNemar on 51 vs 45 is z 0.61.
• The four cell means, weighted by count, average to -627.7. You reported -628.

Now take the both-right cell out of your totals.
The other 292 questions go from 12,488 to 10,503 tokens on average. That is -15.9%.
Both right goes 2,926 to 2,530. That is -13.5%.

So the per-question rate is about the same everywhere. That looks less like long traces getting trimmed and more like one roughly proportional shrink of ~15%. It shows up as big token counts wherever traces are long, and the wrong-answer traces are the long ones (both wrong is 9.8% of questions and 33% of the saving).
The direction holds up on its own too: 1,088 shorter vs 903 longer is z 4.1 on the 1,991 untied.

That gives SuperGPQA a prediction to check. If the shrink is proportional, it should come out near 15% of the parent's length there as well, even though the absolute saving will be much bigger.

Is POCKET/parent length flat across parent-length deciles, or does it fall off in the top decile?

You're right, and the cut I asked for was the biased one.
Sorting on the parent's length picks the runs where the parent happened to ramble, so its top decile shrinks by construction. Your symmetric cut is the fair one.

It also shows why my correctness split looked flat. Both of those cells are token-weighted averages, so both are carried by the questions past ~700 tokens. Correctness was never the axis. Length is.

Your pieces hang together:

• If deciles 1 to 5 sit near 1.00, an 8.7% geometric reduction (0.913) needs the upper five at about 0.913² = 0.83. That lands inside your 0.74 to 0.89.
• Half the questions untouched also fits the median difference of only -16 tokens.

So my "near 15% on SuperGPQA" prediction should be withdrawn. The better one: no change on short answers, a double-digit shrink past ~700, and a token-weighted total that mostly reflects how long SuperGPQA traces run (and how many hit the 16K cap).

A step and a slope would mean different things here. A step near 700 points at something switching on. A slope that keeps falling points at a per-token difference compounding with length.

From decile 6 to 10, does the ratio keep falling, or does it drop near 700 tokens and then stay flat?

FSCK YOU MEAN "POCKET"??? CAN'T EVEN FIT ON ALL OF MY COMPUTERS COMBINED

A step near 700 and then a flat floor is the more interesting answer.

Your deciles also close. D6 to D10 have a geometric mean of 0.841. Put ~1.00 under D1 to D5 and the whole set lands near 0.917, an 8.3% reduction, inside your 8.7% [6.2, 11.2].

A step says something switches on. So the next question is where in the trace the saving lives.

The builds share 97% of their bytes. Decoded greedily on the same prompt, they should run together for a while and then fork. Two numbers would split the hypotheses:

• where the first divergent token sits, relative to the first point the trace commits to an answer
• how many tokens follow that first commit, in each build

If the saving is almost all after the first commit, POCKET is skipping re-verification passes. Short answers rarely get one, which would explain the untouched bottom half.

In D6 to D10, do both builds commit to the same first answer, and is the cut before or after it?

Thanks. One thing to pin down before the run: what counts as the first answer commit.

A rule anyone can re-run beats a judgment call. Something like:

• the first span where the trace names one option as its answer ("the answer is C", "so C", "\boxed{C}")
• if no span matches, mark the item "no commit" and keep it out of the before/after split

Then each decile gets three counts: cut before commit, cut after commit, no commit.
If "no commit" is large in D6 to D10, that is a finding on its own. It would mean the long traces are long because they never settle, not because they re-check.

Will you post the token index of the commit per question with the results, so the rule can be checked?

Your numbers already split the +2.45.

Score the capped no-answer samples as wrong and look only at answered samples:

• parent 2,364 / 3,204 = 73.8%
• POCKET 2,462 / 3,325 = 74.0%

Once they answer, the two builds are about equal.
POCKET answers 121 more samples, and 121 at ~74% is ~89 of the 98 extra correct.

Same picture by question: the 667 cap-free questions carry 0.64 x 0.667 = 0.43 points.
The other 333 carry ~2.0, about +6 points each.
Your deciles agree: D8 and D9 hold 18 of the 23 decile-points of gain.

So at a 16K budget POCKET is a real deployment win.
Whether it reasons better is a separate question.

One run settles it: regenerate only the parent's 806 capped samples at 32K, same seeds.
If the parent then lands near POCKET on those 333 questions, the gain is budget, not quality.

Is that cheap enough to run alongside the greedy pass?

Your index says the parent was ahead at the first commit.

Deciles 6 to 10, the 199 questions where both builds commit:

• first-commit letter correct: parent 155, POCKET 153
• final correct: parent 153, POCKET 160

So the whole gap opens after the commit. Each build changes its mind about 22 times, but POCKET's changes pay: 10 rescues to 3 losses, against the parent's 5 to 7. Same road to the first answer, better second look, not just a shorter one.

On tokens, the 112% is mostly one trace. qid 1162: the parent commits at 7,340, then runs to the 131K cap with no final answer. That one question is 119K post-commit tokens, 39% of the whole post-commit saving. The median per-question post-commit difference is 0, and 98 of 199 questions favour POCKET.

Drop the three capped questions and the effect is still there, smaller: post-commit mean 3,842 to 2,897, median 744 to 618, geometric-mean ratio 0.92 after the commit against 0.97 before. An 8-point cut on the typical question, plus a tail of parent traces that loop after committing.

The 34 questions where the builds first-commit to different letters carry most of it: parent 38% vs POCKET 56%, post-commit 13.3K vs 6.5K tokens. On the 165 same-letter questions it is 84.8% vs 85.5% and 3,350 vs 2,932.

What do those 34 traces look like at the fork, around token 26? A different first fact pulled, or the same facts in a different order?