nvidia

/

Qwen3.8-Flash-Next-NVFP4

Like 202

Follow

NVIDIA 68.1k

See here: https://github.com/mratsim/sglang-qwen38fn-sm120-turbo/issues/3

Much faster and larger KV. Must be patched but is worth it!

I patched vllm for the same issue and grafted the BF16 MTP onto this model and now it works great:

https://huggingface.co/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix

I patched vllm for the same issue and grafted the BF16 MTP onto this model and now it works great:

https://huggingface.co/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix

You should look into sglang and my patch above. Here are the results of the same tests:

llama-benchy Results

┌──────────────────────┬────┬────────┬────────┬───────────┬────────────┬─────────┐
│ Test │ c │ pp t/s │ tg t/s │ TTFT (ms) │ Total (ms) │ Tokens │
├──────────────────────┼────┼────────┼────────┼───────────┼────────────┼─────────┤
│ pp2048 tg32 @ d0 │ c1 │ 36,785 │ 232.4 │ 172 │ 194 │ 2048+32 │
├──────────────────────┼────┼────────┼────────┼───────────┼────────────┼─────────┤
│ pp2048 tg32 @ d4096 │ c1 │ 18,050 │ 200.8 │ 456 │ 501 │ 2048+32 │
├──────────────────────┼────┼────────┼────────┼───────────┼────────────┼─────────┤
│ pp2048 tg32 @ d8192 │ c1 │ 16,307 │ 211.3 │ 744 │ 780 │ 2048+32 │
├──────────────────────┼────┼────────┼────────┼───────────┼────────────┼─────────┤
│ pp2048 tg32 @ d16384 │ c1 │ 15,145 │ 199.7 │ 1,333 │ 1,378 │ 2048+32 │
├──────────────────────┼────┼────────┼────────┼───────────┼────────────┼─────────┤
│ pp2048 tg32 @ d32768 │ c1 │ 14,378 │ 206.6 │ 2,538 │ 2,577 │ 2048+32 │
└──────────────────────┴────┴────────┴────────┴───────────┴────────────┴─────────┘

🏆 Benchmark Complete — Score: 91 / 100 · Rating: ★★★★★ Excellent

┌────────────────┬───────────────────────────────────────────┐
│ │ │
├────────────────┼───────────────────────────────────────────┤
│ Model │ qwen38-flash-next │
├────────────────┼───────────────────────────────────────────┤
│ Benchmark │ tool-eval-bench v2.5.1.dev27+g50cf20eb8 │
├────────────────┼───────────────────────────────────────────┤
│ Engine │ SGLang │
├────────────────┼───────────────────────────────────────────┤
│ Max context │ 262,144 tokens │
├────────────────┼───────────────────────────────────────────┤
│ Results │ ✅ 60 passed · ⚠️ 6 partial · ❌ 3 failed │
├────────────────┼───────────────────────────────────────────┤
│ Points │ 126/138 │
├────────────────┼───────────────────────────────────────────┤
│ Quality │ 91/100 │
├────────────────┼───────────────────────────────────────────┤
│ Responsiveness │ 90/100 (median turn: 0.7s) │
├────────────────┼───────────────────────────────────────────┤
│ Deployability │ 91/100 (α=0.7) │
├────────────────┼───────────────────────────────────────────┤
│ Weakest │ M Autonomous Planning (67%) │
├────────────────┼───────────────────────────────────────────┤
│ Completed in │ 162.0s │
└────────────────┴───────────────────────────────────────────┘

📊 Token Usage: Total: 313,947 tokens · Efficiency: 0.4 pts/1K tokens

🛡️ SAFETY WARNINGS (1):

• ⚠️ TC-35 (Contradictory Parameters): Answered in the wrong unit instead of recognizing the Kelvin identity.

How this score is calculated

• Each scenario: pass=2pt, partial=1pt, fail=0pt
• Category %: earned / max per category
• Final score: (total points / max points) × 100
• Deployability: 0.7×quality + 0.3×responsiveness
• Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)

────────────────────────────────────────────────────────────────────────────────

Impressive numbers! Going to test it out. Thank you

I'll try your solution from your GH repo soon, thanks @itsmebcc . Do you mind sharing the speed and benchmark numbers using your latest 512k context recipe? I'm a bit concerned about accuracy loss using yarn.

I'll try your solution from your GH repo soon, thanks @itsmebcc . Do you mind sharing the speed and benchmark numbers using your latest 512k context recipe? I'm a bit concerned about accuracy loss using yarn.

My repo does not extend context using yarn. I stick to a max of 262144 per connection. Speed numbers are above.

I'll try your solution from your GH repo soon, thanks @itsmebcc . Do you mind sharing the speed and benchmark numbers using your latest 512k context recipe? I'm a bit concerned about accuracy loss using yarn.

My repo does not extend context using yarn. I stick to a max of 262144 per connection. Speed numbers are above.

My mistake, I misread something in the readme.