Code-Writer-V2-Obliterated AWQ Mixed (4-bit)

AWQ 4-bit mixed-precision quant of groxaxo/Code-Writer-V2-Obliterated-BF16 optimized for vLLM serving on 2× RTX 3090 (24 GB each, compute capability 8.6).

This is the format that actually serves on vLLM for the qwen3_5 hybrid-attention architecture. The AutoRound-native (auto_round / GPTQ-packing) format produces degenerate output (only token-id-0 / !) under vLLM 0.23's loader; the same checkpoints generate correctly in plain transformers. AWQ is the only format vLLM serves correctly for this architecture. Full build log and reproduction recipe: github.com/groxaxo/codewriter-v2-awq-quant-guide.

Quantization recipe

Field	Value
Format	`awq` (gemm), `zero_point=True` (asymmetric)
Bits / group	4 / 128
Algorithm	AWQ activation-aware smoothing + RTN, calibration-free (`--model_free`)
Tool	`auto-round 0.13.1` → `--format auto_awq`
Modules kept in BF16	vision tower, all GDN `linear_attn.in_proj_{qkv,z,a,b}`, full-attn `self_attn.{q,k,v}_proj`, layer 0, `lm_head`

Benchmark

Metric	This model	Published AutoRound-W4A16-RTN baseline
PIQA acc (n=500)	0.814 ±0.0174	0.8074
PIQA acc_norm (n=500)	0.832 ±0.0167	—

Beats the existing RTN baseline on the directly-comparable benchmark despite being calibration-free (the AWQ activation-aware smoothing does real work).

Serving (vLLM 0.23, the exact command that loads)

vllm serve groxaxo/Code-Writer-V2-Obliterated-AWQ-Mixed-Lean \
  --served-model-name codewriter-v2-awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --disable-custom-all-reduce \
  --dtype float16

The three non-obvious flags are required on RTX 3090:

--dtype float16 — vLLM rejects bfloat16 for AWQ
--disable-custom-all-reduce — the custom all-reduce kernel crashes on cc 8.6
--max-num-seqs 1 + small --max-num-batched-tokens — default profile OOMs during CUDA-graph capture

VRAM at idle: ~21 GB / GPU. Throughput at --max-num-seqs 1 is conservative; raise it if you have KV-cache headroom.

Architecture (verified from config)

model_type: qwen3_5, Qwen3_5ForConditionalGeneration (VLM — vision tower present)
64 layers: 16 full-attention (every 4th) + 48 GDN/linear-attention
num_key_value_heads = 4 → tensor-parallel must be 2 or 4, never 3
No MTP head → never pass --speculative-config
Text model path: model.language_model.layers.N.*