mlx-community/Ornith-1.5-35B-A3B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants ยท Docs

OptiQ mixed-precision quant of ornith-ai/Ornith-1.5-35B-A3B, a 35B mixture-of-experts vision-language model with 3B active parameters and a bundled MTP speculation head. 24 GB on disk.

What it is

Property Value
Base ornith-ai/Ornith-1.5-35B-A3B (Qwen3.5-MoE, 35B total / 3B active)
Method OptiQ mixed-precision, per-layer 4/8-bit
Bit allocation Reused from the Ornith-1.0-35B OptiQ recipe: the architecture is identical, so the per-layer sensitivity ranking transfers directly and no per-model sweep is needed
Layer split 113 components at 4-bit, 399 at 8-bit
Group size 64
On disk 24 GB
MTP Speculation head preserved in optiq/mtp.safetensors
Vision bf16 vision tower kept in optiq/optiq_vision.safetensors for image input

Following the naming llama.cpp uses for its mixed quants, the "4bit" label denotes the family, not the weighted average.

Run it

The MoE arch and the MTP/vision sidecars register through OptiQ, so import optiq once before loading:

pip install "mlx-optiq>=0.4.27"
import optiq  # registers the arch + MTP/vision sidecars
from mlx_lm import load, generate

model, tok = load("mlx-community/Ornith-1.5-35B-A3B-OptiQ-4bit")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain mixture-of-experts routing in two sentences."}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=400))

For image input plus an OpenAI- and Anthropic-compatible endpoint with mixed-precision KV cache:

optiq serve --model mlx-community/Ornith-1.5-35B-A3B-OptiQ-4bit

This is a reasoning model, so give it a generous token budget.

Using the head outside optiq serve: optiq/mtp.safetensors holds Qwen's original MTP tensors, so its seven RMSNorm weights (mtp.*norm*) are zero-centred, applied as 1 + w. The main weights have the +1 folded in, as MLX expects. Add 1.0 to all seven norms when loading the head into another runtime.

Links

Downloads last month
5,341
Safetensors
Model size
35B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlx-community/Ornith-1.5-35B-A3B-OptiQ-4bit

Quantized
(222)
this model
Quantizations
1 model