debackerl/elephant-1-Q8_0-GGUF

elephant-1 — Q8_0, decision-wired GGUF

baselabs/elephant-1 (9B decision model, Qwen3.5-9B post-train, vision) in Q8_0, with the llama.cpp decision metadata and a systemone prompt template so that POST /v1/systemone works.

• Base model: baselabs/elephant-1 — MIT, Base Labs at Baseten.
• Q8_0 weights and Q8_0 vision tower: mradermacher/elephant-1-GGUF.
• This repo adds metadata/template only. No weights were changed or re-quantized.
• Used scripts from scripts from waseigo/elephant-1-decision-GGUF, MIT license.

Why this repo exists

A plain GGUF conversion of elephant-1 is a qwen35 text model. llama.cpp/llama-swap's /v1/systemone then answers 501 "This model is not a decision model", because the file has no <arch>.decision.* keys and no named systemone template. The decision readout is headless — qev's slot_lm falls back to the LM logits of the slot labels — which llama.cpp already implements as OPENJEV.

This build adds:

Key	Value
qwen35.decision.type	openjev
qwen35.decision.temperature.choice	1.0
qwen35.decision.temperature.score	1.0
qwen35.decision.temperature.noul	1.0
qwen35.attention.causal	false
tokenizer.chat_template.systemone	qev prompt template (see systemone.jinja)

The template reproduces qev's qev/prompt.py layout byte-for-byte for choice, noul and score, wrapped in the base Qwen chat template's markers. Verify with test-chat-template --no-common; the choice/score/noul renders are identical to the transformers apply_chat_template output that qev feeds the model.

Two template details matter and are easy to get wrong:

1. Generate the labels in the template. OPENJEV reads the logits of the label tokens (A–Z, a–z) but does not pass a label to the template. Use {% set letters = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz' %} and {{ letters[loop.index0] }}.
2. Emit newlines explicitly. transformers' Jinja (trim_blocks=True) and llama.cpp's minja trim differently. A literal newline after a block tag disappears in minja, which concatenates option lines. Use {{ '\n' }}.

Files

File	Size	SHA-256
elephant-1-Q8_0.gguf	9,527,504,224 B	652c102967efa9a1adf9c63ebaada547fba189888ca0680944aae595398fdf9d
elephant-1.mmproj-Q8_0.gguf	624,229,760 B	6906d682cc8ca0e353c39e4f9926c9a02c7770fd2f367ace5cb08c5b2cc48603
systemone.jinja	1,479 B	the template above
graft_decision_metadata.py	—	adds the decision keys + template to a plain GGUF
graft_noncausal.py	—	adds qwen35.attention.causal = false

Usage

Build with decision support (llama.cpp b11820 or newer):

llama-server \
  -m elephant-1-Q8_0.gguf \
  --mmproj elephant-1.mmproj-Q8_0.gguf \
  --image-min-tokens 1024 -c 32768 -fa on --jinja \
  --host 127.0.0.1 --port 8999

The log must say decision model type: openjev.

curl -s http://127.0.0.1:8999/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {"billing": "payments and refunds", "shipping": null, "technical": null}
    },
    "urgency": {
      "type": "score",
      "instructions": "How urgent is this?",
      "criteria": ["can wait", "this week", "today", "right now"]
    },
    "angry": {"type": "noul", "instructions": "Is the customer angry?"}
  }
}'

Caveats

• Q8_0: quantized by mradermacher. Quality tracks that quant, not BF16.
• Attention mask: the base declares full_attention: block_bidirectional. llama.cpp has no per-block mask key, so attention.causal = false makes the full-attention layers fully noncausal. Because llama.cpp evaluates the shared prefix on its own and each question suffix afterwards, this reproduces qev's two-block mask (prefix block, suffix block). A prompt that is evaluated as one unsplit batch would not have the same semantics.
• Label space: OPENJEV reads 52 single-token labels (A–Z, a–z). qev's slot_lm defines up to 256 labels (A–Z plus two-letter codes). Up to 26 options is the validated path; larger option sets were not validated.
• Temperature: 1.0. The base repo has no calibration.json; qev's calibration defaults to identity.
• JSON states: a state object is rendered with tojson(indent=2), matching qev's json.dumps(..., indent=2). Non-ASCII escaping may differ from Python's ensure_ascii=False.

Credits and license

• Model: Base Labs at Baseten, baselabs/elephant-1, MIT.
• Quant and vision tower: mradermacher, mradermacher/elephant-1-GGUF.
• Prompt/readout reference: OneJev qev (qev/prompt.py, serve.patch).
• llama.cpp decision models: ggml-org, PR #29818 and follow-ups.

The graft and this packaging are released under the base model's MIT terms. No warranty.