Card: int8 file rebuilt with the models/minicpm5 README command (dynamic_wi8_afp32)

Thanks for the go-ahead on adopting the README-command int8 (the 09-12 thread). This PR replaces `MiniCPM5-2B_int8.litertlm` with the bundle the LiteRT-LM `models/minicpm5/README.md` command produces with `--quantization_recipe=dynamic_wi8_afp32` on the released litert-torch 0.9.4 (2,546,937,984 bytes, sha256 61dafe88…), and rewrites the card and `litertlm_manifest.json` so they describe that file: the canonical template (direct answers unless the app turns thinking on), one 128-token prefill signature, in-graph int8 embedding and lm_head, and the rows measured on it. The int4 file is unchanged (blockwise-32 with the in-place epsilon on its 13 dead rows; the payload under those scales is inert — GSM8K 87 with the same misses on a payload-zeroed copy).

What the new file measured: GSM8K thinking-off 93 (bf16 92) on the Mac GPU; the 8-question gate 6/8 at its no-think default (the bf16 model's own thinking-off score) and 8/8 with thinking on, both backends; Mac M4 Max 2084 / 124.2 tok/s on the GPU and 175 / 37.1 on the CPU; Galaxy S26 full OpenCL delegation (1836/1836 + 1655/1655) with 542–549 / 22.4–22.6 tok/s on the GPU and 93–132 / 15.1–18.2 on the CPU (v0.16.0 kit, cooled and uncapped before each run). The previous int8's fp32-activation declaration is not needed here: this file passes the thinking-on gate on the GPU with the runtime's default fp16 activations.

PR #4 (the iOS entitlement wording and the per-signature node counts) is folded in; if this one lands I will close #4. If any of it does not fit the card, close it and I will carry it elsewhere. Thanks.

Links — the conversion command: https://github.com/google-ai-edge/LiteRT-LM/blob/main/models/minicpm5/README.md#model-conversion · the pbtext the file was built with: https://github.com/google-ai-edge/LiteRT-LM/blob/b5e34ab1/models/minicpm5/LlmMetadataProto.pbtext · the runtime: https://github.com/google-ai-edge/litert

cc @Yoursmiling

MiniCPM5-2B_int8.litertlm

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:58da116a6eb563185eb5e8d0b1f1e1116b632a516cadc0285b6da35b9f886a86
-size 2603278256
+oid sha256:61dafe884025d29019302ef27ffb8c677c13eab5c2eb5efd34873ff7cb35c36e
+size 2546937984

README.md

@@ -25,17 +25,17 @@ This repository hosts the [**LiteRT-LM**](https://ai.google.dev/edge/litert-lm)
 
 ## Available Models
 
-Recommend to use below both **CPU and GPU compatible** models, originally from [mlboydaisuke/MiniCPM5-2B-LiteRT](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-LiteRT)
-**Requires litert-lm ≥ 0.16** (thought channel + `ThinkingConfig`); measured here on 0.17.0.
+Recommend to use below both **CPU and GPU compatible** models. The int8 file is built with the LiteRT-LM [`models/minicpm5` conversion command](https://github.com/google-ai-edge/LiteRT-LM/blob/main/models/minicpm5/README.md#model-conversion) (`--quantization_recipe=dynamic_wi8_afp32`); the int4 file is originally from [mlboydaisuke/MiniCPM5-2B-LiteRT](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-LiteRT).
+**Requires litert-lm ≥ 0.16** (thought channel + `ThinkingConfig`); measured here on litert-lm 0.17.0 (Mac) and the v0.16.0 release kit (Galaxy S26).
 
 | File | Recipe | Size |
 |---|---|---|
 | **`MiniCPM5-2B_int4.litertlm`** | int4 blockwise-32 + OCTAV on linears, int8 embedding | 1.55 GB |
-| **`MiniCPM5-2B_int8.litertlm`** | int8 dynamic on linears + embedding; **fp32 activations declared** (see notes) | 2.60 GB |
+| **`MiniCPM5-2B_int8.litertlm`** | int8 dynamic (channelwise) on linears, embedding and lm_head — `dynamic_wi8_afp32` | 2.55 GB |
 
-The **int4 file is the phone file** (smaller, fastest GPU decode on every platform measured) — best used for direct answers or short reasoning; see the thinking-mode note below. **int8 is the file for reasoning that has to complete**: its thinking chains are ~3–4× shorter than int4's on the same questions and terminate where int4 runs into the token budget. int8's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; an app that carries the `com.apple.developer.kernel.increased-memory-limit` entitlement lifts that limit (not checked with this file).
+The **int4 file is the phone file** (smaller, fastest GPU decode on every platform measured) — best used for direct answers or short reasoning; see the thinking-mode note below. **int8 is the file for reasoning that has to complete**: its thinking chains are ~3–4× shorter than int4's on the same questions and terminate where int4 runs into the token budget. int8's main weight section is 2.54 GB, more than an iOS app maps in one piece with the default entitlements; an app that carries the `com.apple.developer.kernel.increased-memory-limit` entitlement lifts that limit (not checked with this file).
 
-Both files carry the checkpoint's own `chat_template.jinja` byte for byte and declare the `thought` channel, so `enable_thinking`, the tool-calling format, and the runtime's `ThinkingConfig` all work unchanged (details under *Conversion notes*).
+Both files declare the `thought` channel, so the runtime's `ThinkingConfig` and the reasoning/answer split work on both. They carry **different chat templates**, and that changes the thinking default: the int4 file embeds the checkpoint's own `chat_template.jinja` (the model decides, and in practice thinks); the int8 file embeds the LiteRT-LM [`models/minicpm5/chat_template.jinja`](https://github.com/google-ai-edge/LiteRT-LM/blob/b5e34ab1/models/minicpm5/chat_template.jinja) as of 2026-09-01 (`enable_thinking` defaults to **false**: direct answers unless the app turns thinking on). Details under *Controlling thinking* and *Conversion notes*.
 
 Further, below are some CPU-only models for exploration:
 * `minicpm_wi4c_wi8_afp32.litertlm`: This model features mixed INT4/INT8 weight-only quantization with FP32 activations (afp32). MLP projections use channelwise INT4 with Hadamard rotation; all remaining weights (attention, embedding, and lmhead) use channelwise INT8.
@@ -95,47 +95,52 @@ litert-lm run ./MiniCPM5-2B_int4.litertlm --backend gpu --cache no --prompt "...
 
 ### Controlling thinking
 
-Thinking is the model's **default**: with no `ThinkingConfig` it decides for itself and, in practice, reasons before every answer (100–700 characters on trivial questions, thousands on math). To control it:
+The two files start from opposite defaults, because they carry different templates (see *Conversion notes*):
 
-- **Give it a generous output budget (≥ 2048 tokens; 4096 for math).** Truncated mid-thought it produces no final answer at all.
-- The bundle declares the `thought` channel (`<think>` / `</think>`), so the streamed text contains only the answer, and runtimes that expose `ThinkingConfig` can cap or read the reasoning separately.
-- `enable_thinking=false` (via `ThinkingConfig` or the conversation's extra context — both reach the template) switches the model to **direct answers**: two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers below. `enable_thinking=true` pre-fills the think opener explicitly. On the CLI: `--thinking false` / `--thinking-budget N`.
+- **int4** (the checkpoint's own template): thinking is the model's **default** — with no `ThinkingConfig` it decides for itself and, in practice, reasons before every answer (100–700 characters on trivial questions, thousands on math).
+- **int8** (the LiteRT-LM canonical template): with nothing set the template closes the think block itself, so the model gives **direct answers**; `enable_thinking=true` turns the reasoning on.
+
+For both files:
+
+- **Give a reasoning run a generous output budget (≥ 2048 tokens; 4096 for math).** Truncated mid-thought it produces no final answer at all.
+- Both bundles declare the `thought` channel (`<think>` / `</think>`), so the streamed text contains only the answer, and runtimes that expose `ThinkingConfig` can cap or read the reasoning separately.
+- `enable_thinking` set via `ThinkingConfig` or the conversation's extra context reaches either template: `false` switches to direct answers (two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers below); `true` pre-fills the think opener explicitly. On the CLI: `--thinking true|false` / `--thinking-budget N`.
 - Sampling: OpenBMB recommends `temperature 1.0, top_p 0.95`; the gates below are greedy.
-- Prompt format is ChatML (`<|im_start|>role\n…<|im_end|>\n`), 4096-token KV budget, six prefill signatures (1024, 256, 64, 16, 4, 1).
+- Prompt format is ChatML (`<|im_start|>role\n…<|im_end|>\n`), 4096-token KV budget. Prefill signatures: int4 six (1024, 256, 64, 16, 4, 1); int8 one (128 — a longer prompt is prefilled in 128-token chunks).
 
 ## Correctness
 
-Both files score **8/8 on an 8-question sanity gate on both CPU and GPU** (Apple M4 Max), with the reasoning arriving on the thought channel and only the final answer in the streamed text. The bf16 PyTorch model scores 8/8 on the same gate with thinking on or left to the model, and **6/8 with thinking forced off** ("opposite of hot" → "Cool.", the rhyme line → "Green.") — read any thinking-off result against that, not against 8/8.
+On an 8-question sanity gate (Apple M4 Max, greedy, CPU and GPU) the **int4 file scores 8/8 at its default** (the model thinks; the reasoning arrives on the thought channel, only the answer in the streamed text) and the **int8 file scores 6/8 at its default** (direct answers; measured with nothing set on the GPU and with thinking off on both backends) and **8/8 with thinking on**, on both backends. The two thinking-off misses are the bf16 PyTorch model's own: it scores 8/8 with thinking on or left to the model and **6/8 with thinking forced off** ("opposite of hot" → "Cool.", the rhyme line → "Green."), so read any thinking-off result against that, not against 8/8.
 
 **GSM8K** (first 100 test questions, greedy, 0-shot chain-of-thought prompt, **thinking off** — the protocol OpenBMB's own MiniCPM5 cards use, max 2048 new tokens, identical prompt and extraction on every row):
 
 | Configuration | GSM8K |
 |---|---|
 | bf16 PyTorch (MPS), upstream template | **92 %** |
-| int8, CPU | **91 %** |
+| int8, GPU (Metal, WebGPU delegate) | **93 %** |
 | int4, CPU | 86 % |
 | int4, GPU (Metal) | 87 % |
 
-int8 is at parity (7 of its 9 misses are the bf16 model's own). int4 costs about five points on this 42-layer model; the GPU's default fp16 activations cost nothing measurable in no-think mode.
+int8 is at parity: 93 vs 92 is inside one standard error at n=100 (about ±3 points), and six of its seven misses are the bf16 model's own; the seventh (question 8) is a derivation that runs to the 2048-token budget mid-calculation on the GPU, so the extracted number is wrong, while the CPU run on the same file answers it in 2,204 characters. int4 costs about five points on this 42-layer model; the GPU's default fp16 activations cost nothing measurable in no-think mode.
 
-**Thinking mode is where int4 shows its damage.** On ten GSM8K questions with thinking on and a 3584-token budget, the bf16 model closes its reasoning on 9/10 with ~3,000-character chains and int8 reproduces that question for question (9/10, median ~3,200 characters on CPU), while **int4 closes 0/10 on CPU** (median ~13,700 characters — it keeps re-checking and runs into the budget; the answer is usually right inside the thought text but never gets emitted). On the Metal GPU with the runtime's default fp16 activations the same int4 file happened to close 10/10 with ~8,000-character chains, but that is fp16 rounding steering the trajectory, not a property to rely on. If your use needs the reasoning to finish, use int8 or turn thinking off on int4.
+**Thinking mode is where int4 shows its damage.** On ten GSM8K questions with thinking on and a 3584-token budget, the bf16 model closes its reasoning on 9/10 with ~3,000-character chains, and int8 closes **10/10 with ~3,900-character chains (10/10 correct; Metal GPU through the WebGPU delegate, runtime-default fp16 activations)**, while **int4 closes 0/10 on CPU** (median ~13,700 characters — it keeps re-checking and runs into the budget; the answer is usually right inside the thought text but never gets emitted). On the Metal GPU with the runtime's default fp16 activations the same int4 file happened to close 10/10 with ~8,000-character chains, but that is fp16 rounding steering the trajectory, not a property to rely on. If your use needs the reasoning to finish, use int8 or turn thinking off on int4.
 
-On a **Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno)** both files generate correctly on GPU and CPU with **full OpenCL delegation — every node of every signature (1873/1873 on the 1024- to 4-token prefills, 1664/1664 on the 1-token prefill, 1692/1692 on decode), zero rejected ops** — and the runtime separates the reasoning on-device (`[thought] … [/thought]`, then the answer).
+On a **Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno)** both files generate correctly on GPU and CPU; on the GPU with **full OpenCL delegation — every node of every prefill and decode signature, zero rejected ops** (int4: 1873/1873 on the 1024- to 4-token prefills, 1664/1664 on the 1-token prefill, 1692/1692 on decode; int8: 1836/1836 on the 128-token prefill, 1655/1655 on decode) — and the runtime separates the reasoning on-device (`[thought] … [/thought]`, then the answer).
 
 On an **iPhone 17 Pro** the int4 file passes the same 8-question gate on both backends (on-device byte count verified against the source file): **Metal GPU 7/8** (init 5.7 s) and **CPU 7/8** (init 2.2 s); the one miss on each leg is the rhyme line inside the 8-question composite prompt, answered "green" — the same answer the bf16 model gives with thinking off, so a prompt-format artifact rather than conversion damage.
 
-Multi-turn: three-turn conversations (introduce a name and a city, an arithmetic question, then "which city do I live in?") hold on both files under all three thinking modes with the name and city recalled — the template's history rendering stays consistent with what the runtime already streamed.
+Multi-turn: three-turn conversations (introduce a name and a city, an arithmetic question, then "which city do I live in?") hold on both files under all three thinking modes with the name and city recalled — each template's history rendering stays consistent with what the runtime already streamed.
 
 ## Performance
 
-`litert-lm benchmark` (litert-lm 0.17.0), Apple M4 Max, `-p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024`, quiet machine, serialized, ≥300 s rest before each GPU reading; each backend confirmed to generate real text before its number was recorded:
+`litert-lm benchmark` (litert-lm 0.17.0), Apple M4 Max, `-p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024`, quiet machine, serialized, ≥300 s rest before each GPU reading; each backend confirmed to generate real text before its number was recorded; all four rows from one session (2026-09-13):
 
 | File | Backend | Prefill (256) | Decode | TTFT | Init |
 |---|---|---|---|---|---|
-| int4 | **GPU (Metal)** | **1699 tok/s** | **92.8 tok/s** | 0.16 s | 3.7 s |
-| int4 | CPU | 149 tok/s | 31.1 tok/s | 1.76 s | 4.5 s |
-| int8 (fp32 activations) | GPU (Metal) | 1405 tok/s | 74.7 tok/s | 0.20 s | 3.0 s |
-| int8 (fp32 activations) | CPU | 161 tok/s | 30.0 tok/s | 1.62 s | 15.0 s |
+| int4 | **GPU (Metal)** | **1836 tok/s** | **93.0 tok/s** | 0.15 s | 3.8 s |
+| int4 | CPU | 148 tok/s | 31.4 tok/s | 1.77 s | 4.4 s |
+| int8 | **GPU (Metal)** | **2084 tok/s** | **124.2 tok/s** | 0.13 s | 1.8 s |
+| int8 | CPU | 175 tok/s | 37.1 tok/s | 1.49 s | 4.7 s |
 
 Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; `litert_lm_advanced_main` from the litert-lm v0.16.0 release kit, 205-token prompt with `--benchmark`, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):
 
@@ -143,21 +148,27 @@ Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; `litert_lm_advanced_main` from
 |---|---|---|---|---|---|---|
 | int4 | **GPU (OpenCL)** | **401–411 tok/s** | **16.1–18.6 tok/s** | 0.56 s | 11.2–13.1 s | 1.14 GB |
 | int4 | CPU | 39–72 tok/s | 15.6–15.8 tok/s | 2.9–5.3 s | 3.1–5.7 s | 2.12 GB |
-| int8 (fp32 activations) | GPU (OpenCL) | 150–160 tok/s | 10.9–12.8 tok/s | 1.4 s | 4.0–6.3 s | 1.10 GB |
-| int8 (fp32 activations) | CPU | 103–157 tok/s | 11.7 tok/s | 1.4–2.1 s | 0.3 s | 2.90 GB |
+| int8 | GPU (OpenCL) | 542–549 tok/s | 22.4–22.6 tok/s | 0.42–0.43 s | 5.6–6.4 s | 0.82 GB |
+| int8 | CPU | 93–132 tok/s | 15.1–18.2 tok/s | 1.6–2.3 s | 3.6–4.2 s | 3.07 GB |
 
-GPU wins prefill (5–10× on the phone, 11× on the Mac) and time-to-first-token everywhere; on Adreno the int4 GPU decode edge over the same-device CPU is modest (~1.1×), while the int8 file's fp32 activations bring its GPU decode level with its CPU.
+The int8 rows are the README-command build (one 128-token prefill signature, `odml.cache_update` composites, runtime-default fp16 GPU activations) and are not a like-for-like update of the previous int8 rows on this card (fp32 activations declared, six signatures). GPU wins prefill (4–10× on the phone, 11–12× on the Mac) and time-to-first-token everywhere; on Adreno the int4 GPU decode edge over the same-device CPU is modest (~1.1×), while the int8 file decodes 1.2–1.5× faster on the GPU than on its CPU at about a quarter of the process memory. The int8 phone rows were taken at 37.9 °C skin temperature with no CPU frequency cap in force before each run.
 
 ## Conversion notes
 
-Converted with [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1 from a pristine released stack (reproduction: [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm), `bash scripts/reproduce_llm.sh minicpm5-2b`). Four things make these files run on both backends:
+The two files come from two different conversion paths.
+
+**int8 — the LiteRT-LM `models/minicpm5` command, as written.** `litert-torch export_hf` (litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, litert-lm-builder 0.16.1, transformers 5.14.1 — the released wheels) with `--quantization_recipe=dynamic_wi8_afp32 --use_bool_mask=True --apply_gpu_composites=True --cache_length=32771 --experimental_lightweight_conversion` and `--litert_lm_llm_metadata_override` pointing at the checked-in [`LlmMetadataProto.pbtext`](https://github.com/google-ai-edge/LiteRT-LM/blob/b5e34ab1/models/minicpm5/LlmMetadataProto.pbtext) as of 2026-09-01 (commit `b5e34ab1`; the file on `main` moved on 2026-09-11 to a template that takes message content as parts only); only the model id (`openbmb/MiniCPM5-2B`), the metadata path and the output directory differ from the [README block](https://github.com/google-ai-edge/LiteRT-LM/blob/main/models/minicpm5/README.md#model-conversion). What that produces:
+
+- Weights: int8 per-channel on every linear including the 130560×2048 embedding and lm_head, all in the single prefill/decode graph (no separate embedder section); activations at the runtime default. Graph: `prefill_128` 1836 ops / `decode` 1655 ops, `odml.cache_update` composites, boolean mask; no custom op.
+- Metadata = the pbtext: start token `<s>`, stops `</s>` and `<|im_end|>`, `max_num_tokens 4096`, `llm_model_type minicpm5`, the `thought` channel as `<think>\n` / `</think>`, and the LiteRT-LM canonical `chat_template.jinja` (ChatML, tool definitions in a `<tools>` block, `enable_thinking | default(false)`). `--cache_length=32771` is the runtime's magic number: the KV cache is sized from the metadata's 4096 at load.
+- Tokenizer: the checkpoint's `tokenizer.json` (byte-level BPE) as an HF tokenizer section.
+
+**int4 — a blockwise recipe with two pieces of handling** (reproduction: [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm), `bash scripts/reproduce_llm.sh minicpm5-2b`; litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1):
 
 - **The chat template is the checkpoint's `chat_template.jinja`, byte for byte**, embedded on the runtime's jinja path — the same packaging as [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). That is what keeps `enable_thinking` and the tool-calling format available to the app; the `thought` channel is declared alongside it.
 - **Start token `<s>`** is correct for this family: the template's own `{{ bos_token }}` renders empty at runtime and the engine prepends the metadata start token, so the model sees exactly one `<s>` as it does upstream. Stops are the model's `</s>` and `<|im_end|>`.
-- **int4 needed a zero-scale fix.** Decoder layer 0's MLP contains 13 all-zero rows; blockwise quantization emits a zero scale for each of their blocks, which the CPU (XNNPACK) path refuses to load (`unsupported scale value (0.000000) … for INT4 tensor`) while the GPU path silently accepts. The scales were replaced in place by a tiny positive epsilon (the quantized values in those blocks are zero, so the dequantized weights are unchanged) — 3,328 bytes of a 1.55 GB file.
-- **int8 declares fp32 activations in-bundle** (`prefer_activation_type = "fp32"` in the bundle's `model.toml`; weights untouched). With the runtime's default fp16 GPU activations, the int8 model's reasoning on one gate question ran 2000+ tokens without closing `</think>` (it fails the 8-question gate); with fp32 declared it closes in ~450 tokens and passes 8/8, at a cost of ~14 % GPU decode speed. On a ten-question thinking-on GSM8K subset the two dtypes were closer (fp16 finished 10/10, fp32 7/10), so this is a measured trade for the gate, not a cure. The int4 file keeps the fp16 default: it passes the gate there, and declaring fp32 only reproduces the CPU reference's non-terminating chains (see Correctness).
-
-Also: the 130560×2048 input-embedding table sits in its own section (embedder externalised); the tokenizer is embedded as the upstream `tokenizer.json` (byte-level BPE survives intact); int4 is blockwise-32 with OCTAV clipping on the linears and int8 on the embedding, int8 is dynamic per-channel on linears and embedding. A block-128 int4 variant was built and rejected: it loses 8 GSM8K points (79 vs 87, thinking off), fails the 8-question gate on CPU, and does not shorten the thinking chains.
+- **int4 needed a zero-scale fix.** Decoder layer 0's MLP contains 13 dead rows (every weight below 3e-23); blockwise quantization emits a zero scale for each of their blocks, which the CPU (XNNPACK) path refuses to load (`unsupported scale value (0.000000) … for INT4 tensor`) while the GPU path silently accepts. The scales were replaced in place by a tiny positive epsilon — 3,328 bytes of a 1.55 GB file.
+- The 130560×2048 input-embedding table sits in its own section (embedder externalised, int8); linears are blockwise-32 int4 with OCTAV clipping; GPU activations at the runtime default (fp16). A block-128 int4 variant was built and rejected: it loses 8 GSM8K points (79 vs 87, thinking off), fails the 8-question gate on CPU, and does not shorten the thinking chains.
 
 ## Machine-readable manifest
 
@@ -168,7 +179,7 @@ Also: the 130560×2048 input-embedding table sits in its own section (embedder e
 - 🤗 Original model (BF16): [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B)
 - 🤗 Smaller sibling: [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B)
 - 📦 GitHub: [OpenBMB/MiniCPM](https://github.com/OpenBMB/MiniCPM)
-- 🛠️ LiteRT docs: [ai.google.dev/edge/litert](https://ai.google.dev/edge/litert)
+- 🛠️ LiteRT docs: [ai.google.dev/edge/litert](https://ai.google.dev/edge/litert) · runtime source: [github.com/google-ai-edge/litert](https://github.com/google-ai-edge/litert) / [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM)
 
 ---
 
@@ -176,7 +187,7 @@ Also: the 130560×2048 input-embedding table sits in its own section (embedder e
 
 Released under the **Apache-2.0 License**, consistent with the upstream [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B).
 
-**Changes from the original work** (for the two recommended files): weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; 13 all-zero weight rows' quantization scales set to an epsilon (no numeric change); tokenizer and chat template repackaged into the `.litertlm` bundle with runtime metadata (thought channel, stop tokens, activation-dtype preference). No fine-tuning.
+**Changes from the original work** (for the two recommended files): weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; in the int4 file, 13 dead weight rows' quantization scales set to an epsilon (measured inert: GSM8K 87 with the identical miss list on a copy whose payload under those scales is zeroed); tokenizer and chat template repackaged into the `.litertlm` bundle with runtime metadata (thought channel, stop tokens; the int8 file carries the LiteRT-LM canonical template instead of the checkpoint's). No fine-tuning.
 
 ## Citation
 

litertlm_manifest.json

@@ -1,12 +1,12 @@
 {
   "manifest_schema": "0.1.2",
   "repo": "litert-community/MiniCPM5-2B",
-  "generated": "2026-09-08",
+  "generated": "2026-09-13",
   "generator": "make_manifest.py",
   "model": {
     "display_name": "MiniCPM5-2B",
     "base_model": "openbmb/MiniCPM5-2B",
-    "architecture": "Dense LlamaForCausalLM (42L, hidden 2048, GQA 16:2, untied 130560-vocab embeddings, rope_theta 5e6, no rope scaling), 2.52B params; hybrid-reasoning chat template (enable_thinking) carried verbatim with a declared thought channel",
+    "architecture": "Dense LlamaForCausalLM (42L, hidden 2048, GQA 16:2, untied 130560-vocab embeddings, rope_theta 5e6, no rope scaling), 2.52B params; hybrid-reasoning chat template (enable_thinking) with a declared thought channel (the int4 file carries the checkpoint template verbatim, the int8 file the LiteRT-LM canonical one)",
     "parameters_b": 2.52,
     "license": "apache-2.0",
     "context_length": 4096,
@@ -70,7 +70,7 @@
         {
           "platform": "macos",
           "backend": "gpu",
-          "reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no, generation gated per backend first: gpu 1699 tok/s prefill / 92.8 tok/s decode vs cpu 149 / 31.1"
+          "reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no, generation gated per backend first: gpu 1836 tok/s prefill / 93.0 tok/s decode vs cpu 148 / 31.4 (2026-09-13; the 2026-09-08 readings were 1699 / 92.8 vs 149 / 31.1)"
         },
         {
           "platform": "ios",
@@ -84,7 +84,7 @@
           "Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0",
           "Hybrid-reasoning model: with no ThinkingConfig it thinks before every answer; budget max output tokens >= 2048 (4096 for math). enable_thinking=false (ThinkingConfig or conversation extra_context) switches to direct answers",
           "Thinking-mode caveat measured on this file: on GSM8K-class questions its reasoning chains run ~4x longer than int8's and do not close inside 3584 tokens on the CPU reference (0/10 closed; int8 9/10, bf16 9/10). Use int8 when the reasoning must complete, or turn thinking off",
-          "iPhone 17 Pro: passes the on-device gate on gpu and cpu (init 5.7 s / 2.2 s); the int8 file's 2.33 GB main section needs the com.apple.developer.kernel.increased-memory-limit entitlement on iOS (not checked with that file)",
+          "iPhone 17 Pro: passes the on-device gate on gpu and cpu (init 5.7 s / 2.2 s); the int8 file's 2.54 GB main section needs the com.apple.developer.kernel.increased-memory-limit entitlement on iOS (not checked with that file)",
           "The 13 all-zero MLP rows in decoder layer 0 make raw blockwise-int4 exports fail to load on XNNPACK (scale 0); this file carries the epsilon fix"
         ]
       },
@@ -96,15 +96,15 @@
           "runtime": "litert-lm 0.17.0",
           "prompt_tokens": 256,
           "decode_tokens": 256,
-          "prefill_tps": 148.58,
-          "decode_tps": 31.12,
-          "ttft_s": 1.76,
-          "load_s": 4.52,
+          "prefill_tps": 147.65,
+          "decode_tps": 31.38,
+          "ttft_s": 1.77,
+          "load_s": 4.44,
           "max_num_tokens": 1024,
           "cache": "no",
           "runs": 3,
-          "date": "2026-09-08",
-          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, backend gated on a real generation first"
+          "date": "2026-09-13",
+          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, CPU cells first, backend gated on a real generation first (same session as the int8 rows; the 2026-09-08 reading was 148.9 / 31.12)"
         },
         {
           "device": "Apple M4 Max",
@@ -113,15 +113,15 @@
           "runtime": "litert-lm 0.17.0",
           "prompt_tokens": 256,
           "decode_tokens": 256,
-          "prefill_tps": 1699.15,
-          "decode_tps": 92.79,
-          "ttft_s": 0.16,
-          "load_s": 3.65,
+          "prefill_tps": 1835.85,
+          "decode_tps": 92.96,
+          "ttft_s": 0.15,
+          "load_s": 3.77,
           "max_num_tokens": 1024,
           "cache": "no",
           "runs": 3,
-          "date": "2026-09-08",
-          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, >=300 s rest before the GPU cell, backend gated on a real generation first"
+          "date": "2026-09-13",
+          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, >=300 s rest before the GPU cell, backend gated on a real generation first (same session as the int8 rows; the 2026-09-08 reading was 1699 / 92.79)"
         },
         {
           "device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
@@ -162,12 +162,16 @@
     },
     {
       "file": "MiniCPM5-2B_int8.litertlm",
-      "sha256": "58da116a6eb563185eb5e8d0b1f1e1116b632a516cadc0285b6da35b9f886a86",
-      "size_bytes": 2603278256,
+      "sha256": "61dafe884025d29019302ef27ffb8c677c13eab5c2eb5efd34873ff7cb35c36e",
+      "size_bytes": 2546937984,
       "sections": [
         {
           "type": "LlmMetadataProto",
-          "size_bytes": 9346
+          "size_bytes": 3948
+        },
+        {
+          "type": "ExecutorMetadataProto",
+          "size_bytes": 5807
         },
         {
           "type": "HF_Tokenizer_Zlib",
@@ -175,16 +179,11 @@
         },
         {
           "type": "TFLiteModel",
-          "size_bytes": 2330645824,
+          "size_bytes": 2544824448,
           "model_type": "tf_lite_prefill_decode"
-        },
-        {
-          "type": "TFLiteModel",
-          "size_bytes": 270524336,
-          "model_type": "tf_lite_embedder"
         }
       ],
-      "quantization": "dynamic int8 on linears + embedding (externalized embedder section); prefer_activation_type = fp32 declared in the bundle for the GPU executor",
+      "quantization": "dynamic int8 (channelwise) on every linear, the embedding and the lm_head, all in the prefill/decode graph - the LiteRT-LM models/minicpm5 README command with --quantization_recipe=dynamic_wi8_afp32 on the released litert-torch 0.9.4 / ai-edge-quantizer 0.9.0; GPU activations at the runtime default (fp16)",
       "backends": [
         "cpu",
         "gpu"
@@ -195,22 +194,22 @@
           "platform": "android",
           "device_class": "flagship",
           "backend": "gpu",
-          "reason": "measured on one Galaxy S26 with one binary (2026-09-08, 205-token prompt, 2 runs per backend, cooled before each run): gpu prefill 150.2-159.5 tok/s vs cpu 102.8-157.2, decode 10.86-12.75 vs 11.67-11.75 tok/s (a wash), TTFT 1.4 s vs 1.4-2.1 s, peak VmHWM 0.94-1.10 GB vs 2.90 GB; full LITERT_CL delegation and the gate answer verified through the thought channel. With fp32 activations declared the gpu no longer wins decode; it is recommended for the 2.6x lower process memory"
+          "reason": "measured on one Galaxy S26 with one binary (2026-09-13, v0.16.0 release kit litert_lm_advanced_main, 205-token prompt with --benchmark, 2 runs per backend, SKIN 37.9 C and no CPU frequency cap before every run): gpu prefill 541.5-549.2 tok/s vs cpu 92.6-131.7, TTFT 0.42-0.43 s vs 1.64-2.29 s, decode 22.4-22.6 vs 15.1-18.2 tok/s (1.2-1.5x), peak VmHWM 814-820 MB vs 2,938-3,065 MB; full LITERT_CL delegation (1836/1836 prefill_128, 1655/1655 decode) and the gate answer verified on both backends (also 8/8 on the GPU through the litertlm-android 0.17.0 AAR)"
         },
         {
           "platform": "macos",
           "backend": "gpu",
-          "reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no, generation gated per backend first: gpu 1405 tok/s prefill / 74.7 tok/s decode vs cpu 161 / 30.0"
+          "reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, generation gated per backend first: gpu 2084 tok/s prefill / 124.2 tok/s decode vs cpu 175 / 37.1"
         }
       ],
       "requirements": {
-        "peak_ram_mb": 2896,
+        "peak_ram_mb": 3065,
         "platform_notes": [
-          "Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0",
-          "Hybrid-reasoning model: with no ThinkingConfig it thinks before every answer; budget max output tokens >= 2048 (4096 for math). enable_thinking=false switches to direct answers",
-          "fp32 activations are declared in-bundle: with the runtime's default fp16 GPU activations this model's reasoning on one gate question ran 2000+ tokens without closing </think> (7/8, degenerate); with fp32 it closes in ~450 tokens and passes 8/8, at ~14 % GPU decode cost (Mac). The declaration is a measured trade for the gate, not a general cure",
-          "iOS: the 2.33 GB main weight section is above what a default-entitlement app maps in one piece; the com.apple.developer.kernel.increased-memory-limit entitlement lifts that limit (not checked with this file)",
-          "GSM8K (thinking off, n=100, greedy): 91 vs bf16 92 - at parity; with thinking on it reproduces the bf16 model question for question on a 10-question subset (9/10 chains close, ~3.2k characters)"
+          "Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0 (Mac) and on the v0.16.0 release kit (Galaxy S26)",
+          "Template = the LiteRT-LM models/minicpm5 canonical chat_template.jinja: with nothing set the model answers directly (enable_thinking defaults to false); enable_thinking=true via ThinkingConfig or the conversation's extra_context turns the reasoning on - budget max output tokens >= 2048 (4096 for math) for a thinking run",
+          "One prefill signature (128 tokens); a longer prompt is prefilled in 128-token chunks. KV budget 4096 (max_num_tokens); the command's cache_length 32771 is the runtime's magic number and the cache is sized to 4096 at load",
+          "iOS: the 2.54 GB main weight section is above what a default-entitlement app maps in one piece; the com.apple.developer.kernel.increased-memory-limit entitlement lifts that limit (not checked with this file)",
+          "GSM8K (thinking off, n=100, greedy, max 2048 tokens): 93 on the Mac GPU vs bf16 92 - at parity (inside one standard error at n=100); six of its seven misses are the bf16 model's own. 8-question gate: 6/8 at the no-think default (= the bf16 model's own thinking-off score) and 8/8 with thinking on, on both backends; thinking-on 10-question GSM8K subset: 10/10 chains close, 10/10 correct (Mac GPU, fp16-default activations)"
         ]
       },
       "measured": [
@@ -221,15 +220,15 @@
           "runtime": "litert-lm 0.17.0",
           "prompt_tokens": 256,
           "decode_tokens": 256,
-          "prefill_tps": 160.87,
-          "decode_tps": 30.02,
-          "ttft_s": 1.62,
-          "load_s": 15.03,
+          "prefill_tps": 175.28,
+          "decode_tps": 37.06,
+          "ttft_s": 1.49,
+          "load_s": 4.71,
           "max_num_tokens": 1024,
           "cache": "no",
           "runs": 3,
-          "date": "2026-09-08",
-          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, backend gated on a real generation first"
+          "date": "2026-09-13",
+          "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, CPU cells first, backend gated on a real generation first"
         },
         {
           "device": "Apple M4 Max",
@@ -238,51 +237,51 @@
           "runtime": "litert-lm 0.17.0",
           "prompt_tokens": 256,
           "decode_tokens": 256,
-          "prefill_tps": 1405.16,
-          "decode_tps": 74.65,
-          "ttft_s": 0.2,
-          "load_s": 3.0,
+          "prefill_tps": 2084.28,
+          "decode_tps": 124.24,
+          "ttft_s": 0.13,
+          "load_s": 1.76,
           "max_num_tokens": 1024,
           "cache": "no",
           "runs": 3,
-          "date": "2026-09-08",
+          "date": "2026-09-13",
           "source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, >=300 s rest before the GPU cell, backend gated on a real generation first"
         },
         {
           "device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
           "backend": "gpu",
-          "runtime": "litert_lm_advanced_main v0.16.0, LiteRT CL delegate",
-          "prompt_tokens": 205,
-          "decode_tokens": 1021,
-          "prefill_tps": "150.2-159.5",
-          "decode_tps": "10.86-12.75",
-          "ttft_s": "1.35-1.44",
-          "load_s": 6.3,
-          "peak_memory_mb": 1098,
-          "cache": "no",
+          "runtime": "litert_lm_advanced_main v0.16.0 release kit, LiteRT CL delegate",
+          "prompt_tokens": 207,
+          "decode_tokens": 182,
+          "prefill_tps": "541.5-549.2",
+          "decode_tps": "22.36-22.57",
+          "ttft_s": "0.42-0.43",
+          "load_s": 6.43,
+          "peak_memory_mb": 820,
+          "cache": "kit default (no cache flag; the gate leg ran first, as on the 2026-09-08 rows)",
           "runs": 2,
-          "date": "2026-09-08",
-          "source": "205-token prompt with --benchmark, 2 runs, device cooled below 42 C before each run; GPU gate full LITERT_CL delegation (every node of every subgraph: 1873/1873 on the multi-token prefills, 1664/1664 on prefill_1, 1692/1692 on decode), generation verified on the gate prompt through the thought channel (engine init 6.3 / 4.0 s); the cpu row of the same date is the same-device control"
+          "date": "2026-09-13",
+          "source": "205-token prompt with --benchmark, 2 runs (engine init 6.43 / 5.59 s), SKIN 37.9 C and scaling_max_freq == cpuinfo_max_freq checked before each run, 180 s rest between legs; GPU gate full LITERT_CL delegation (1836/1836 prefill_128, 1655/1655 decode), generation verified on the gate prompt (The capital of France is Paris.); the cpu row of the same date is the same-device control"
         },
         {
           "device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
           "backend": "cpu",
-          "runtime": "litert_lm_advanced_main v0.16.0 (XNNPACK CPU)",
-          "prompt_tokens": 205,
-          "decode_tokens": 1021,
-          "prefill_tps": "102.8-157.2",
-          "decode_tps": "11.67-11.75",
-          "ttft_s": "1.4-2.1",
-          "load_s": 0.27,
-          "peak_memory_mb": 2896,
-          "cache": "no",
+          "runtime": "litert_lm_advanced_main v0.16.0 release kit (XNNPACK CPU)",
+          "prompt_tokens": 207,
+          "decode_tokens": 169,
+          "prefill_tps": "92.6-131.7",
+          "decode_tps": "15.14-18.15",
+          "ttft_s": "1.64-2.29",
+          "load_s": 3.56,
+          "peak_memory_mb": 3065,
+          "cache": "kit default (no cache flag; the gate leg ran first, as on the 2026-09-08 rows)",
           "runs": 2,
-          "date": "2026-09-08",
-          "source": "same-device CPU control: identical binary, prompt file and flags, only --backend differs; 205-token prompt with --benchmark, 2 runs (engine init 0.27 / 0.30 s), cooled before each run"
+          "date": "2026-09-13",
+          "source": "same-device CPU control: identical binary, prompt file and flags, only --backend differs; 205-token prompt with --benchmark, 2 runs (engine init 3.56 / 4.25 s), SKIN 37.9 C and no CPU frequency cap before each run, 180 s rest between legs; CPU gate answers The capital of France is Paris. (XNNPACK 1577/1836 prefill_128 + 1392/1655 decode, the rest on the interpreter)"
         }
       ],
       "known_issues": [
-        "GPU with the runtime's default fp16 activations (i.e. if the in-bundle fp32 declaration is overridden): one 8-question gate item's reasoning ran 2000+ tokens without closing </think>. Keep the declared fp32 activations for this file"
+        "GSM8K (thinking off, 2048-token budget) question 8: on the GPU the derivation runs to the budget mid-calculation and the extracted number is wrong; the CPU run on the same file answers it in 2,204 characters"
       ]
     }
   ]