MiniCPM5-2B (LiteRT-LM)

This repository hosts the LiteRT-LM (LiteRT formerly known as TensorFlow Lite) version of openbmb/MiniCPM5-2B, optimized for fully on-device inference on mobile and edge hardware.

---

Available Models

Recommend to use below both CPU and GPU compatible models. The int8 file is built with the LiteRT-LM models/minicpm5 conversion command (--quantization_recipe=dynamic_wi8_afp32); the int4 file is originally from mlboydaisuke/MiniCPM5-2B-LiteRT. Requires litert-lm ≥ 0.16 (thought channel + ThinkingConfig); measured here on litert-lm 0.17.0 (Mac) and the v0.16.0 release kit (Galaxy S26).

File	Recipe	Size
MiniCPM5-2B_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding	1.55 GB
MiniCPM5-2B_int8.litertlm	int8 dynamic (channelwise) on linears, embedding and lm_head — dynamic_wi8_afp32	2.55 GB

The int4 file is the phone file (smaller, fastest GPU decode on every platform measured) — best used for direct answers or short reasoning; see the thinking-mode note below. int8 is the file for reasoning that has to complete: its thinking chains are ~3–4× shorter than int4's on the same questions and terminate where int4 runs into the token budget. int8's main weight section is 2.54 GB, more than an iOS app maps in one piece with the default entitlements; an app that carries the com.apple.developer.kernel.increased-memory-limit entitlement lifts that limit (not checked with this file).

Both files declare the thought channel, so the runtime's ThinkingConfig and the reasoning/answer split work on both. They carry different chat templates, and that changes the thinking default: the int4 file embeds the checkpoint's own chat_template.jinja (the model decides, and in practice thinks); the int8 file embeds the LiteRT-LM models/minicpm5/chat_template.jinja as of 2026-09-01 (enable_thinking defaults to false: direct answers unless the app turns thinking on). Details under Controlling thinking and Conversion notes.

Further, below are some CPU-only models for exploration:

• minicpm_wi4c_wi8_afp32.litertlm: This model features mixed INT4/INT8 weight-only quantization with FP32 activations (afp32). MLP projections use channelwise INT4 with Hadamard rotation; all remaining weights (attention, embedding, and lmhead) use channelwise INT8.
• minicpm_wi8_afp32.litertlm: This model features weight-only INT8 quantization (wi8) with FP32 activations (afp32).

What is MiniCPM?

MiniCPM5-2B is part of the MiniCPM5 series from OpenBMB. It is a dense 2.5B-parameter Transformer built specifically for on-device, local, and resource-constrained deployment.

Highlights

• 🧠 Hybrid Reasoning — a single checkpoint serves as both a fast assistant and a deliberate reasoner via a built-in <think> template (enable_thinking).
• 🛠️ Tool use — native tool-calling support in the chat template.
• 📏 Long context — native 131,072-token context length.
• 📱 Built for the edge — compact footprint designed for local assistants, coding agents, and tool-use workflows.

Model Information

Item	Value
Type	Causal Language Model
Architecture	Standard LlamaForCausalLM
Parameters	2,516,756,480 (~2.5B)
Non-Embedding Parameters	1,981,982,720
Layers	42
Attention Heads (GQA)	16 (Q) / 2 (KV), head dim 128
Hidden / Intermediate Size	2048 / 6144
Vocabulary Size	130,560
Context Length	131,072

---

Use the model

Edge Gallery App (Android)

1. Get the App: Install the app from Google Play or download the latest APK from the GitHub releases page.
2. Importing the Model: Navigate to the Model manager within the app and click the "+" (plus) icon in the bottom-right corner. Two options will appear:
   • Import from HF (Recommended): Select this option, and a dialog box will appear showing an example Hugging Face model URL. Enter the HF link for the desired .litertlm model and click submit. The model will then appear in your list, and you can proceed to download it (a Hugging Face account login is required).
   • From local model file: First, download the .litertlm model directly to your Android device, OR download it to your computer and push it via ADB (e.g., adb push MiniCPM5-2B_int4.litertlm /sdcard/Download/). Then, select this option, choose the downloaded file from your storage, configure your preferred parameters, and tap "Import".

For full details on importing models and other features, see the Edge Gallery App Wiki.

To build the demo app from source, please follow the instructions from the GitHub repository.

Try It (Desktop/CLI)

Install uv and run the model directly from the LiteRT-LM command line:

uv tool install litert-lm
uvx litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm --prompt="What is the capital of France?"

# local file, GPU backend
litert-lm run ./MiniCPM5-2B_int4.litertlm --backend gpu --cache no --prompt "..."

Controlling thinking

The two files start from opposite defaults, because they carry different templates (see Conversion notes):

• int4 (the checkpoint's own template): thinking is the model's default — with no ThinkingConfig it decides for itself and, in practice, reasons before every answer (100–700 characters on trivial questions, thousands on math).
• int8 (the LiteRT-LM canonical template): with nothing set the template closes the think block itself, so the model gives direct answers; enable_thinking=true turns the reasoning on.

For both files:

• Give a reasoning run a generous output budget (≥ 2048 tokens; 4096 for math). Truncated mid-thought it produces no final answer at all.
• Both bundles declare the thought channel (<think> / </think>), so the streamed text contains only the answer, and runtimes that expose ThinkingConfig can cap or read the reasoning separately.
• enable_thinking set via ThinkingConfig or the conversation's extra context reaches either template: false switches to direct answers (two- to seven-token replies on the gate questions, ~10× faster turns, and the GSM8K numbers below); true pre-fills the think opener explicitly. On the CLI: --thinking true|false / --thinking-budget N.
• Sampling: OpenBMB recommends temperature 1.0, top_p 0.95; the gates below are greedy.
• Prompt format is ChatML (<|im_start|>role\n…<|im_end|>\n), 4096-token KV budget. Prefill signatures: int4 six (1024, 256, 64, 16, 4, 1); int8 one (128 — a longer prompt is prefilled in 128-token chunks).

Correctness

On an 8-question sanity gate (Apple M4 Max, greedy, CPU and GPU) the int4 file scores 8/8 at its default (the model thinks; the reasoning arrives on the thought channel, only the answer in the streamed text) and the int8 file scores 6/8 at its default (direct answers; measured with nothing set on the GPU and with thinking off on both backends) and 8/8 with thinking on, on both backends. The two thinking-off misses are the bf16 PyTorch model's own: it scores 8/8 with thinking on or left to the model and 6/8 with thinking forced off ("opposite of hot" → "Cool.", the rhyme line → "Green."), so read any thinking-off result against that, not against 8/8.

GSM8K (first 100 test questions, greedy, 0-shot chain-of-thought prompt, thinking off — the protocol OpenBMB's own MiniCPM5 cards use, max 2048 new tokens, identical prompt and extraction on every row):

Configuration	GSM8K
bf16 PyTorch (MPS), upstream template	92 %
int8, GPU (Metal, WebGPU delegate)	93 %
int4, CPU	86 %
int4, GPU (Metal)	87 %

int8 is at parity: 93 vs 92 is inside one standard error at n=100 (about ±3 points), and six of its seven misses are the bf16 model's own; the seventh (question 8) is a derivation that runs to the 2048-token budget mid-calculation on the GPU, so the extracted number is wrong, while the CPU run on the same file answers it in 2,204 characters. int4 costs about five points on this 42-layer model; the GPU's default fp16 activations cost nothing measurable in no-think mode.

Thinking mode is where int4 shows its damage. On ten GSM8K questions with thinking on and a 3584-token budget, the bf16 model closes its reasoning on 9/10 with ~3,000-character chains, and int8 closes 10/10 with ~3,900-character chains (10/10 correct; Metal GPU through the WebGPU delegate, runtime-default fp16 activations), while int4 closes 0/10 on CPU (median ~13,700 characters — it keeps re-checking and runs into the budget; the answer is usually right inside the thought text but never gets emitted). On the Metal GPU with the runtime's default fp16 activations the same int4 file happened to close 10/10 with ~8,000-character chains, but that is fp16 rounding steering the trajectory, not a property to rely on. If your use needs the reasoning to finish, use int8 or turn thinking off on int4.

On a Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno) both files generate correctly on GPU and CPU; on the GPU with full OpenCL delegation — every node of every prefill and decode signature, zero rejected ops (int4: 1873/1873 on the 1024- to 4-token prefills, 1664/1664 on the 1-token prefill, 1692/1692 on decode; int8: 1836/1836 on the 128-token prefill, 1655/1655 on decode) — and the runtime separates the reasoning on-device ([thought] … [/thought], then the answer).

On an iPhone 17 Pro the int4 file passes the same 8-question gate on both backends (on-device byte count verified against the source file): Metal GPU 7/8 (init 5.7 s) and CPU 7/8 (init 2.2 s); the one miss on each leg is the rhyme line inside the 8-question composite prompt, answered "green" — the same answer the bf16 model gives with thinking off, so a prompt-format artifact rather than conversion damage.

Multi-turn: three-turn conversations (introduce a name and a city, an arithmetic question, then "which city do I live in?") hold on both files under all three thinking modes with the name and city recalled — each template's history rendering stays consistent with what the runtime already streamed.

Performance

litert-lm benchmark (litert-lm 0.17.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, serialized, ≥300 s rest before each GPU reading; each backend confirmed to generate real text before its number was recorded; all four rows from one session (2026-09-13):

File	Backend	Prefill (256)	Decode	TTFT	Init
int4	GPU (Metal)	1836 tok/s	93.0 tok/s	0.15 s	3.8 s
int4	CPU	148 tok/s	31.4 tok/s	1.77 s	4.4 s
int8	GPU (Metal)	2084 tok/s	124.2 tok/s	0.13 s	1.8 s
int8	CPU	175 tok/s	37.1 tok/s	1.49 s	4.7 s

Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno; litert_lm_advanced_main from the litert-lm v0.16.0 release kit, 205-token prompt with --benchmark, 2 runs per cell, ranges shown; a reasoning model decodes its own full response, so decode-turn lengths vary):

File	Backend	Prefill (205)	Decode	TTFT	Init	Peak RSS
int4	GPU (OpenCL)	401–411 tok/s	16.1–18.6 tok/s	0.56 s	11.2–13.1 s	1.14 GB
int4	CPU	39–72 tok/s	15.6–15.8 tok/s	2.9–5.3 s	3.1–5.7 s	2.12 GB
int8	GPU (OpenCL)	542–549 tok/s	22.4–22.6 tok/s	0.42–0.43 s	5.6–6.4 s	0.82 GB
int8	CPU	93–132 tok/s	15.1–18.2 tok/s	1.6–2.3 s	3.6–4.2 s	3.07 GB

The int8 rows are the README-command build (one 128-token prefill signature, odml.cache_update composites, runtime-default fp16 GPU activations) and are not a like-for-like update of the previous int8 rows on this card (fp32 activations declared, six signatures). GPU wins prefill (4–10× on the phone, 11–12× on the Mac) and time-to-first-token everywhere; on Adreno the int4 GPU decode edge over the same-device CPU is modest (~1.1×), while the int8 file decodes 1.2–1.5× faster on the GPU than on its CPU at about a quarter of the process memory. The int8 phone rows were taken at 37.9 °C skin temperature with no CPU frequency cap in force before each run.

Conversion notes

The two files come from two different conversion paths.

int8 — the LiteRT-LM models/minicpm5 command, as written. litert-torch export_hf (litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, litert-lm-builder 0.16.1, transformers 5.14.1 — the released wheels) with --quantization_recipe=dynamic_wi8_afp32 --use_bool_mask=True --apply_gpu_composites=True --cache_length=32771 --experimental_lightweight_conversion and --litert_lm_llm_metadata_override pointing at the checked-in LlmMetadataProto.pbtext as of 2026-09-01 (commit b5e34ab1; the file on main moved on 2026-09-11 to a template that takes message content as parts only); only the model id (openbmb/MiniCPM5-2B), the metadata path and the output directory differ from the README block. What that produces:

• Weights: int8 per-channel on every linear including the 130560×2048 embedding and lm_head, all in the single prefill/decode graph (no separate embedder section); activations at the runtime default. Graph: prefill_128 1836 ops / decode 1655 ops, odml.cache_update composites, boolean mask; no custom op.
• Metadata = the pbtext: start token <s>, stops </s> and <|im_end|>, max_num_tokens 4096, llm_model_type minicpm5, the thought channel as <think>\n / </think>, and the LiteRT-LM canonical chat_template.jinja (ChatML, tool definitions in a <tools> block, enable_thinking | default(false)). --cache_length=32771 is the runtime's magic number: the KV cache is sized from the metadata's 4096 at load.
• Tokenizer: the checkpoint's tokenizer.json (byte-level BPE) as an HF tokenizer section.

int4 — a blockwise recipe with two pieces of handling (reproduction: hf-to-litertlm, bash scripts/reproduce_llm.sh minicpm5-2b; litert-torch 0.9.3 / litert-converter 0.4.0 / ai-edge-quantizer 0.9.0 / litert-lm-builder 0.16.1):

• The chat template is the checkpoint's chat_template.jinja, byte for byte, embedded on the runtime's jinja path — the same packaging as litert-community/MiniCPM5-1B. That is what keeps enable_thinking and the tool-calling format available to the app; the thought channel is declared alongside it.
• Start token <s> is correct for this family: the template's own {{ bos_token }} renders empty at runtime and the engine prepends the metadata start token, so the model sees exactly one <s> as it does upstream. Stops are the model's </s> and <|im_end|>.
• int4 needed a zero-scale fix. Decoder layer 0's MLP contains 13 dead rows (every weight below 3e-23); blockwise quantization emits a zero scale for each of their blocks, which the CPU (XNNPACK) path refuses to load (unsupported scale value (0.000000) … for INT4 tensor) while the GPU path silently accepts. The scales were replaced in place by a tiny positive epsilon — 3,328 bytes of a 1.55 GB file.
• The 130560×2048 input-embedding table sits in its own section (embedder externalised, int8); linears are blockwise-32 int4 with OCTAV clipping; GPU activations at the runtime default (fp16). A block-128 int4 variant was built and rejected: it loses 8 GSM8K points (79 vs 87, thinking off), fails the 8-question gate on CPU, and does not shorten the thinking chains.

Machine-readable manifest

litertlm_manifest.json in this repo describes both recommended files (sha256, size, sections, context length, verified backends, the measured rows above with their conditions, and known issues) for tooling that picks a file per device.

Links

• 🤗 Original model (BF16): openbmb/MiniCPM5-2B
• 🤗 Smaller sibling: litert-community/MiniCPM5-1B
• 📦 GitHub: OpenBMB/MiniCPM
• 🛠️ LiteRT docs: ai.google.dev/edge/litert · runtime source: github.com/google-ai-edge/litert / LiteRT-LM

---

License

Released under the Apache-2.0 License, consistent with the upstream openbmb/MiniCPM5-2B.

Changes from the original work (for the two recommended files): weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; in the int4 file, 13 dead weight rows' quantization scales set to an epsilon (measured inert: GSM8K 87 with the identical miss list on a copy whose payload under those scales is zeroed); tokenizer and chat template repackaged into the .litertlm bundle with runtime metadata (thought channel, stop tokens; the int8 file carries the LiteRT-LM canonical template instead of the checkpoint's). No fine-tuning.

Citation

@article{minicpm4,
  title={MiniCPM4: Ultra-efficient LLMs on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}