granite-speech-5.0-2b — LiteRT-LM

This repository is a conversion of ibm-granite/granite-speech-5.0-2b to LiteRT-LM, made with litert-torch 0.10.0 and litert-lm-builder 0.19.1 for LiteRT-LM 0.19.1: it takes 16 kHz audio and returns the transcript, and with granite-speech-5.0-2b_ekv2048.litertlm a Galaxy S26 transcribes a 9.9-second clip in 2.47 s, with the first text after 1.21 s (language model and audio encoder on the GPU, mean of 2 processes × 2 calls, 2026-11-18).

On 2026-11-17 granite-speech-5.0-2b.litertlm was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. On a Galaxy S26 GPU the new file's language model decodes 25.9 tokens/s after a 1,024-token prefill, against 15.0 tokens/s for the file it replaced in the same series of runs (LiteRT-LM v0.19.0; the last part of Performance). granite-speech-5.0-2b_ekv2048.litertlm was not replaced then. The new file was run only with LiteRT-LM 0.19.0 to 0.19.2; earlier runtimes were not tried with it. Apart from the parts that name the file of 2026-11-17, this card's measurements of granite-speech-5.0-2b.litertlm were made with earlier files of that name.

granite-speech-5.0-2b is IBM's compact speech-language model for multilingual speech recognition and speech translation in English, French, German, Spanish, Portuguese and Japanese (translation also from English to Italian and Mandarin). It has about 2.3 billion parameters: a 16-layer conformer speech encoder, a 2-layer window query transformer that turns every 0.3 s of encoder output into 3 embeddings (10 per second of audio), and a 40-layer Granite language model built from an intermediate checkpoint of granite-4.0-1b-base. The source card recommends segments of up to about 2 minutes. IBM also publishes GGUF files and an ONNX build.

Which file

File Audio in one message Audio encoder runs on Galaxy S26, 9.9 s clip: first text / full transcript
granite-speech-5.0-2b_ekv2048.litertlm (added 2026-11-07) 116–152 s (context 2,048 tokens) GPU 1.21 s / 2.47 s (language model on the GPU), 1.44 s / 3.08 s (on the CPU)
granite-speech-5.0-2b.litertlm (added 2026-11-06) 234–307 s (context 4,096 tokens) CPU 3.83 s / 5.87 s (language model on the GPU), 5.16 s / 7.65 s (on the CPU), both with an earlier file

Use granite-speech-5.0-2b_ekv2048.litertlm unless one message has to hold more than about 2 minutes of audio. The audio range depends on how long the reply is: it is the longest clip that still fits at the highest, the 95th-percentile and the median reply length of our 220 test clips (How to send audio). The phone numbers come from one Galaxy S26, those of granite-speech-5.0-2b_ekv2048.litertlm from 2026-11-18 and those of granite-speech-5.0-2b.litertlm from 2026-11-07 (Performance).

Files

File What is inside Size sha256
granite-speech-5.0-2b_ekv2048.litertlm Three graphs (Section table below): the language model's decoder and token embedding table, and the audio encoder. Language model: int8 weights (one scale per output channel), KV cache of 2,048 tokens, prefill signatures of 64, 256 and 1,024 tokens, no activation-type declaration (on the GPU it runs float16 activations, the runtime's default). Audio encoder (log-mel front end, conformer, query transformer, projection): rewritten in operations the GPU accepts (tensors of rank 4 or less, no GATHER_ND), float16 weights computed in float32, one 24-second window. 2,824,472,576 B 9f4d3dae0cb7b7d5abfdf0fc1ac9840ab01b546ad6c39e85a1011eee8ce5fe50
granite-speech-5.0-2b.litertlm Three graphs, as above. Language model: the same int8 recipe, KV cache of 4,096 tokens, the same prefill signatures; the file asks the GPU for float32 activations. Audio encoder: float16 weights computed in float32, one 36-second window, CPU only. 2,847,148,032 B 6ab95431c9c423cdc7822fc4e2961227fb05902916f34da9e7db97c720b30bdc
litertlm_manifest.json Machine-readable description of both files
Section Content _ekv2048 granite-speech-5.0-2b.litertlm
tf_lite_audio_encoder_hw log-mel front end, 16-layer conformer, 2-layer query transformer, projection to the language model 941,890,544 B 964,053,120 B
tf_lite_prefill_decode Granite language model, int8 graph 39,150,352 B + weights 1,631,586,304 B graph 39,656,200 B + weights 1,631,586,304 B
tf_lite_embedder token embedding table, int8 graph 4,824,712 B + weights 205,522,944 B graph 4,824,712 B + weights 205,522,944 B
HF_Tokenizer_Zlib the source's tokenizer.json 1,404,656 B (compressed) 1,404,656 B (compressed)

Both files carry the source repository's tokenizer.json byte for byte, a chat template that renders the source's prompt USER: <|audio|>{instruction}\n ASSISTANT: from LiteRT-LM's message parts, the stop token <|end_of_text|> (100257), no start token (the source tokenizer adds none), and the model type generic_model.

How to send audio

  • One message holds one audio item, then the instruction as text. Use the prompts of the source card, for example transcribe the speech with proper punctuation and capitalization. (punctuated transcript), can you transcribe the speech into a written format? (plain transcript), transcribe the speech to text. Keywords: <kw1>, <kw2>, ... (keyword biasing) or translate the speech to <language>. (translation). Put the audio first, as the source card's examples do (<|audio|> before the instruction); the template keeps the order of the parts.
  • Send 16 kHz mono WAV. We sent 16-bit PCM (the 220 test clips) and 32-bit float (the source card's multilingual_sample.wav); other sample rates, stereo files and other containers were not tested. The runtime cuts the audio into 10 ms frames and gives the encoder fixed windows: 24 s in _ekv2048, 36 s in granite-speech-5.0-2b.litertlm. The encoder always computes a whole window, also for a short clip: one 24-second window takes 114 ms on an M4 Max GPU (Metal) and 476 ms on its CPU (XNNPACK, 4 threads) (the encoder graph alone in ai-edge-litert 2.3.0, median of 10; a 5.9-second clip took the same time), and inside a LiteRT-LM call on the Galaxy S26 GPU 0.77 s with the language model on the GPU and 0.87 s with it on the CPU (the 9.9-second clip, Performance). Only the audio itself becomes input for the language model (10 positions per second).
  • Longer audio can go in one message, up to the context. A clip longer than one window is encoded window by window and the windows' outputs are joined. A 62.5-second test clip sent in one message to an earlier _ekv2048, the one this repo held on 2026-11-07, differed from the source model's single-pass transcript by 1.29 % word error rate with the language model on the Mac CPU and on the Mac GPU; for scale, the source model's own transcript of the same clip cut in two pieces differed from its single-pass transcript by the same 1.29 %. The context holds the prompt and the reply together. With _ekv2048 (2,048 tokens) one message fits about 116 s of audio at the highest reply length of our 220 test clips (in tokens per second of audio), 131 s at their 95th percentile and 152 s at their median; with granite-speech-5.0-2b.litertlm (4,096 tokens) 234, 264 and 307 s. A message that does not fit is refused with INVALID_ARGUMENT, not cut (seen with granite-speech-5.0-2b.litertlm: a 421.6-second clip came to 4,236 tokens).
  • Digital silence at the very end is ignored. The runtime does not tell the encoder how long the clip is, so the encoder takes the last non-zero sample as the end. A file that ends in exact zeros is read as slightly shorter: 19 of our 220 test files end that way (1 to 274 zero samples); on the 5 with 173 to 274 the encoder got one frame fewer than the source model, and the source language model's transcript from that shorter input was unchanged on all 5.
  • Audio encoder backend. _ekv2048: the GPU (audio_backend GPU, the command-line flag --audio-backend gpu), run on the Mac through LiteRT-LM's GPU path (WebGPU) and with OpenCL on a Galaxy S26 and, with the earlier _ekv2048, on a Pixel 11 Pro; the Metal time above is the encoder graph alone in ai-edge-litert. granite-speech-5.0-2b.litertlm: the CPU only (--audio-backend cpu); the Mac's GPU delegate refuses several of its encoder's operations, among them GATHER_ND and 5-dimensional tensors, and creating a conversation then fails with INTERNAL. In both files the language model can run on the CPU or the GPU.

Usage

One clip with the LiteRT-LM command-line tool (uvx comes with uv; with pip, pip install litert-lm==0.19.1 and run litert-lm run … instead):

pip install -U huggingface_hub
hf download litert-community/granite-speech-5.0-2b granite-speech-5.0-2b_ekv2048.litertlm --local-dir .
hf download ibm-granite/granite-speech-5.0-2b multilingual_sample.wav --local-dir .
uvx --from litert-lm==0.19.1 litert-lm run granite-speech-5.0-2b_ekv2048.litertlm --backend gpu --audio-backend gpu --attachment multilingual_sample.wav --prompt "transcribe the speech with proper punctuation and capitalization." < /dev/null

The same from Python (pip install litert-lm==0.19.1):

import litert_lm
from litert_lm import Content, Contents, interfaces

engine = litert_lm.Engine(
    "granite-speech-5.0-2b_ekv2048.litertlm",
    backend=interfaces.Backend.GPU(),
    audio_backend=interfaces.Backend.GPU(),
)
conversation = engine.create_conversation(sampler_config=interfaces.SamplerConfig(top_k=1))
message = Contents.of(
    Content.AudioFile("multilingual_sample.wav"),
    Content.Text("transcribe the speech with proper punctuation and capitalization."),
)
reply = conversation.send_message(message)
print("".join(part["text"] for part in reply["content"] if part["type"] == "text"))

interfaces.Backend.CPU() (or --backend cpu) puts the language model on the CPU; on the Galaxy S26 Backend.CPU(thread_count=6) reached the first text sooner than the default with the earlier _ekv2048 (Performance). With granite-speech-5.0-2b.litertlm, keep the audio encoder on the CPU (audio_backend=interfaces.Backend.CPU(), --audio-backend cpu). On the Mac, the first run of the command-line tool wrote caches next to the model file (about 1.6 GB for the language model on either backend and 7 MB for the audio encoder), and later runs load them.

Use a new conversation for each clip; every number on this card comes from one conversation per clip. From the second message of a conversation on, LiteRT-LM 0.19.1 puts the previous answer's stop token into the context with a zero embedding (read in the runtime's source; these files' language model takes embeddings, not token ids), so a second transcript in one conversation would start from a context the model never saw; we did not measure it on these files.

If you download these files with the litert-lm CLI's --from-huggingface-repo option instead of hf download, and the CLI stops right after the download with no error message of its own (a bus error; exit status 138 on a Mac) or with an error log that contains too large for file size, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in ~/.litert-lm/cache/huggingface/litert-community/granite-speech-5.0-2b/ and uses it again on later runs without checking its size. Delete that file and run the command again (after an import, run the import again). This was measured with other models' files, not with these.

If you downloaded granite-speech-5.0-2b.litertlm with the litert-lm CLI's --from-huggingface-repo before it was replaced on 2026-11-17 (12:03 UTC), delete it from ~/.litert-lm/cache/huggingface/litert-community/granite-speech-5.0-2b/, then run your command again (after an import, run the import again). Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo. The previous file stays at revision b1b534e.

Performance

Galaxy S26 (Android 16), LiteRT-LM 0.19.1 (the litert-lm-api 0.19.1 Android wheel from PyPI), the _ekv2048 rows on 2026-11-18 and the granite-speech-5.0-2b.litertlm rows on 2026-11-07, measured the same way. Clip: one streaming call per clip with a new conversation each, after a warm-up clip, the mean of 2 processes × 2 calls. Audio encoding: the call's time minus its prefill and decode, less the same remainder of a text-only call, on the 9.9-second clip. Peak memory: the highest peak resident memory (VmHWM) of these timing processes, GPU buffers not included. CPU work ran at the runtime's default thread count. The phone's thermal governor capped the CPU frequency in most samples of every row (_ekv2048: 20 of 36 with the language model on the GPU, 33 of 38 on the CPU; granite-speech-5.0-2b.litertlm: 45 of 53 and 56 of 59), and the GPU clock as well in the rows with the language model on the GPU (_ekv2048: 4 of 36), so these are throttled numbers. Creating the engine is not in them.

File Language model on Audio encoder on 9.9 s clip: first text / full transcript Audio encoding Peak memory
_ekv2048 GPU GPU 1.21 s / 2.47 s 0.77 s 2.5 GB
_ekv2048 CPU GPU 1.44 s / 3.08 s 0.87 s 4.4 GB
granite-speech-5.0-2b.litertlm (an earlier file) GPU CPU 3.83 s / 5.87 s 2.85 s 4.0 GB
granite-speech-5.0-2b.litertlm (an earlier file) CPU CPU 5.16 s / 7.65 s 4.12 s 6.1 GB

Language model alone on the Galaxy S26 (litert_lm.Benchmark, a forced prefill of 1,024 tokens, then 256 decoded tokens, the mean of runs 2 and 3 in two processes, other series of runs of 2026-11-07): in two paired series of runs, a file whose two language-model sections are the same bytes as those of an earlier _ekv2048, the one this repo held on 2026-11-07, gave 933.2 tokens/s prefill and 23.59 tokens/s decode on the GPU and 229.6 and 15.09 on the CPU, and granite-speech-5.0-2b.litertlm (an earlier file of that name) 226.6 and 14.88 on the GPU and 137.9 and 10.03 on the CPU; that earlier _ekv2048 itself gave 229.2 and 15.34 on the CPU in the thread-count series of runs below.

CPU thread count on the Galaxy S26 (the earlier _ekv2048, language model on the CPU, one series of runs, the default and 6 threads alternating, 2 processes each): with Backend.CPU(thread_count=6) the 9.9-second clip took 1.43 s to the first text and 2.91 s in all, against 1.63 s and 3.26 s at the default (4 in the LiteRT-LM 0.19.1 source); over a 5.9, a 9.9 and a 29.4-second clip, 6 threads took 0.85 times the default's time to the first text. The benchmark above was not faster with 6 threads (decode 14.86 against 15.34 tokens/s). In a later series of runs 8 threads were not shown to be faster than 6: the three-clip time to the first text was 4 % lower with 8, but 0.23 of that 0.26 s came from the audio encoder on the GPU, whose time fell from one measurement to the next through the run (4.7 → 3.7 s summed over the three clips), a drift the alternating order did not cancel; the language model's own part (first text minus the encoder's time) was 1.2 % lower with 8. Other phones were not measured.

Mac M4 Max (128 GB), LiteRT-LM 0.19.1 from PyPI (its Mac GPU path is WebGPU), the earlier _ekv2048, 2026-11-07. Language model: litert-lm benchmark granite-speech-5.0-2b_ekv2048.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend <cpu|gpu> (CPU at 4 threads). Clip: a 10.0-second FLEURS clip, the third conversation of a process (after a warm-up clip and a 5.9-second clip), median of 5 processes, the audio encoder on the GPU in both rows. Encoder: one 24-second window of the file's encoder graph alone in ai-edge-litert 2.3.0 (GPU: Metal; CPU: XNNPACK, 4 threads), median of 10. Creating the engine took 2.0 s (GPU) and 11.0 s (CPU) in the benchmark runs, which keep no cache (--cache no).

Language model on Prefill (tokens/s) Decode (tokens/s) Encoder, one 24 s window 10 s clip: first text / full transcript
GPU 3,373.13 109.87 114.1 ms (GPU) 0.226 s / 0.445 s
CPU 841.62 36.21 476.1 ms (CPU) 0.410 s / 1.049 s

Galaxy S26 and Pixel 11 Pro: granite-speech-5.0-2b.litertlm of 2026-11-17

These runs time the language model alone. Each run prefills 1,024 tokens and generates 256 tokens, three times in one process (three iterations), with a text prompt and the context set to 2,048 tokens. The program is litert_lm_advanced_main built from the LiteRT-LM v0.19.0 source (commit 771b06e1; the tag's own commit after it changes only Package.swift and README.md), run with --backend=<gpu|cpu> --async=false --use_session=false --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=3 --max_num_tokens=2048 --max_output_tokens=2048 on 2026-11-17; the CPU runs used 4 threads, the program's default. Four runs went in the order previous file, new file, new file, previous file: once on the Galaxy S26 GPU, once on the Galaxy S26 CPU (after one warm-up run of the new file, which is not counted) and three times on the Pixel 11 Pro GPU. Each run is a new process that starts without the runtime's cache files. Each value is one run's mean of iterations 2–3, so it leaves out iteration 1, the first pass after the engine starts; the ratios are taken before rounding.

Phone, language model on Prefill, tok/s (previous, new, new, previous) Decode, tok/s (previous, new, new, previous) New / previous, runs 2 and 1 New / previous, runs 3 and 4
Galaxy S26, GPU 190.17, 208.08, 203.54, 187.66 14.97, 25.96, 25.78, 15.06 prefill ×1.094, decode ×1.734 prefill ×1.085, decode ×1.712
Galaxy S26, CPU 151.91, 156.61, 159.56, 152.54 11.03, 11.10, 11.09, 10.67 prefill ×1.031, decode ×1.007 prefill ×1.046, decode ×1.040
Pixel 11 Pro, GPU, first four runs 174.59, 191.40, 192.93, 174.89 7.44, 10.94, 10.94, 7.24 prefill ×1.096, decode ×1.470 prefill ×1.103, decode ×1.510
Pixel 11 Pro, GPU, second four runs 172.97, 189.97, 191.83, 168.37 7.42, 10.61, 10.83, 7.27 prefill ×1.098, decode ×1.430 prefill ×1.139, decode ×1.491
Pixel 11 Pro, GPU, third four runs 172.77, 188.85, 191.98, 171.66 7.48, 10.62, 10.58, 7.22 prefill ×1.093, decode ×1.419 prefill ×1.118, decode ×1.466

The mean of each file's two runs:

Phone, language model on New file: prefill / decode / time to first token Previous file: prefill / decode / time to first token
Galaxy S26, GPU 205.81 tok/s / 25.87 tok/s / 5.03 s 188.91 tok/s / 15.01 tok/s / 5.52 s
Galaxy S26, CPU 158.08 tok/s / 11.10 tok/s / 6.58 s 152.22 tok/s / 10.85 tok/s / 6.86 s
Pixel 11 Pro, GPU, first four runs 192.16 tok/s / 10.94 tok/s / 5.42 s 174.74 tok/s / 7.34 tok/s / 6.00 s
Pixel 11 Pro, GPU, second four runs 190.90 tok/s / 10.72 tok/s / 5.46 s 170.67 tok/s / 7.34 tok/s / 6.14 s
Pixel 11 Pro, GPU, third four runs 190.41 tok/s / 10.60 tok/s / 5.48 s 172.21 tok/s / 7.35 tok/s / 6.08 s

On the Galaxy S26, by these means the new file's language model is faster on the GPU than on the CPU: 205.81 against 158.08 tokens/s in prefill and 25.87 against 11.10 in decode. In iteration 1, which the means leave out, the CPU prefilled faster (276.64 and 277.66 tokens/s in the new file's two runs, against 228.60 and 228.75 on the GPU) and the GPU decoded faster (26.20 and 26.49 against 12.70 and 12.46). On the phones, the only clip runs with the new file are those of the last paragraph below, made to compare the replies (one call per process, the language model on the GPU), not as timing runs; none put its language model on the CPU. The Pixel 11 Pro CPU was not run with it.

On the Galaxy S26 (SM-S942Q) every GPU run started at Android's thermal status 0 and ended at 2. The GPU driver's own thermal level (0 = no limit) rose from one iteration to the next and reached 4 in iteration 3 of every run, with both files (a GPU clock limit of 1,000 MHz, against 1,300 MHz at level 0). Of the CPU runs, the new file's two started at thermal status 0 and the previous file's two at 1; during every one of them the phone lowered the frequency limits of both CPU core groups, to the same lowest values in all four. A system app (com.google.android.gms) was in the foreground when each Galaxy S26 run started. On the Pixel 11 Pro every run started at thermal status 0 and ended at 1, and in 10 of the 12 runs (5 of the 6 with each file) the phone's thermal service lowered the GPU's power budget below its resting value at some point.

On both phones, the new file and the file it replaced gave the same reply to each of two clips, byte for byte: one LibriSpeech clip and one Japanese FLEURS clip, with the instruction transcribe the speech with proper punctuation and capitalization., the language model on the GPU, the audio encoder on the CPU and greedy decoding. The reply to the LibriSpeech clip was mister Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.

Accuracy

Apart from the Galaxy S26 timing processes of 2026-11-18, this section's measurements of _ekv2048 were made with an earlier file of that name, the one this repo held on 2026-11-07, before its language-model weights were moved to sections at the end of the file (Known runtime issues).

220 clips: English = 20 LibriSpeech dev-clean clips + 50 FLEURS en_us test clips; Japanese = 50 FLEURS ja_jp test clips; French, German, Spanish and Portuguese = 25 FLEURS test clips each. The instruction is transcribe the speech with proper punctuation and capitalization., greedy decoding, at most 256 new tokens, one conversation per clip. English is scored with Whisper's English text normalizer, the four other European languages with Whisper's basic normalizer (openai-whisper 20250625), Japanese by characters after NFKC normalization and lowercasing with punctuation and spaces removed. The source model runs in transformers 5.22.1 (bfloat16 weights computed in float32 on the CPU); _ekv2048 runs in LiteRT-LM 0.19.1 on a Mac M4 Max with its audio encoder on the GPU:

Model English WER Japanese CER French + German + Spanish + Portuguese WER
Source model (transformers) 4.60 % (73/1,588) 5.13 % (138/2,689) 12.10 % (286/2,363)
_ekv2048, language model on the CPU 4.41 % (70/1,588) 5.21 % (140/2,689) 11.93 % (282/2,363)
_ekv2048, language model on the GPU 4.53 % (72/1,588) 5.24 % (141/2,689) 12.15 % (287/2,363)

The thresholds, set before the file was built, were the source model's numbers plus 0.5 points (English) and plus 1.0 point (Japanese, and the four European languages together). No reply was empty, repeated itself or ran to the token limit. The transcripts are character for character the source model's (punctuation and case included) on 180 of the 220 clips with the language model on the CPU and on 195 with it on the GPU. granite-speech-5.0-2b.litertlm is within 0.2 points of these numbers in every group (the block below).

The source card's multilingual_sample.wav (24.9 s, English and then French) with the instruction above kept both languages in _ekv2048: the transcript differed from the source model's by 0.00 % word error rate with the language model on the CPU and on the GPU, on the Mac (Python and the command-line tool) and on the Galaxy S26.

On a Pixel 11 Pro, a 38-clip subset of the 220 ran with the same instruction and decoding: the 20 LibriSpeech clips, the first 10 Japanese clips and the first 2 French, German, Spanish and Portuguese clips. The thresholds were the source model's number on these clips plus 0.5 point for English, 1.0 for Japanese and 1.0 for the four European languages together, and in both runs all 38 replies stopped on their own, none empty or repeating:

Run English WER Japanese CER French + German + Spanish + Portuguese WER
Source model (transformers), the same 38 clips 2.43 % 7.26 % 2.03 %
_ekv2048, language model on the CPU 2.21 % 7.59 % 2.03 %
_ekv2048, language model on the GPU 1.99 % 7.59 % 2.03 %

The Pixel 11 Pro ran the earlier _ekv2048 for this check only, not in timing processes. On the Galaxy S26, the current _ekv2048 ran the timing processes of the Performance table on 2026-11-18; their transcripts of the three clips were identical to a Mac run of the earlier _ekv2048 with the same backend in 4 of 4 processes.

Translation and keyword biasing, against the source model's text: translate the speech to English. on one French, one German and one Spanish FLEURS clip gave the source model's words on the French clip with the language model on either backend and on the Spanish clip with it on the GPU; the German clip differs by one word on both backends ("cultural-determinist" for "cultural-deterministic") and the Spanish clip by one name on the CPU. transcribe the speech to text. Keywords: … on 3 LibriSpeech clips gave the source model's text on 2 of 3 on both backends; the third differs by one word and contains the same keywords as the source model's.

granite-speech-5.0-2b.litertlm: measurements from 2026-11-06

These are the previous card's numbers. Re-timed on 2026-11-07 in the series of runs of the Performance table's granite-speech-5.0-2b.litertlm rows, the same file reached the first text in 5.16 s (language model on the CPU) and 3.83 s (GPU) against 4.36 s and 7.47 s here; the phone's throttling differed between the two days, and we did not separate the cause.

Galaxy S26 (Android 16), LiteRT-LM 0.19.1 (the litert-lm-api 0.19.1 Android wheel from PyPI), 2026-11-06. Prefill and decode: litert_lm.Benchmark with a forced prefill of 1,024 tokens, then 256 decoded tokens (at most 2,048 tokens), the mean of runs 2 and 3 in two processes. Clip: one streaming call per clip with a new conversation each, the mean of 2 calls after a warm-up clip. Encoder: the audio encoding inside the call (the call's time minus its prefill and decode), the mean over a 5.9, a 9.9 and a 29.4-second clip. Peak memory: the highest peak resident memory (VmHWM) of our processes on that backend, the accuracy runs included (the timing process itself peaked at 5.1 GB on the CPU and 3.9 GB on the GPU); GPU buffers not included. The audio encoder is on the CPU in both rows, at the runtime's default CPU thread count. The phone's thermal governor capped the CPU frequency in almost every sample of these runs, and in the GPU row the GPU clock as well (thermal status up to 3), so these are throttled numbers; in the GPU row the CPU encoder took 5.6 s against 3.7 s in the CPU row, and we did not separate the cause. Creating the engine is not in these numbers: about 4.3 s on the CPU and 10.0 s on the GPU in a cold process (0.6 s and 6.7 s with the runtime's caches present). On the Pixel 11 Pro of the accuracy check, whose CPU was also capped, the same 9.9-second clip took 32.5 s with the language model on the CPU and 27.7 s on the GPU.

Phone Language model on Prefill (tokens/s) Decode (tokens/s) Encoder, one 36 s window 9.9 s clip: first text / full transcript Peak memory
Galaxy S26 CPU 137.9 9.87 3,686 ms 4.36 s / 6.50 s 6.2 GB
Galaxy S26 GPU 203.5 14.84 5,619 ms 7.47 s / 9.54 s 3.9 GB

Mac M4 Max (128 GB), LiteRT-LM 0.19.1 from PyPI. Language model: litert-lm benchmark granite-speech-5.0-2b.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend <cpu|gpu>. Clip: a 10.0-second FLEURS clip, the third clip of each process (model loaded, two clips already transcribed), median of 3 processes; the audio encoder on the CPU in every row. Encoder: the file's audio encoder graph alone in the ai-edge-litert 2.3.0 interpreter (XNNPACK, 4 threads), one 36-second window, median of 5.

Language model on Prefill (tokens/s) Decode (tokens/s) Encoder, one 36 s window 10 s clip: first text / full transcript
GPU 2,351.68 84.18 1,069.5 ms 1.18 s / 1.46 s
CPU 500.55 26.65 1,069.5 ms 1.41 s / 2.22 s

Accuracy on the 220 clips above (Mac M4 Max, the audio encoder on the CPU, the same scoring and thresholds):

Language model on English WER Japanese CER French + German + Spanish + Portuguese WER
CPU 4.53 % (72/1,588) 5.02 % (135/2,689) 11.93 % (282/2,363)
GPU 4.72 % (75/1,588) 5.32 % (143/2,689) 12.06 % (285/2,363)

The transcripts are character for character the source model's on 181 of the 220 clips with the language model on the CPU and on 209 with it on the GPU. On the phones, the same 38-clip subset with the audio encoder on the CPU met the same kind of thresholds in all four runs, and in each run all 38 replies stopped on their own, none empty or repeating:

Run English WER Japanese CER French + German + Spanish + Portuguese WER
Galaxy S26, language model on the CPU 2.43 % 6.11 % 2.03 %
Galaxy S26, language model on the GPU 2.43 % 7.26 % 2.03 %
Pixel 11 Pro, language model on the CPU 2.21 % 7.43 % 2.03 %
Pixel 11 Pro, language model on the GPU 2.43 % 7.26 % 2.03 %

With the language model on the GPU, both phones' transcripts of the 38 clips are identical to a Mac GPU reference run of this file on 38 of 38 clips; with it on the CPU, on 33 of 38 on both phones (the same 5 clips differ). Those references are separate Mac runs made for this comparison (CPU at one thread; GPU the text at least two of three processes agreed on), not the 220-clip runs above.

Translation and keyword biasing against the source model's text: translate the speech to English. on the same three FLEURS clips gave the source model's translation word for word on the Mac GPU (3 of 3) and on 2 of 3 on the Mac CPU (one word differs); the keyword prompt on the same 3 LibriSpeech clips gave the source model's text on both (3 of 3).

Limitations

  • The audio encoder computes a full window for every clip, also for a short one (24 s in _ekv2048, 36 s in granite-speech-5.0-2b.litertlm). A 12-second window was faster but missed the Japanese threshold, so it was not shipped.
  • No NPU file: an audio encoder compiled for the Galaxy S26 NPU turned the English part of multilingual_sample.wav into French. The cause was the NPU's sigmoid, which returns 0 for inputs below about −5.7; rewriting the SiLU of the 32 conformer feed-forward blocks with exp fixed it (that clip kept both languages with the language model on the CPU and on the GPU, one run each). The fixed encoder is slower on clips up to 24 s: its 36-second window takes about 1.6 s per call on the NPU, against 0.8 s for _ekv2048's 24-second window on the GPU. In one series of runs on the phone, with the language model on the GPU, the first text of the 9.9-second clip came after 2.16 s with it and after 1.32 s with an earlier _ekv2048 (the one this repo held on 2026-11-07; mean of 2 processes × 2 calls each); on a 29.4-second clip, which takes two GPU windows, the two were level. So it is not published. No int4 file: on a Mac CPU an int4 language model prefilled at 42 % of int8's rate (230 against 548 tokens/s), which lengthens the time to the first text.
  • With the language model on the CPU, LiteRT-LM quantizes the activations to int8 at run time: in one run per backend the transcripts of the earlier _ekv2048 were character for character the source model's on 180 of 220 clips against 195 on the GPU, at error rates within 0.3 points of each other (Accuracy). A second CPU process can print a slightly different transcript of the same clip (in our check of granite-speech-5.0-2b.litertlm, two more commas).
  • The context is 2,048 tokens in _ekv2048 and 4,096 in granite-speech-5.0-2b.litertlm, prompt and reply together; a message that does not fit is refused, not cut (How to send audio).
  • On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error.

Conversion notes

  • Source: ibm-granite/granite-speech-5.0-2b at revision 98160169. out_llm.safetensors (the encoder's BPE CTC head) is not used: the speech-to-text model in transformers does not load it.
  • Runtime: every measurement of the files on this card used LiteRT-LM 0.19.1, which runs the graphs on LiteRT, except the encoder-alone times (the encoder graph alone in ai-edge-litert 2.3.0) and the runs with the file of 2026-11-17, which name their runtime where they are reported.
  • Language model: litert-torch 0.10.0 export_hf of the Granite decoder taken out of the checkpoint, with the token embedding exported separately, prefill signatures 64, 256 and 1,024, quantization recipe dynamic_wi8_afp32; KV cache 2,048 for _ekv2048 and 4,096 for granite-speech-5.0-2b.litertlm. The embedding multiplier (×12) stays in the language model graph, where it also scales the audio rows, as in transformers.
  • Audio encoder: a PyTorch module that computes what the source's feature extractor, encoder and projector compute on the unpadded clip (the same mel filterbank and window, the clip length taken from the audio, masks for the partial attention block and the padded frames), traced with litert-torch; float16 weights. On the 220 clips above the PyTorch module (in float32) gave the source's number of audio tokens on every clip, and its output fed to the source language model gave the source's transcript on every clip. For _ekv2048 the same computation is written with tensors of rank 4 or less (the relative-position shift as a skew, the convolutions as 2-D convolutions, no GATHER_ND, BROADCAST_TO or DIV), so the GPU delegate takes the whole graph, and exported for a 24-second window; both encoders were judged on the 220 transcripts above.
  • Packed with litert-lm-builder 0.19.1 (litert-torch 0.10.0's own packing calls a builder API that 0.19 no longer has).
  • Reproduce: the scripts are in granite_speech_2b_work/ of hf-to-litertlm, and its REPRODUCE.md has the commands for each file: granite-speech-5.0-2b.litertlm and granite-speech-5.0-2b_ekv2048.litertlm. Each was run from a clean clone (the first file's commands on 2026-11-06, the second file's on 2026-11-07), and each run rebuilt every section of its file with the same bytes. Only the header differs: the packer writes a new uuid and creation time on every run (39 bytes of the first file's header, 41 of the second's), so a rebuilt file's own sha256 never matches the published one. Those commands rebuild the files as they were before 2026-11-09: both were re-packed on 2026-11-09 (UTC) with their language-model weights in TFLiteWeights sections at the end of the file (REPRODUCE.md, 2026-11-09, litert-lm-builder 0.19.1), and granite-speech-5.0-2b.litertlm was replaced again on 2026-11-17 (next section).

2026-11-17 — GPU graph (weights unchanged)

In granite-speech-5.0-2b.litertlm the prefill/decode graph now has the GPU form: 40 odml.cache_update composites for the KV cache and 80 attention composites. The previous file had neither; a check of its graph counted 80 DYNAMIC_UPDATE_SLICE and 80 BATCH_MATMUL ops instead. The 1,024-token prefill signature, the KV-cache length (4,096) and the int8 weight quantization read the same before and after. granite-speech-5.0-2b_ekv2048.litertlm was not replaced: its graph keeps the earlier form.

Only the prefill/decode graph section differs from the previous file: it went from 39,150,352 B to 39,656,200 B. The seven other sections (the two metadata sections, the tokenizer, the embedder graph, the audio encoder and the two weight sections) have the same size and sha256 as in the previous file. The file went from sha256 09e61db2… (2,846,640,128 B) to 6ab95431… (2,847,148,032 B).

The two files were compared on a Mac M4 Max with LiteRT-LM 0.19.1 on 2026-11-16, on the clips of the Accuracy section, with greedy decoding and the audio encoder on the CPU. With the language model on the GPU they gave the same reply on all 220 clips, so the error rates are the same for both: English WER 4.72 %, Japanese CER 5.32 %, the four European languages together 12.06 %. With it on the CPU at 1 thread, on 35 of the clips (the first 5 of each of the 7 sets), every reply was the same byte for byte.

With litert-lm 0.19.0, 0.19.1 and 0.19.2 on the Mac, on the CPU and on the GPU, the new file and the previous file both transcribed two test clips, to the same text byte for byte. The phone runs are in the last part of Performance.

Updates

  • 2026-11-07: added granite-speech-5.0-2b_ekv2048.litertlm. Its audio encoder runs on the GPU and reads 24-second windows, and its language model has a 2,048-token context and no activation-type declaration; on a Galaxy S26 the 9.9-second clip reached the first text in 1.21 s against 3.83 s for granite-speech-5.0-2b.litertlm in the same series of runs (both with the language model on the GPU). granite-speech-5.0-2b.litertlm is unchanged (c90a3dae…, 2,846,558,336 B) and stays the file for messages of more than about 2 minutes of audio. New: bf8eb5a8…, 2,824,395,760 B; the tokenizer section is the same bytes. Checked with LiteRT-LM 0.19.1, greedy decoding: 220 clips on a Mac M4 Max with the language model on the CPU and on the GPU (both inside the thresholds), 38 clips on a Pixel 11 Pro on both backends (inside the thresholds), and the source card's sample on a Galaxy S26 on both backends (both languages kept).

Known runtime issues

Measurements of granite-speech-5.0-2b_ekv2048.litertlm on this page made before 2026-11-09 used earlier files, not the current one.

Android GPU with LiteRT-LM v0.18.0 and v0.18.1. granite-speech-5.0-2b_ekv2048.litertlm and granite-speech-5.0-2b.litertlm keep some of their weights outside the model graph (LiteRT's external weights), and on a Pixel 8a the GPU backend of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on a Qwen3-0.6B file with external weights (LiteRT-LM#3986).

License

Apache License 2.0, the same license as the source model. The files here are converted from ibm-granite/granite-speech-5.0-2b: the weights are quantized, and the prompt template is written for LiteRT-LM as described above.

Downloads last month
461
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for litert-community/granite-speech-5.0-2b

Quantized
(9)
this model