# LLM performance

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-performance%2F141451%3E%5C
**Category:** Beginners\
**Created:** [February 17, 2025, 8:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-performance%2F141451 "2025-02-17T20:39:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![InXis](https://avatars.discourse-cdn.com/v4/letter/i/49beb7/32.png) [@InXis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FInXis)%5C
**Post date:** [February 17, 2025, 8:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-performance%2F141451%2F1 "2025-02-17T20:39:58Z")

</div>

Need help understanding published performance data for latest LLMs (o1-mini, deepseek, gemini, gpt, …)?

Is there a standard hardware configuration (1xH100) used to compare LLMs performance?

Are LLMs performance benchmarks based on theoretical/calculated speed?

- usually no hyperparameters, no hardware specifications, no input data considerations are published (only, …, we are the fastest)

My experience with 4090 on Windows with 7B models (no quant) - usually around 1 token per second on 4090.

Is the peformance much better on Linux (Nvidia Triton performance optimizations and other LInux only libraries), how much better?

As an theoretical example:  
NVidia RTX 4090 with a memory bandwidth of 1008 GB/s, reading 27 GB takes approximately 27 ms. Therefore, we can expect around 27 ms per token for tokens with low position numbers, with minimal impact from the KV-cache. If 8-bit weights are used, reading 13.5 GB takes about 13.3 ms. These estimates represent the theoretical minimum time per token.

Maximum expected speed is 1000ms / 27ms = 37 tokens per second (ideal situation) for 16-bit weights?

Thank You

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [February 18, 2025, 3:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllm-performance%2F141451%2F2 "2025-02-18T03:05:30Z")

</div>

There is no unified attempt yet. However, there are quite a few benchmarks and leaderboards. I hope this is helpful…

> **[GitHub - XiongjieDai/GPU-Benchmarks-on-LLM-Inference: Multiple NVIDIA GPUs or Apple Silicon for Large...](https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference)**
>
> Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?

> **[Spaces - Hugging Face](https://huggingface.co/spaces?category=model-benchmarking)**
>
> Discover amazing ML apps made by the community