# Llama 3 performance is 4 mins. can get it in seconds?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-3-performance-is-4-mins-can-get-it-in-seconds%2F95479%3E%5C
**Category:** Models\
**Created:** [July 4, 2024, 8:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-3-performance-is-4-mins-can-get-it-in-seconds%2F95479 "2024-07-04T08:03:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Magesh78](https://avatars.discourse-cdn.com/v4/letter/m/cc9497/32.png) [@Magesh78](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMagesh78)%5C
**Post date:** [July 4, 2024, 8:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-3-performance-is-4-mins-can-get-it-in-seconds%2F95479%2F1 "2024-07-04T08:03:23Z")

</div>

Hello Everyone,

Model - llama\_3\_8B\_Instruct  
Hardware - VM 32GB RAM and 16 GB (one GPU-Tesla T4)

I want to extract structured information in JSON format from the email input data. I used transformer library to implement the model. I set max\_new\_tokens as 300 and temperature as 0.01 for my output generation. Apart from pre and post processing of input data, model.generate() method takes long time to execute, so my output response time around 4 mins.

GPU utilization shows 80 percent when running the model.

Do you guys think that its reasonable processing time?  
Is it possible reduce the output generation time under 1 min or in few seconds? (without GPU upgrade)  
Kindly pour your thoughts on it?

Thanks.

---

<div class="post-metadata">

**Author:** ![sriram6399](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sriram6399](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsriram6399)%5C
**Post date:** [March 23, 2025, 10:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-3-performance-is-4-mins-can-get-it-in-seconds%2F95479%2F2 "2025-03-23T22:05:15Z")

</div>

Experiencing a similar issue. Any suggestions or fixes are greatly appreciated.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 24, 2025, 3:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-3-performance-is-4-mins-can-get-it-in-seconds%2F95479%2F3 "2025-03-24T03:50:10Z")

</div>

If you have 16GB of VRAM, it won’t be enough for the 8B model. Some of it will be offloaded to regular RAM, and it will be very slow. We recommend using quantization as follows, or using a program like Ollama that basically uses quantization.

### Quantization with Transformers

```auto
pip install bitsandbytes

```

```py
from transformers import pipeline, BitsAndBytesConfig
import torch

nf4_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
pipe = pipeline("text-generation", model=model_id, quantization_config=nf4_config)

```

> **[Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Overview](https://huggingface.co/docs/transformers/main/quantization/overview)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

### Ollama

> **[Use Ollama with any GGUF Model on Hugging Face Hub](https://huggingface.co/docs/hub/ollama)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
