# Making llama text generation, deterministic

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaking-llama-text-generation-deterministic%2F50437%3E%5C
**Category:** Models\
**Created:** [August 11, 2023, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaking-llama-text-generation-deterministic%2F50437 "2023-08-11T13:52:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![rachith](https://avatars.discourse-cdn.com/v4/letter/r/e0b2c6/32.png) [@rachith](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frachith)%5C
**Post date:** [August 11, 2023, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaking-llama-text-generation-deterministic%2F50437%2F1 "2023-08-11T13:52:37Z")

</div>

Following the text generation code template [here](https://huggingface.co/docs/transformers/main/model_doc/llama2), I’ve been trying to generate some outputs from llama2 but running into stochastic generations.  
For instance, running the same prompt through the model.generate() twice results in two different outputs as shown in the example below.

I’ve used model.generate() with other LLMs (e.g., flant5) with the other parameters remaining the same and have obtained deterministic outputs.  
Also tried `AutoModelForCausalLM` instead of `LLamaForCausalLM` but still got different outputs each time for the same prompt.

How do I make sure I get the same text generated each time?

Code to reproduce:

```auto
from transformers import AutoTokenizer, LlamaForCausalLM

model_name = "meta-llama/Llama-2-13b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, cache_dir="/data2/racball/llms")
model = LlamaForCausalLM.from_pretrained(
    model_name,
    cache_dir="/data2/racball/llms",
    device_map = "sequential",
)

prompt = "What is up?"
inputs = tokenizer(prompt, return_tensors="pt")
# Generate
generate_ids = model.generate(inputs.input_ids, max_length=30)

tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

# Run1: 'What is up?\n\nI have a problem with my `docker-compose.yml` file. I have a service that should run a'
# Run2: "What is up?\n\nIt's been a while since I've posted, but I've been pretty busy with work and other"

```

---

<div class="post-metadata">

**Author:** ![rachith](https://avatars.discourse-cdn.com/v4/letter/r/e0b2c6/32.png) [@rachith](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frachith)%5C
**Post date:** [August 16, 2023, 2:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmaking-llama-text-generation-deterministic%2F50437%2F2 "2023-08-16T02:02:05Z")

</div>

Resolved [here]  
([Is non-determinism in outputs generated by LlamaForCausalLM, the expected behavior? · Issue #25507 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/25507#issuecomment-1678498896))

TLDR: set `do_sample = False` in `model.generate(inputs.input_ids, max_length=30)` as in `model.generate(inputs.input_ids, max_length=30,do_sample=False)`