# \-inf values for logit score outputs with model.generate

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 27, 2023, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108 "2023-12-27T03:52:20Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinusankars](https://avatars.discourse-cdn.com/v4/letter/v/67e7ee/32.png) [@vinusankars](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvinusankars)%5C
**Post date:** [December 27, 2023, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108%2F1 "2023-12-27T03:52:20Z")

</div>

I have been trying to obtain the logit values for each token output as follows. However, the output that I get is a vector with `-inf` in all positions except at the position of the predicted token. This is observed when `generation_config.do_sample = True`. When `generation_config.do_sample = False`, the output that I get is a vector with definite values in it.

I am confused since I thought the values when `do_sample=True` should be definite and not `-inf` for any non-deterministic token sampling to work well.

Can someone please explain why this is happening?

Here is the code:

```auto
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, device_map='auto')

model.generation_config.do_sample = True
model.generation_config.temperature = 0.6

sen = "[INST] What is Sun? [/INST]"

output = model.generate(**tokenizer(sen, return_tensors='pt').to(0), max_length=20, output_scores=True, return_dict_in_generate=True)    
print(output.scores)

```

Output screenshot:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/8/28b11a2b0aa46219472aa946b2353eb66135b416.png)

---

<div class="post-metadata">

**Author:** ![mbismay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mbismay/32/25267_2.png) [@mbismay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmbismay)%5C
**Post date:** [May 2, 2024, 7:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108%2F2 "2024-05-02T19:28:03Z")

</div>

Same issue! Could anyone please explain how to tackle this?

---

<div class="post-metadata">

**Author:** ![LazerLambda](https://avatars.discourse-cdn.com/v4/letter/l/7ba0ec/32.png) [@LazerLambda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLazerLambda)%5C
**Post date:** [September 20, 2024, 12:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108%2F3 "2024-09-20T00:02:12Z")

</div>

Same problem here! Any solutions yet?

---

<div class="post-metadata">

**Author:** ![Kripi](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@Kripi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKripi)%5C
**Post date:** [January 2, 2025, 3:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Finf-values-for-logit-score-outputs-with-model-generate%2F67108%2F4 "2025-01-02T15:40:29Z")

</div>

To get raw logits, pass `output_logits` instead of `output_scores` to `generate()` and use `output.logits`.
