# What is the difference between logits and scores?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%3E%5C
**Category:** Beginners\
**Created:** [April 2, 2024, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796 "2024-04-02T09:11:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![u7122029](https://avatars.discourse-cdn.com/v4/letter/u/49beb7/32.png) [@u7122029](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fu7122029)%5C
**Post date:** [April 2, 2024, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%2F1 "2024-04-02T09:11:17Z")

</div>

> **[Utilities for Generation](https://huggingface.co/docs/transformers/v4.39.3/en/internal/generation_utils#transformers.generation.GenerateDecoderOnlyOutput)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The documentation in the link above makes me believe that `scores` is just a “processed” version of `logits`. This begs the question: how exactly are these `logits` processed?

I took a sample of these `score`s myself, and they look no different in meaning to `logits`. They don’t look like probabilities since some of them are clearly negative or above 1.0.

```py
tensor([[-7.5898, -5.9922, 18.5625, ..., -8.4844, -4.7539, -4.6758]],
                             device='cuda:0'))

```

Can someone please explain to me what these scores really are?

---

<div class="post-metadata">

**Author:** ![swtb](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swtb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fswtb)%5C
**Post date:** [April 3, 2024, 1:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%2F2 "2024-04-03T13:00:22Z")

</div>

The links documentation is just the data class. I think the method of processing would vary based on where this class is used. I would guess in many cases the logits would just be softmaxed into a probability distribution.

---

<div class="post-metadata">

**Author:** ![li55555](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/li55555/32/31106_2.png) [@li55555](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fli55555)%5C
**Post date:** [September 1, 2024, 1:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%2F3 "2024-09-01T13:44:11Z")

</div>

I believe all the processing functions, including top-k and temperature operations, can be found in the file transformers/generation/logits\_process.py. They are called by class GenerationMixin.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [September 2, 2024, 7:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%2F4 "2024-09-02T07:40:30Z")

</div>

Scores are equal to logits in case of greedy decoding, but are different in case of more fancy decoding methods like contrastive decoding (in that case, the scores of the selected tokens are the logits after contrastive penalties and re-ranking have been applied).

The way logits are processed into scores depends on the decoding algorithm (see [this](https://huggingface.co/docs/transformers/en/generation_strategies#decoding-strategies) for an overview).

---

<div class="post-metadata">

**Author:** ![Moharrrr](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@Moharrrr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMoharrrr)%5C
**Post date:** [November 19, 2024, 5:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhat-is-the-difference-between-logits-and-scores%2F79796%2F5 "2024-11-19T17:51:20Z")

</div>

It seems they are the same (using the greedy)

```python
inpu_text = "What would happen if I eat Kharboze and honey together?"

inputs = tokenizer(inpu_text, return_tensors="pt").to(model.device)

gen_utilities = {
    "return_dict_in_generate": True,
    "output_scores": True,
    "output_logits": True,
    # "output_hidden_states": True,
    # "output_attentions": True,
}

gen_sampling = {
    "greedy": {"do_sample": False, "temperature": 1.0, "top_k": None, "top_p": None},
    "beam": {"num_beams": 3},
    "top_k": {"do_sample": True, "top_k": 50, "temperature": 1.0},
    "top_p": {"do_sample": True, "top_p": 0.8, "temperature": 0.7},
    "typical": {"typical_p": 0.95, "temperature": 1.0},
    "contrastive": {"penalty_alpha": 0.6, "top_k": 50},
}

max_new_tokens = 20

gen_out = model.generate( **inputs,** gen_utilities, **gen_sampling["greedy"], max_new_tokens=20)

scores = gen_out.scores
logits = gen_out.logits

for i in range(max_new_tokens):
    print(torch.equal(logits[i], scores[i]))

```

Output:

```bash
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True
True

```
