# Compute log probabilities of any sequence provided

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%3E%5C
**Category:** Beginners\
**Created:** [November 15, 2021, 7:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710 "2021-11-15T07:32:45Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![PastelBelem8](https://avatars.discourse-cdn.com/v4/letter/p/e274bd/32.png) [@PastelBelem8](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPastelBelem8)%5C
**Post date:** [November 15, 2021, 7:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F1 "2021-11-15T07:32:45Z")

</div>

Hey there!

I’m using `allenai/unifiedqa-t5-small` model to obtain the log probabilities of a given sequence (which is not necessarily the one generated by the model). In particular, I’m interested in having the probability distribution that is conditioned on the previous tokens in the sequence.

So far, I’ve been using the forward method and providing the sentence I want to obtain the logits for as the `labels` argument. However, I am not 100% confident the resulting logits are conditioned on the sentence I provided as labels (i.e., is the forward method working in a teacher forcing fashion and, thus guarantees that log probabilities are conditioned on the `labels` parameter, rather than the argmax)?

A second question is whether it is necessary to provide the pad\_input\_id character as the first character in the `labels` argument.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [May 18, 2022, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F2 "2022-05-18T21:32:07Z")

</div>

Hey @PastelBelem8,

Would this PR: [[Generation] Fix Transition probs by patrickvonplaten · Pull Request #17311 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/17311) and this forum post: [Generation Probabilities: How to compute probabilities of output scores for GPT2](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgeneration-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2%2F3175) help?

---

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSeohyeong)%5C
**Post date:** [January 28, 2023, 7:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F3 "2023-01-28T07:33:09Z")

</div>

Hey @PastelBelem8, have you figured out how to get the log probabilities of any given sequence yet?

I’m trying to implement the same thing using the T5 model. My approach was to use [`.compute_transition_scores()`](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores) and reconstruct the sequence score.

As written in the documentation, `.compute_transition_scores()` takes `sequences` and `scores` and returns the transition score for each token id in the sequences provided. The snippet below takes the output of the `model.generate()` and get transitions scores on the generated example. However, we could pass token\_ids for the wanted sequence to `.compute_transition_scores()` instead of `outputs.sequences`.

```auto
from transformers import AutoModelWithLMHead, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llenai/unifiedqa-t5-small")
model = AutoModelWithLMHead.from_pretrained("llenai/unifiedqa-t5-small")
inputs = tokenizer(["question: <q> context: <c>"], return_tensors="pt")

outputs = model.generate(inputs.input_ids, return_dict_in_generate=True, output_scores=True)

# instead of this
transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=False
)

# do this
wanted_seq = tokenizer.batch_encode_plus(["wanted sequence"], return_tensors="pt").input_ids
wanted_seq = torch.cat([torch.tensor([[0]]), tar_input_ids], dim=1)
transition_scores = model.compute_transition_scores(
    wanted_seq, outputs.scores, normalize_logits=False
)

```

However, I’ve realized when the length of the wanted\_seq \> length of the generated sequence from `.generate()`, `outputs.scores` does not cover the full length of the wanted\_sequence input\_ids. So I’ve been trying to modify the behavior of `.generate()` to get the full length of scores by passing `stopping_criteria`. However, I still haven’t figured out how to do so.

What I’d like to have is the scores of the length I wanted (up to 32 in the case below, without early stopping). Can @patrickvonplaten help me with this? I’ve tried below but it’s throwing me an error.

```auto
from transformers.generation.stopping_criteria import StoppingCriteriaList, MaxLengthCriteria

stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(32)])
            outputs = model.generate(
                inputs['input_ids'].cuda() if torch.cuda.is_available() else inputs["input_ids"],
                num_beams=args.num_beams,
                max_length=32,
                stopping_criteria=stopping_criteria,
                early_stopping=False,
                output_scores=True,
                return_dict_in_generate=True,
            )

```

---

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCbelem)%5C
**Post date:** [January 28, 2023, 5:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F4 "2023-01-28T17:14:53Z")

</div>

Hey!

I did find a way to compute those scores! I think the [new release of HuggingFace](https://github.com/huggingface/transformers/releases/tag/v4.26.0) had significant changes in terms of computing scores for sequences (I haven’t tried computing the scores yet).

If you still want to use your method I would suggest you try specifying the argument for `min_length` during generate which leads to generations longer than that specified number.

From what I gathered you’re trying to compute the log probabilities of any provided sequence, right? Over the time I’ve been iterating over what other ways we can use to compute those probabilities and I gathered them in [this notebook](https://colab.research.google.com/drive/1Q8VAwCPB12ZzYH79nAuiiSSnVDkZ2-u7?usp=sharing).

I hope this helps 🙂

---

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSeohyeong)%5C
**Post date:** [February 2, 2023, 8:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F5 "2023-02-02T08:29:53Z")

</div>

Hey @Cbelem! Thank you for sharing the notebook!

---

<div class="post-metadata">

**Author:** ![dpernes](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@dpernes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdpernes)%5C
**Post date:** [February 17, 2023, 6:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F6 "2023-02-17T18:00:30Z")

</div>

Hey @Cbelem,

Thank you for sharing this notebook.

I wonder if you have compared the `sequences_scores` output by the `generate()` method with the `loss` output by the `forward()` when you provide the generated sequence as `labels`. Assuming `forward()` does teacher forcing as it should, the `loss` should be the negative log-probability of the sequence normalized by its length, so I was expecting `loss * sequence_length` and `sequences_scores` to be the same in absolute value, but it turns out they are not.

Minimal example:

```auto
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer

device = torch.device('cuda' if torch.cuda.is_available() else "cpu")

# load a T5-small model
model = T5ForConditionalGeneration.from_pretrained('t5-small').to(device)
tokenizer = T5Tokenizer.from_pretrained('t5-small', model_max_length=512)
model.eval()

# define some source text and tokenize it
source_text = "This is a source sentence."
source_ids = tokenizer(source_text, return_tensors="pt").input_ids.to(device)

# generate the output using beam search
gen_outputs = model.generate(
    inputs=source_ids,
    num_beams=2,
    min_length=0,
    max_length=512,
    length_penalty=0,
    output_scores=True,
    return_dict_in_generate=True,
)

# compute the loss for the generated sequence
loss = model(
    input_ids=source_ids,
    attention_mask=torch.ones_like(source_ids),
    labels=gen_outputs.sequences,
    return_dict=True
).loss.item()

# compare the scores given by generate() with the loss given by forward()
print('scores:', gen_outputs.sequences_scores.item())
print('loss * seq_len:', loss * gen_outputs.sequences.shape[-1])
print('loss:', loss)

```

This will output:

```auto
scores: -3.2493550777435303
loss * seq_len: 13.989073991775513
loss: 1.5543415546417236

```

I don’t see any reason for the mismatch. Am I missing something obvious?

---

<div class="post-metadata">

**Author:** ![Cbelem](https://avatars.discourse-cdn.com/v4/letter/c/65b543/32.png) [@Cbelem](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCbelem)%5C
**Post date:** [February 19, 2023, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F7 "2023-02-19T16:38:55Z")

</div>

Hi @dpernes !

This is a great question! In the past, I’ve seen similar threads about [sequences-scores do not compute what is expected for beam\_search](https://github.com/huggingface/transformers/issues/16413#issuecomment-1088907369) and also this one about [the mismatch between the log probabilities of the beam search and the compute transitions](https://github.com/huggingface/transformers/issues/15869#issuecomment-1136690454). I believe they [updated this API](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fannouncement-generation-get-probabilities-for-generated-output%2F30075) and is now easier to get these scores.

I’m going to look into your examples and try to understand a bit better what’s going on. What transformers version are you using?

---

<div class="post-metadata">

**Author:** ![dpernes](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@dpernes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdpernes)%5C
**Post date:** [February 20, 2023, 10:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F8 "2023-02-20T10:46:01Z")

</div>

Hi @Cbelem! Thank you for your help 🙂

> I believe they [updated this API](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fannouncement-generation-get-probabilities-for-generated-output%2F30075) and is now easier to get these scores.

Yes, I tried with the new function `compute_transition_scores` and the scores match those provided by `generate`, but the mismatch with the loss persists. Maybe @joaogante can explain the mismatch.

> What transformers version are you using?

I am using v4.26.1

Minimal example (updated with `compute_transition_scores`):

```auto
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer

device = torch.device('cuda' if torch.cuda.is_available() else "cpu")

# load a T5-small model
model = T5ForConditionalGeneration.from_pretrained('t5-small').to(device)
tokenizer = T5Tokenizer.from_pretrained('t5-small', model_max_length=512)
model.eval()

# define some source text and tokenize it
source_text = "This is a source sentence."
source_ids = tokenizer(source_text, return_tensors="pt").input_ids.to(device)

# generate the output using beam search
gen_outputs = model.generate(
    inputs=source_ids,
    num_beams=2,
    min_length=0,
    max_length=512,
    length_penalty=0,
    output_scores=True,
    return_dict_in_generate=True,
)

# compute the scores using compute_transition_scores()
scores = model.compute_transition_scores(
    sequences=gen_outputs.sequences,
    scores=gen_outputs.scores,
    beam_indices=gen_outputs.beam_indices,
)

# compute the loss for the generated sequence
loss = model(
    input_ids=source_ids,
    attention_mask=torch.ones_like(source_ids),
    labels=gen_outputs.sequences,
    return_dict=True
).loss.item()

# compare the scores given by generate() with the loss given by forward()
print('scores (generate):', gen_outputs.sequences_scores.item())
print('scores (compute_transition_scores):', scores.sum().item())
print('loss * seq_len:', loss * gen_outputs.sequences.shape[-1])
print('loss:', loss)

```

Output:

```auto
scores (generate): -3.2493550777435303
scores (compute_transition_scores): -3.2493550777435303
loss * seq_len: 13.989073991775513
loss: 1.5543415546417236

```

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjoaogante)%5C
**Post date:** [February 21, 2023, 5:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F9 "2023-02-21T17:25:11Z")

</div>

Before replying to the other points, I want to highlight the following:

⚠ Please do not use the method in [this comment](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F3) (cc @Seohyeong) to get scores for an arbitrary sequence.

Text generation is auto-regressive: it predicts the next tokens based on the tokens predicted so far. The `scores` field in the output contains the logits for all tokens in the vocabulary at each position. Yes, you can obtain a score for any token. But that score is only correct if the preceding tokens are exactly the same! In [this colab](https://colab.research.google.com/drive/1JQ1upJeyHNHv_BXKNfoKw7V_lwG55h4T?usp=sharing) you will see that if we change the model inputs (i.e. the source of the `scores`), their values will change for a few selected tokens.

👉 See [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fannouncement-generation-get-probabilities-for-generated-output%2F30075%2F17) for an example on how to compute the token-level scores for any sequence 🙂

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjoaogante)%5C
**Post date:** [February 21, 2023, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F10 "2023-02-21T18:14:20Z")

</div>

Hey @dpernes 👋 A tiny correction is needed in your example, and we get the matching numbers as expected. In a nutshell, `gen_outputs.sequences` contains the BOS token, which is not predicted by the model – all outputs start with it. So, when setting `labels`, we have to skip it 😃

Translated to code, replace the last lines of your script with

```python
# compute the loss for the generated sequence
loss = model(
    input_ids=source_ids,
    attention_mask=torch.ones_like(source_ids),
    labels=gen_outputs.sequences[:, 1:], # skip BOS token
    return_dict=True
).loss.item()

# compare the scores given by generate() with the loss given by forward()
print('scores (generate):', gen_outputs.sequences_scores.item())
print('scores (compute_transition_scores):', scores.sum().item())
print('loss * seq_len:', loss * (gen_outputs.sequences.shape[-1] - 1)) # correct length
print('loss:', loss)

```

The whole script now outputs:

```auto
scores (generate): -3.249357223510742
scores (compute_transition_scores): -3.2493574619293213
loss * seq_len: 3.2493574619293213
loss: 0.40616968274116516

```

---

<div class="post-metadata">

**Author:** ![dpernes](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@dpernes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdpernes)%5C
**Post date:** [February 22, 2023, 11:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F11 "2023-02-22T11:31:20Z")

</div>

Oh, I didn’t realize you were including the BOS token in the generated sequence. Makes a lot of sense, now, thank you! 🙂

---

<div class="post-metadata">

**Author:** ![YWDallas](https://avatars.discourse-cdn.com/v4/letter/y/7993a0/32.png) [@YWDallas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYWDallas)%5C
**Post date:** [February 7, 2024, 11:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F12 "2024-02-07T23:38:15Z")

</div>

Thank you so much!

If I were to calculate the perplexity score of the generated sequence, exp(loss) based on your calculation should be correct, right?

---

<div class="post-metadata">

**Author:** ![mohit-raghavendra](https://avatars.discourse-cdn.com/v4/letter/m/cdc98d/32.png) [@mohit-raghavendra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmohit-raghavendra)%5C
**Post date:** [February 8, 2024, 12:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F13 "2024-02-08T00:23:31Z")

</div>

I am trying to reproduce the sentiment generation task in the DPO paper.

It states that the reward can be modeled as below, with π(y|x) being the sequence probability from a finetuned policy model.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/6/4/643942f61c6e1c98269839bcec81db04412adb9c.png)

How do I compute this π(y|x) ? I am assuming that I can get this by calling generate on the finetuned model and computing sequence probabilities.

I have this current setup to sample 4 sequences and get the sequence probability for all 4 generations:

```python
# Load a GPT-2 model and tokenizer
prompt = ["Enchanted was a movie "]
input_ids = loaded_tokenizer.encode(prompt, return_tensors="pt").to(device)
output = loaded_model.generate(
    input_ids, 
    max_length=100, 
    top_k=50,
    top_p=0.95,
    do_sample=True,
    num_return_sequences=4,
    return_dict_in_generate=True, 
    output_scores=True,
    pad_token_id=50256)

logits = loaded_model.compute_transition_scores(
    output.sequences, output.scores, normalize_logits=True)

log_probs = logits.sum(dim=1)

```

Am I correct in assuming that logits are log-probs for each generated token for each sequence? And if I want the overall probability for each generated sequence, I can sum up the logits for each sequence?

---

<div class="post-metadata">

**Author:** ![jaydeepb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jaydeepb/32/14906_2.png) [@jaydeepb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaydeepb)%5C
**Post date:** [May 17, 2024, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F14 "2024-05-17T03:47:00Z")

</div>

@joaogante could we use that method if we have the same prompts (preceding tokens) and target sequence but different models? In other words, for different models, I’d like to compute log probs for a sequence given the prompt.

---

<div class="post-metadata">

**Author:** ![Wafaa-mohammed](https://avatars.discourse-cdn.com/v4/letter/w/59ef9b/32.png) [@Wafaa-mohammed](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FWafaa-mohammed)%5C
**Post date:** [June 12, 2024, 6:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F15 "2024-06-12T18:08:36Z")

</div>

Hi @jaydeepb, I am trying to do the same, I want to get log probabilities of an output sentence given a prompt, my model of interest is LLaMA-2. have you figured how to do it? I would appreciate it if you could share the script.

---

<div class="post-metadata">

**Author:** ![Wafaa-mohammed](https://avatars.discourse-cdn.com/v4/letter/w/59ef9b/32.png) [@Wafaa-mohammed](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FWafaa-mohammed)%5C
**Post date:** [August 13, 2024, 1:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F16 "2024-08-13T13:49:25Z")

</div>

Can someone double check if what I am doing here is correct?

```auto
input_tokens = tokenizer.encode(source_text, add_special_tokens=False, return_tensors="pt").to(device)
input_tokens_updated = input_tokens.clone().to(torch.int64).to(device)
output_tokens = tokenizer.encode(tgt_text, add_special_tokens=False, return_tensors="pt")[0].to(device)
log_sum = 0

for i in range(len(output_tokens)):
    # Predict with the given model
    with torch.no_grad():
        outputs = model.generate(input_tokens_updated, max_new_tokens=1, output_logits=True, return_dict_in_generate=True)
        logit_predictions = outputs.logits[0]

    # Extract the log probability of the output token
    token = tokenizer.decode(output_tokens[i])
    log_probs = torch.nn.functional.log_softmax(logit_predictions, dim=-1)
    out_token_logit = logit_predictions[0, output_tokens[i]]
    out_token_log_prob = log_probs[0, output_tokens[i]]
    log_sum += out_token_log_prob
    print(f"Token: {token}", "logit: ", out_token_logit, "log prob: ", out_token_log_prob)

    # Incrementally add an output token to the current sequence
    input_tokens_updated = torch.cat([input_tokens_updated, output_tokens[i].reshape(1, 1)], dim=1)
    print([tokenizer.decode(token) for token in input_tokens_updated])
    print("============")
print(f"Total Log Sum Probability: {log_sum}")

```

at each step, I am generating one token and calculating the log probability of the target token I am interested in. I assume that the final log\_sum represents the log probability of the target sentence.

---

<div class="post-metadata">

**Author:** ![hlnchen](https://avatars.discourse-cdn.com/v4/letter/h/a9adbd/32.png) [@hlnchen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhlnchen)%5C
**Post date:** [August 21, 2024, 9:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F17 "2024-08-21T21:03:55Z")

</div>

Hi, assuming your batch of input sequence `batch` is left-padded, i.e. each row is like `<pad>{context}{response}`, where `<pad>` represents 0 to many padding tokens. And we would like to compute the probability that model generates response conditioned on context: `model(response|context)` or the log probability of that.

I believe a nicer implementation is the following, assuming you already have two tensors:

- input\_ids (batch\_size, seq\_len): the token ids of your input batch of context+responses, left padded, of shape
- context\_lengths (batch\_size, 1): the length of the pad tokens + context for each sequence

```auto
import torch
from trl.trainer.utils import forward

raw_outputs = forward(model, input_ids, pad_token_id) # Raw logits
token_distribution = torch.log_softmax(raw_outputs.logits, dim=-1) # transform to log probs
sequence_logprob = torch.gather(reward_all_logprob, 2, input_ids.unsqueeze(-1)).squeeze(-1) # Flatten to the entire sequence log probs

# Now we only need to sum the part from actual response using a mask from context length
indices = torch.arange(sequence_logprob.shape[1], device=sequence_logprob.device).repeat(sequence_logprob.shape[0], 1)
mask = (indices >= context_lengths) * (indices < sequence_logprob.shape[1] - 1)
response_logprob = (sequence_logprob * mask).sum(1).unsqueeze(1) # (batch_size, 1)

```

If your batch has a constant context length across all rows, then the masking can be avoided by directly slicing the raw logits which saves some computation.

Also, the helper function `forward` is defined as, which requires all sequences are left padded. However I am not sure that `torch.masked_fill` is necessary here. In my opinion this step is redundant.

```auto
def forward(
    model: torch.nn.Module,
    query_responses: torch.Tensor,
    pad_token_id: int,
) -> torch.nn.Module:
    """
    Performs a forward pass through the model with the given query responses and pad token ID.

    Args:
        model (`torch.nn.Module`):
            The model to perform the forward pass.
        query_responses (`torch.Tensor`):
            The tensor containing the query responses.
        pad_token_id (`int`):
            The token ID representing the pad token.

    Returns:
        `torch.nn.Module`:
            The output of the model, including hidden states.
    """
    attention_mask = query_responses != pad_token_id
    position_ids = attention_mask.cumsum(1) - attention_mask.long()
    input_ids = torch.masked_fill(query_responses, ~attention_mask, 0)
    return model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        position_ids=position_ids,
        return_dict=True,
        output_hidden_states=True,
    )

```

---

<div class="post-metadata">

**Author:** ![ageron](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ageron/32/47827_2.png) [@ageron](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fageron)%5C
**Post date:** [May 19, 2025, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompute-log-probabilities-of-any-sequence-provided%2F11710%2F18 "2025-05-19T10:21:16Z")

</div>

You can actually use the `F.cross_entropy()` function for this. Below is a code example that computes the conditional probability of an answer given a prompt. The answer is not the one the model would give, so I think that’s what the OP was asking for.

```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

# load the tokenizer & causal LM model (e.g., GPT-2)
model_id = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", torch_dtype="auto")

prompt = "What is the area of a unit circle?"
answer = "It's pi."
full_input = f"{prompt} {answer}"

# run the model and get the logits
input_ids = tokenizer(full_input, return_tensors="pt").input_ids
input_ids = input_ids.to(model.device)
all_logits = model(input_ids=input_ids).logits

# compute the conditional log probabilities of the chosen targets given the
# previous tokens. The targets are the next tokens (i.e., the shifted
# input_ids).
logits = all_logits[:, :-1].permute(0, 2, 1)
target_token_ids = input_ids[:, 1:]
cond_log_probas = -F.cross_entropy(logits, target_token_ids, reduction="none")

# Print each target token's conditional proba given the previous tokens
for token_id, cond_log_proba in zip(target_token_ids[0], cond_log_probas[0]):
    target_token = tokenizer.decode(token_id)
    cond_proba = torch.exp(cond_log_proba).item()
    print(f"{target_token}\t{cond_proba:.1%}")

# Compute the conditional log probability of the answer given the prompt
answer_start_index = max(len(tokenizer(prompt).input_ids) - 1, 0)
# Note that we subtracted 1 because the last token of the prompt is where
# the first answer token is predicted. max(..., 0) to avoid a negative index.
answer_log_proba = cond_log_probas[0, answer_start_index:].sum().item()
print(f"{answer_log_proba=}")

```

This should output:

```auto
 is	4.1%
 the	17.9%
 area	0.0%
 of	28.9%
 a	0.8%
 unit	0.0%
 circle	0.0%
?	23.0%
 It	2.4%
's	35.2%
 pi	0.0%
.	29.4%
answer_log_proba=-17.349838256835938

```
