# Pre-train PEGASUS model from scratch

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%3E%5C
**Category:** Models\
**Created:** [March 18, 2021, 12:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544 "2021-03-18T00:04:40Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ithieund](https://avatars.discourse-cdn.com/v4/letter/i/f475e1/32.png) [@ithieund](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fithieund)%5C
**Post date:** [March 18, 2021, 12:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F1 "2021-03-18T00:04:40Z")

</div>

Hi @sgugger ,  
I want to do a pre-training PEGASUS model from scratch, can you five me some suggestion?

First, can I do this approach ([How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)) to train this model from scratch?  
Secondly, how can I control the \<mask\_1\> tokens to mask sentences (GSG objective)? And how can I specify the strategy of masking sentences like the selected one in PEGASUS paper?

Thank you very much!

---

<div class="post-metadata">

**Author:** ![ithieund](https://avatars.discourse-cdn.com/v4/letter/i/f475e1/32.png) [@ithieund](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fithieund)%5C
**Post date:** [March 21, 2021, 5:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F2 "2021-03-21T17:38:53Z")

</div>

Can anyone give me some suggestion?

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F3 "2021-03-22T15:47:26Z")

</div>

[https://github.com/huggingface/transformers/tree/master/examples/language-modeling](https://github.com/huggingface/transformers/tree/master/examples/language-modeling)

I think you can start from here to finetune pegausus on the language you want, you should choose causal language modeling for pegasus if i’m not mistaken

---

<div class="post-metadata">

**Author:** ![ithieund](https://avatars.discourse-cdn.com/v4/letter/i/f475e1/32.png) [@ithieund](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fithieund)%5C
**Post date:** [March 23, 2021, 4:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F4 "2021-03-23T04:34:20Z")

</div>

I don’t think you.  
Beside MLM objective like BERT-based models, PEGASUS has another special training objective called GSG and that make it powerful for abstractive text summarization.

I want to archive that powerful model for my language 🙂

---

<div class="post-metadata">

**Author:** ![jominmathew](https://avatars.discourse-cdn.com/v4/letter/j/76d3ee/32.png) [@jominmathew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjominmathew)%5C
**Post date:** [March 23, 2021, 5:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F5 "2021-03-23T05:24:19Z")

</div>

I dont think pre-training Pegasus is supported still.

---

<div class="post-metadata">

**Author:** ![Skylixia](https://avatars.discourse-cdn.com/v4/letter/s/439d5e/32.png) [@Skylixia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSkylixia)%5C
**Post date:** [April 16, 2021, 7:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F6 "2021-04-16T19:59:46Z")

</div>

I would also be interested in this 🙂

---

<div class="post-metadata">

**Author:** ![Skylixia](https://avatars.discourse-cdn.com/v4/letter/s/439d5e/32.png) [@Skylixia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSkylixia)%5C
**Post date:** [April 19, 2021, 7:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F7 "2021-04-19T19:00:13Z")

</div>

@patrickvonplaten  
In order to implement the PEGASUS pretraining objective ourselves, could we follow the same [approach you suggested](https://github.com/huggingface/transformers/issues/5096#issuecomment-645860271) for mBART ?  
This means by adapting to the objective presented in the [paper](https://arxiv.org/pdf/1912.08777.pdf) it would become:

```auto
from transformers import PegasusTokenizer, PegasusForConditionalGeneration, PegasusConfig

tok = PegasusTokenizer.from_pretrained("google/pegasus")
model = PegasusForConditionalGeneration(PegasusConfig())

input_string = ["Pegasus is <mask_2> . <mask_1> it <mask_2> the model ."
decoder_input_string = "<s> It is pure white ."
labels_string = "It is pure white . <eos>"

input_ids = tok(input_string, add_special_tokens=False, return_tensors="pt").input_ids
decoder_input_ids =tok(decoder_input_string, add_special_tokens=False, return_tensors="pt").input_ids
labels = tok(labels_string, add_special_tokens=False, return_tensors="pt").input_ids
 
loss = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids, labels=labels)[0]

```

Naturally, to automate it for pretraining one should implement the mask selection procedure of the dataset (top ROUGE sentences).  
Then, the loss we compute would be put in the method \_step.  
Is this reasonable or I’m missing something ?

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [April 25, 2021, 6:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-pegasus-model-from-scratch%2F4544%2F8 "2021-04-25T18:07:04Z")

</div>

Answered here: [Pretrain PEGASUS from scratch · Issue #8536 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/8536) 🙂