# How is T5 pretrained?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 9, 2021, 8:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222 "2021-07-09T20:47:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tstandley](https://avatars.discourse-cdn.com/v4/letter/t/bcef8e/32.png) [@tstandley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftstandley)%5C
**Post date:** [July 9, 2021, 8:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222%2F1 "2021-07-09T20:47:23Z")

</div>

Hi all.

I’m creating a pretrianed T5 model with:

```auto
T5ForConditionalGeneration.from_pretrained("t5-small")

```

How is this model pretrained? It seems to me that the model weights I get here were trained at least on the GLUE dataset (and probably others).

I’d like it to only be pretrained on C4. Are those weights around somewhere? How do I get a model pretrained that way?

Thanks!

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [July 10, 2021, 10:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222%2F2 "2021-07-10T10:29:00Z")

</div>

This is quite a general question. You should find everything you need in their paper.

[[1910.10683] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (arxiv.org)](https://arxiv.org/abs/1910.10683)

---

<div class="post-metadata">

**Author:** ![tstandley](https://avatars.discourse-cdn.com/v4/letter/t/bcef8e/32.png) [@tstandley](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftstandley)%5C
**Post date:** [July 10, 2021, 6:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222%2F3 "2021-07-10T18:39:29Z")

</div>

Yes, I’ve read the paper. My question is about the Huggingface implementation.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 12, 2021, 12:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-t5-pretrained%2F8222%2F4 "2021-07-12T12:09:30Z")

</div>

This is the same model as the one release by the authors of that paper.