# T5 training from scratch

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%3E%5C
**Category:** Beginners\
**Created:** [November 4, 2020, 5:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898 "2020-11-04T17:42:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon47807654)%5C
**Post date:** [November 4, 2020, 5:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F1 "2020-11-04T17:42:33Z")

</div>

Hi all,  
I would like to train a T5 model (t5-base version) without loading the pretrained weights, if I write the following:  
from transformers import T5Config, T5Model  
config = T5Config.from\_pretrained(‘t5-base’)  
model = T5Model(config)  
It will produce the t5-base sized T5 model without loading the checkpoint weights?  
Thanks

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [November 4, 2020, 11:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F2 "2020-11-04T23:16:10Z")

</div>

> [@anon47807654](#):
>
> from transformers import T5Config, T5Model  
> config = T5Config.from\_pretrained(‘t5-base’)  
> model = T5Model(config)

Yes, that’s correct. You will definitely see the “pretrained weights” download progress bar in the other case 🙂

---

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon47807654)%5C
**Post date:** [November 5, 2020, 9:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F3 "2020-11-05T09:26:35Z")

</div>

Thanks for your reply, if I will see the download progress bar it means that the pretrained weights have been loaded?  
The fact that the T5Config was initialized by “from\_pretrained” doesn’t imply downloading something?

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [November 5, 2020, 9:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F4 "2020-11-05T09:50:23Z")

</div>

That’s only config (eg. hyperparameters) 😉

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [November 5, 2020, 7:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F5 "2020-11-05T19:01:02Z")

</div>

> [@anon47807654](#):
>
> been loaded?  
> The fact that the T5Config was initialized by “from\_pretrained” doesn’t imply downloading something?

There will be a little progress bar of \< 1mb. The weights are \> 1GB.

---

<div class="post-metadata">

**Author:** ![anon47807654](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@anon47807654](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon47807654)%5C
**Post date:** [November 5, 2020, 7:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-training-from-scratch%2F1898%2F6 "2020-11-05T19:44:05Z")

</div>

Yeah, I’ve noticed it. Thank you all!