# PreTrain GPT2 from scratch in Spanish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086 "2021-06-23T11:07:30Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F1 "2021-06-23T11:07:31Z")

</div>

# GPT2 for Spanish

Currently, there is no GPT2 model that was trained from scratch for Spanish on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=gpt2,es) . For this project, the goal is to create a strong language generation model for Spanish.

## Model

A randomly initialized GPT2 model

## Datasets

One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).

## Available training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Spanish language.

## (Optional) Challenges

The dataset on OSCAR is quite large → around 150 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged to `datasets` in a couple of days. See PR [here](https://github.com/huggingface/datasets/pull/2375) and the docs [here](https://40476-250213286-gh.circle-artifacts.com/0/docs/_build/html/dataset_streaming.html).

## (Optional) Links to read upon

The most important read would be the following colab:

- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)

Other reads that might be interesting include:

- [Page Not Found](https://www.vanderbilt.edu/digitalhumanities/gpt-2-no-habla-espanol-artificial-intelligence-anglocentrism-and-the-non-human-side-of-dh/)
- [Train GPT-2 in your own language | Towards Data Science](https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171)

---

<div class="post-metadata">

**Author:** ![mariagrandury](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariagrandury/32/27732_2.png) [@mariagrandury](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariagrandury)%5C
**Post date:** [June 23, 2021, 3:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F2 "2021-06-23T15:41:11Z")

</div>

I’m definitely in! Check this @mrm8488 it’s exactly what we talked about

---

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [June 23, 2021, 3:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F3 "2021-06-23T15:43:57Z")

</div>

Yes, it is! I was thinking about using Spanish portion of mC4 instead of OSCAR. What do you think @patrickvonplaten and @mariagrandury ??

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 7:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F4 "2021-06-23T19:57:15Z")

</div>

Awesome, great that you guys are already so interested in this 🙂  
The nice thing about OSCAR is that it is supported in the `datasets` library and we now also support datasets streaming: [Load a Dataset in Streaming mode — datasets 1.8.0 documentation](https://huggingface.co/docs/datasets/master/dataset_streaming.html) which means that you surely won’t run into any disk space memory problems. We sadly don’t support this out of the box for mC4 (it’s not even in `datasets` yet), but I’d definitely recommend to try it out! Maybe you can even first train a model on OSCAR and then a second one on mC4

---

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [June 23, 2021, 8:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F5 "2021-06-23T20:33:10Z")

</div>

Great, @patrickvonplaten. We will. So we have the script, the dataset and the knowledge. Let’s start the party!

---

<div class="post-metadata">

**Author:** ![Pablogps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pablogps/32/32528_2.png) [@Pablogps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPablogps)%5C
**Post date:** [June 24, 2021, 9:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F6 "2021-06-24T09:38:02Z")

</div>

This sounds good! As I said in the project for [BERTIN](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F9)%2C I’d like to help in either one.

---

<div class="post-metadata">

**Author:** ![srisweet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srisweet/32/1829_2.png) [@srisweet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsrisweet)%5C
**Post date:** [June 27, 2021, 5:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F8 "2021-06-27T05:49:39Z")

</div>

Hello Team,  
I am interested to be a part of such an amazing project & team. I will try my best to contribute to the Spanish GPT2 model. It would be nice if we could discuss some more learning resources that would be useful for this project.I can work in any time zone that is comfortable for everyone in the team. It will be interesting to work on both OSCAR & mC4 datasets…

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F9 "2021-06-28T16:54:36Z")

</div>

Awesome! let’s officially define this project 🙂

Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something.

---

<div class="post-metadata">

**Author:** ![jdposa](https://avatars.discourse-cdn.com/v4/letter/j/f05b48/32.png) [@jdposa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjdposa)%5C
**Post date:** [June 28, 2021, 11:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F10 "2021-06-28T23:21:25Z")

</div>

hi @valhalla ,

I am definitely interested in joining. Could you add me to the spreadhset?

Thanks,

---

<div class="post-metadata">

**Author:** ![shpotes](https://avatars.discourse-cdn.com/v4/letter/s/87869e/32.png) [@shpotes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fshpotes)%5C
**Post date:** [June 28, 2021, 11:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F11 "2021-06-28T23:24:30Z")

</div>

I would love to contribute to this project 😃

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 30, 2021, 8:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F13 "2021-06-30T08:53:42Z")

</div>

@jdposa , @shpotes , you are part of the team 🙂

---

<div class="post-metadata">

**Author:** ![jorgealro](https://avatars.discourse-cdn.com/v4/letter/j/e68b1a/32.png) [@jorgealro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjorgealro)%5C
**Post date:** [July 1, 2021, 8:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F14 "2021-07-01T08:15:32Z")

</div>

Hey, count with me if possible! I’d like to contribute

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 9:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086%2F15 "2021-07-01T09:43:45Z")

</div>

you’re in 🙂
