# BERTIN: PreTrain RoBERTa-large from scratch in Spanish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125 "2021-06-23T15:34:27Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![versae](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/versae/32/2153_2.png) [@versae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fversae)%5C
**Post date:** [June 23, 2021, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F1 "2021-06-23T15:34:27Z")

</div>

# BERTIN

The idea is to use the [Spanish portion of mC4](https://github.com/allenai/allennlp/discussions/5265) (which roughly amounts for 1TB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggested [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fopen-to-the-community-community-week-using-jax-flax-for-nlp-cv%2F7104%2F14).

## 2. Language

The model will be trained in Spanish (regardless of variety).

## 3. Model

RoBERTa-large

## 4. Datasets

Spanish portion of mC4 of about 1TB of uncompressed data.

## 5. Training scripts

There are already Flax scripts to pre-train RoBERTa that we can easily use:

```auto
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

```

## 6. Challenges

It is too much data. We need a way to reduce the amount of data to finish on time. Options:

1. Random sampling.
2. Perplexity sampling using percentiles and a Spanish language model. One option here is to use a 5-gram Kneser-Ney model as implemented in the KenLM library (Heafield, 2011) and released by Facebook.

## 7. Desired project outcome

A Spanish monolingual well performing model on the usual benchmarks.

## 8. Reads

```auto
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

```

---

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [June 23, 2021, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F2 "2021-06-23T15:38:58Z")

</div>

So interesting! I am in!

---

<div class="post-metadata">

**Author:** ![mariagrandury](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariagrandury/32/27732_2.png) [@mariagrandury](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariagrandury)%5C
**Post date:** [June 23, 2021, 3:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F3 "2021-06-23T15:57:36Z")

</div>

Me too! Nice idea 🤗

---

<div class="post-metadata">

**Author:** ![josecalvo](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@josecalvo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjosecalvo)%5C
**Post date:** [June 23, 2021, 5:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F4 "2021-06-23T17:36:17Z")

</div>

Yes! Count me in!

---

<div class="post-metadata">

**Author:** ![vitojph](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vitojph/32/1915_2.png) [@vitojph](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvitojph)%5C
**Post date:** [June 24, 2021, 6:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F6 "2021-06-24T06:47:50Z")

</div>

Hey! Awesome project! I’m in 😛

---

<div class="post-metadata">

**Author:** ![Paulo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/paulo/32/1919_2.png) [@Paulo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulo)%5C
**Post date:** [June 24, 2021, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F7 "2021-06-24T07:50:27Z")

</div>

I think you have already a full working team, but just in case you need more people, I’d be willing to help. I don’t know if I could be of much use, but the opportunity to contribute to a model called BERTIN is too good to let it go 😁

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 24, 2021, 8:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F8 "2021-06-24T08:31:38Z")

</div>

Awesome to see so much interest here! We can also make multiple teams for the same project - so feel free to keep posting if you’re interested 🙂

---

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [June 24, 2021, 8:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F9 "2021-06-24T08:32:59Z")

</div>

For this project we can have at least 2 teams!

---

<div class="post-metadata">

**Author:** ![Pablogps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pablogps/32/32528_2.png) [@Pablogps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPablogps)%5C
**Post date:** [June 24, 2021, 9:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F10 "2021-06-24T09:37:04Z")

</div>

This project sounds good! I’d like to help with either this one or the [Spanish GPT-2](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-spanish%2F7086)%2C wherever makes more sense to add one person.

---

<div class="post-metadata">

**Author:** ![hcordobes](https://avatars.discourse-cdn.com/v4/letter/h/3ab097/32.png) [@hcordobes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhcordobes)%5C
**Post date:** [June 24, 2021, 10:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F11 "2021-06-24T10:56:29Z")

</div>

I also think this is a very good effort. If I can be of any help please count me in.

---

<div class="post-metadata">

**Author:** ![arteagac](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arteagac/32/32065_2.png) [@arteagac](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farteagac)%5C
**Post date:** [June 24, 2021, 4:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F12 "2021-06-24T16:50:11Z")

</div>

Fantastic. I’d love to participate in one of the teams, if possible. Thanks!

---

<div class="post-metadata">

**Author:** ![jorgealro](https://avatars.discourse-cdn.com/v4/letter/j/e68b1a/32.png) [@jorgealro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjorgealro)%5C
**Post date:** [June 25, 2021, 8:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F13 "2021-06-25T08:57:20Z")

</div>

What a good idea. Please, count me in if possible!!

---

<div class="post-metadata">

**Author:** ![srisweet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srisweet/32/1829_2.png) [@srisweet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsrisweet)%5C
**Post date:** [June 27, 2021, 5:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F15 "2021-06-27T05:45:07Z")

</div>

Hello @versae & Team,  
I am interested to be a part of such an amazing project & team. I will try my best to contribute to the BERTIN model. It would be nice if we could discuss some more learning resources that would be useful for this project.I can work in any time zone that is comfortable for everyone in the team.

---

<div class="post-metadata">

**Author:** ![versae](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/versae/32/2153_2.png) [@versae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fversae)%5C
**Post date:** [June 27, 2021, 6:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F16 "2021-06-27T18:56:23Z")

</div>

I’m overwhelmed by the good reception of the idea. Thanks y’all! If we are awarded the TPUs, we must decide on a communication channel a bit more effective than the forum, maybe. A Slack channel? Discord? E-mail?

---

<div class="post-metadata">

**Author:** ![mariagrandury](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariagrandury/32/27732_2.png) [@mariagrandury](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariagrandury)%5C
**Post date:** [June 27, 2021, 9:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F17 "2021-06-27T21:40:30Z")

</div>

It was a really nice idea @versae! If you want we could use the Slack group of [NLP en ES](https://twitter.com/nlp_en_es) 🤗 and create a channel for us. Some of the people in this thread are already in the community, if the others wanna join you can use this [invitation](https://join.slack.com/t/nlp-en-es/shared_invite/zt-s8armi39-7EbkJnPGMVQgHaonG_Qg2Q).

---

<div class="post-metadata">

**Author:** ![birgermoell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/birgermoell/32/2052_2.png) [@birgermoell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbirgermoell)%5C
**Post date:** [June 28, 2021, 9:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F18 "2021-06-28T09:27:28Z")

</div>

This seems like a super interesting project. I would love to be a part of this project.

---

<div class="post-metadata">

**Author:** ![edugp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/edugp/32/2025_2.png) [@edugp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fedugp)%5C
**Post date:** [June 28, 2021, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F19 "2021-06-28T15:51:09Z")

</div>

This sounds awesome! I would be interested in joining as well, if there’s still room for more people! 🙂

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 28, 2021, 4:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F20 "2021-06-28T16:10:18Z")

</div>

Super exciting about this project with so much interest! Given that we have so many people, I think it makes sense to officially define 2 groups of 7 people (more people can join later :-))

I’ve divided people that are interested into two groups and added all the names [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing).

Let me know if you don’t want to be in the team and I take you out 🙂

I think the two teams should talk to each other/ help each other as much as possible, but as stated on Slack before having a single team of 14 is not efficient IMO. Therefore, I propose that both teams take their own approach to doing the dataset preprocessing, model definition etc…

One important aspect that should also be discussed a bit is how the pretrained RoBERTa model should be evaluated ? Maybe on [XTREME](https://arxiv.org/abs/2003.11080) after pretraining?

---

<div class="post-metadata">

**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrm8488)%5C
**Post date:** [June 28, 2021, 4:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F21 "2021-06-28T16:25:17Z")

</div>

XTREME and PAWS-X could be a good starting point

---

<div class="post-metadata">

**Author:** ![versae](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/versae/32/2153_2.png) [@versae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fversae)%5C
**Post date:** [June 28, 2021, 4:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125%2F22 "2021-06-28T16:27:51Z")

</div>

I think XTREME is perfect. Not sure if XTREME superseeds PAWS-X or they are just different.

So if I understand this correctly, this means we could pre-train 2 versions of BERTIN?

It that’s the case, that’s something the 2 teams need to talk about and agree on. My cents would be to test the 2 different approaches to shrink the corpus size: random sampling, perplexity sampling.

[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertin-pretrain-roberta-large-from-scratch-in-spanish%2F7125.md%3Fpage%3D2)
