# Pretrain RoBERTa-large from scratch in Swedish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-swedish%2F7497%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 28, 2021, 9:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-swedish%2F7497 "2021-06-28T09:34:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![birgermoell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/birgermoell/32/2052_2.png) [@birgermoell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbirgermoell)%5C
**Post date:** [June 28, 2021, 9:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-swedish%2F7497%2F1 "2021-06-28T09:34:15Z")

</div>

# Roberta Swedish

The project idea is identical to the one for Pretraining Roberta in Spanish but instead using the Swedish dataset.

The idea is to use the [Swedish portion of mC4](https://github.com/allenai/allennlp/discussions/5265) (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. It might be a good idea to start in 128, as suggested [here](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fopen-to-the-community-community-week-using-jax-flax-for-nlp-cv%2F7104%2F14).

## 2. Language

The model will be trained in Swedish.

## 3. Model

RoBERTa-large

## 4. Datasets

Swedish portion of mC4 of about 100gb of uncompressed data.

## 5. Training scripts

There are already Flax scripts to pre-train RoBERTa that we can easily use:

```auto
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

```

## 6. Challenges

Will the data be enough to be able to train a good model?

## 7. Desired project outcome

A Swedish monolingual well performing model on the usual benchmarks.

## 8. Reads

```auto
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

```

---

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaattrupdan)%5C
**Post date:** [June 28, 2021, 5:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-swedish%2F7497%2F2 "2021-06-28T17:38:01Z")

</div>

Just a suggestion: How about we pretrain a common Scandinavian model instead, as our languages are so similar? I’ve made a project suggestion here: [Scandinavian RoBERTa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscandinavian-roberta%2F7533).

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 5, 2021, 2:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-swedish%2F7497%2F3 "2021-07-05T14:04:52Z")

</div>

Officially defining this one as well 🙂
