# Pretrain RoBERTa-large from scratch in Finnish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-finnish%2F7511%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 28, 2021, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-finnish%2F7511 "2021-06-28T11:06:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![RASMUS](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rasmus/32/21273_2.png) [@RASMUS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRASMUS)%5C
**Post date:** [June 28, 2021, 11:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-finnish%2F7511%2F1 "2021-06-28T11:06:48Z")

</div>

# Finnish RoBERTA-large

The project idea is somewhat identical to the one for Pretraining Roberta in Spanish but instead using the Finnish datasets

The idea is to use the [Finnish portion of mC4](https://github.com/allenai/allennlp/discussions/5265) (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512. Also we might try few other datasets (OSCAR, STT, Yle news)

## 2. Language

The model will be trained in Finnish.

## 3. Model

RoBERTa-large  
(Maybe also some other models if time is left)

## 4. Datasets

Finnish portion of mC4 of about 100gb  
Yle news dataset  
STT news

## 5. Training scripts

There are already Flax scripts to pre-train RoBERTa that we can easily use:

```auto
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

```

## 6. Challenges

Will the data be enough to be able to train a good model?

## 7. Desired project outcome

A Finnish monolingual well performing model on the usual benchmarks. We hope to beat SOTA model for this task. Current SOTA is TurkuNLP/finbert

## 8. Reads

```auto
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

```

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 1:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-large-from-scratch-in-finnish%2F7511%2F4 "2021-06-29T13:39:55Z")

</div>

Great! Finalizing this project 🙂
