# PreTrain GPT2 from scratch in Swedish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087 "2021-06-23T11:11:07Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaattrupdan)%5C
**Post date:** [June 28, 2021, 9:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F4 "2021-06-28T21:35:59Z")

</div>

Hi! Considering the similarity between the Scandinavian languages, I suggest we might achieve a higher performance by utilising data from all of the languages. Just a suggestion. I made a project proposal here: [Scandinavian RoBERTa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscandinavian-roberta%2F7533). It’s using RoBERTa and not GPT-2 however, but I’m not too fuzzed about the model architecture, to be honest.

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087)._
