# PreTrain GPT2 from scratch in Swedish

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087 "2021-06-23T11:11:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F1 "2021-06-23T11:11:07Z")

</div>

# GPT2 for Swedish

Currently, there is no GPT2 model that was trained from scratch for Swedish on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=gpt2,sv) . For this project, the goal is to create a strong language generation model for Swedish.

## Model

A randomly initialized GPT2 model

## Datasets

One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).

## Available training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Spanish language. A nice generation demo can be created for this.

## (Optional) Challenges

It might be possible that there is not enough data for the model to perform reasonably well on text generation. In this case, one would have to look at other datasets as well, like `mc4`.

## (Optional) Links to read upon

The most important read would be the following colab:

- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)

---

<div class="post-metadata">

**Author:** ![birgermoell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/birgermoell/32/2052_2.png) [@birgermoell](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbirgermoell)%5C
**Post date:** [June 28, 2021, 8:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F3 "2021-06-28T08:42:17Z")

</div>

This sounds like a great project.

Just for reference, there is a Swedish generative model based on Flashback data.

> **[2104.05277.pdf](https://arxiv.org/pdf/2104.05277.pdf)**
>
> 297.70 KB

> **[GitHub - TobiasNorlund/flashback-gpt: Official repo to the paper "Building a...](https://github.com/TobiasNorlund/flashback-gpt)**
>
> Official repo to the paper "Building a Swedish Open-Domain Conversational Language Model" - GitHub - TobiasNorlund/flashback-gpt: Official repo to the paper "Building a Swedish Open-...

---

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsaattrupdan)%5C
**Post date:** [June 28, 2021, 9:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F4 "2021-06-28T21:35:59Z")

</div>

Hi! Considering the similarity between the Scandinavian languages, I suggest we might achieve a higher performance by utilising data from all of the languages. Just a suggestion. I made a project proposal here: [Scandinavian RoBERTa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fscandinavian-roberta%2F7533). It’s using RoBERTa and not GPT-2 however, but I’m not too fuzzed about the model architecture, to be honest.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 1:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F5 "2021-06-29T13:57:10Z")

</div>

Cool, finalizing this project 🙂

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 1:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-swedish%2F7087%2F6 "2021-06-29T13:59:52Z")

</div>

Think we can have both a Scandinavian and a Swedish GPT2 project. It would be great to find some more people working on GPT2 in Swedish as well 🙂

I’ve just put @birgermoell and @Gabriel now
