# PreTrain GPT2 from scratch in Russian

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-russian%2F7089%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 11:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-russian%2F7089 "2021-06-23T11:16:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-russian%2F7089%2F1 "2021-06-23T11:16:17Z")

</div>

# GPT2 for Russian

Currently, there is no GPT2 model that was trained from scratch for Turkish on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=gpt2,ru) . For this project, the goal is to create a strong language generation model for Russian.

## Model

A randomly initialized GPT2 model

## Datasets

One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).

## Available training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Russian language. A nice generation demo can be created for this.

## (Optional) Challenges

The dataset on OSCAR is very large: \> 300GB. One might want to explore dataset streaming techniques here. Dataset streaming will be merged to `datasets` in a couple of days. See PR [here](https://github.com/huggingface/datasets/pull/2375) and the docs [here](https://40476-250213286-gh.circle-artifacts.com/0/docs/_build/html/dataset_streaming.html).

## (Optional) Links to read upon

The most important read would be the following colab:

- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)

---

<div class="post-metadata">

**Author:** ![kperv](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@kperv](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkperv)%5C
**Post date:** [July 1, 2021, 9:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-russian%2F7089%2F2 "2021-07-01T09:48:23Z")

</div>

I’m interested.
