# PreTrain GPT2 from scratch in Indonesia

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 24, 2021, 6:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288 "2021-06-24T18:52:19Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![munggok](https://avatars.discourse-cdn.com/v4/letter/m/919ad9/32.png) [@munggok](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmunggok)%5C
**Post date:** [June 24, 2021, 6:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F1 "2021-06-24T18:52:19Z")

</div>

# GPT2 for Indonesia

Currently, there is only few GPT2 model that was trained from scratch for Indonesian (small size only) on the hub: [Hugging Face – The AI community building the future.](https://huggingface.co/models?filter=gpt2,id) . For this project, the goal is to create a strong language generation model for Indonesia.

## Model

A randomly initialized GPT2 model

## Datasets

One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [oscar · Datasets at Hugging Face](https://huggingface.co/datasets/oscar).

A portion mc4 [MC4] ([The C4 Multilingual Dataset · allenai/allennlp · Discussion #5265 · GitHub](https://github.com/allenai/allennlp/discussions/5265)) and  
CC100 [[CC-100: Monolingual Datasets from Web Crawl Data](http://data.statmt.org/cc-100/)] can also be used for this task

## Available training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Indonesia language.

## (Optional) Challenges

The dataset on MC4 and CC100 is quite large → around 350 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged to `datasets` in a couple of days. See PR [here](https://github.com/huggingface/datasets/pull/2375) and the docs [here](https://40476-250213286-gh.circle-artifacts.com/0/docs/_build/html/dataset_streaming.html).

Since mc4 not available in datasets yet, adjustment for streaming dataset could be required

## (Optional) Links to read upon

The most important read would be the following colab:

- [Google Colaboratory](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)

- [https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171](https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171)

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcahya)%5C
**Post date:** [June 24, 2021, 8:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F2 "2021-06-24T20:11:35Z")

</div>

Thanks for creating the proposal, I am in.

---

<div class="post-metadata">

**Author:** ![Galuh](https://avatars.discourse-cdn.com/v4/letter/g/b2d939/32.png) [@Galuh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGaluh)%5C
**Post date:** [June 24, 2021, 11:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F3 "2021-06-24T23:16:27Z")

</div>

Thanks for the proposal, count me in!

---

<div class="post-metadata">

**Author:** ![Wikidepia](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wikidepia/32/1099_2.png) [@Wikidepia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FWikidepia)%5C
**Post date:** [June 25, 2021, 7:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F5 "2021-06-25T07:27:42Z")

</div>

Thanks for the proposal, I’m in!

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcahya)%5C
**Post date:** [June 25, 2021, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F6 "2021-06-25T08:01:39Z")

</div>

Hopefully we can pretrain gpt2-large using the datasets mentioned above within only one week 🙂  
Btw, to make our discussion easier, we will use again telegram.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 25, 2021, 5:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F7 "2021-06-25T17:30:14Z")

</div>

Cool idea and great description - we can probably already fix this team on Monday 🙂

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcahya)%5C
**Post date:** [June 28, 2021, 11:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F8 "2021-06-28T23:18:40Z")

</div>

We received additional 1.2GB dataset of “indonesian-journals”, which is a collection of journals/papers from several indonesian universities.

---

<div class="post-metadata">

**Author:** ![Suzana](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@Suzana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSuzana)%5C
**Post date:** [June 29, 2021, 7:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F9 "2021-06-29T07:26:41Z")

</div>

Awesome, let’s officially define this project! 🙌  
Added everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). 🙂

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcahya)%5C
**Post date:** [June 29, 2021, 9:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F10 "2021-06-29T09:18:41Z")

</div>

Thanks @Suzana. May I ask you to add also @munggok and @ayameRushia to the official sheet please (I can’t update it my self)? Thanks.

---

<div class="post-metadata">

**Author:** ![ayameRushia](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ayamerushia/32/16473_2.png) [@ayameRushia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FayameRushia)%5C
**Post date:** [June 29, 2021, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F11 "2021-06-29T09:30:52Z")

</div>

May I join too?

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcahya)%5C
**Post date:** [June 29, 2021, 9:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F12 "2021-06-29T09:35:37Z")

</div>

Of course 🙂

---

<div class="post-metadata">

**Author:** ![Suzana](https://avatars.discourse-cdn.com/v4/letter/s/c0e974/32.png) [@Suzana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSuzana)%5C
**Post date:** [June 29, 2021, 11:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F13 "2021-06-29T11:15:41Z")

</div>

Done! @cahya

---

<div class="post-metadata">

**Author:** ![muhammadfhadli](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muhammadfhadli/32/39543_2.png) [@muhammadfhadli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuhammadfhadli)%5C
**Post date:** [June 29, 2021, 5:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F14 "2021-06-29T17:58:43Z")

</div>

Hi, thank you for the proposal, Count me in.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 30, 2021, 12:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-indonesia%2F7288%2F15 "2021-06-30T12:31:09Z")

</div>

Done! @muhammadfhadli
