# PreTrain GPT2 from scratch in Bengali

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122 "2021-06-23T15:30:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [June 23, 2021, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F1 "2021-06-23T15:30:43Z")

</div>

# GPT2 for Bengali

Currently, there is no GPT2 model that was trained from scratch for Bengali on the hub. For this project, the goal is to create a strong language generation model for Bengali using GPT2 Model.

## 2. Language

Bengali.

## 3. Model

A randomly initialized GPT2 model.

## 4. Datasets

One can make use of OSCAR the dataset is also available through the datasets library here: [oscar · Datasets at Hugging Face](https://huggingface.co/datasets/oscar). The total Bengali resource in OSCAR is 11 GB.

Another source can be the mC4 dataset which is available in _[AllenAI](https://github.com/allenai/allennlp/discussions/5265)_. The resource size is 29GB.

## 5. Training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be tweaked for training GPT2.

## 6. Challenges

- Fix a good tokenizer that covers Bengali vocabulary properly and make sure that the LM doesn’t become character-level LM.

## 7. Desired project outcome

The desired project output is a GPT2 model that is able to generate Bengali language.

## 8. Reads

The most important read would be the following colab:

- [Google Colaboratory](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)

Other reads that might be interesting include:

- [GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt University](https://www.vanderbilt.edu/digitalhumanities/gpt-2-no-habla-espanol-artificial-intelligence-anglocentrism-and-the-non-human-side-of-dh/)
- [https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171](https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171)

---

<div class="post-metadata">

**Author:** ![tasnim](https://avatars.discourse-cdn.com/v4/letter/t/7ba0ec/32.png) [@tasnim](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftasnim)%5C
**Post date:** [June 23, 2021, 3:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F2 "2021-06-23T15:56:39Z")

</div>

I am also a Bengali speaker. I am in!

---

<div class="post-metadata">

**Author:** ![khalidsaifullaah](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/khalidsaifullaah/32/18191_2.png) [@khalidsaifullaah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkhalidsaifullaah)%5C
**Post date:** [June 23, 2021, 4:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F3 "2021-06-23T16:22:12Z")

</div>

A Bengali text generation model would be totally great. This would be revolutionary for Bengali NLP research community! ❤  
I’m sooo in!!! 💥 🔥

(I really hope this topic gets selected 🤞)

---

<div class="post-metadata">

**Author:** ![ibraheemmoosa](https://avatars.discourse-cdn.com/v4/letter/i/a587f6/32.png) [@ibraheemmoosa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fibraheemmoosa)%5C
**Post date:** [June 25, 2021, 3:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F4 "2021-06-25T15:10:20Z")

</div>

Text generation is the dream.  
BTW does it have to be GPT-2. Are there more efficient (compute or sample) models now?

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 25, 2021, 5:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F5 "2021-06-25T17:20:03Z")

</div>

This looks like a great topic! Let’s wait if more people are interested until Monday and make this an official team than on Monday 🙂

---

<div class="post-metadata">

**Author:** ![Tahsin-Mayeesha](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tahsin-mayeesha/32/2005_2.png) [@Tahsin-Mayeesha](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTahsin-Mayeesha)%5C
**Post date:** [June 27, 2021, 12:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F6 "2021-06-27T12:47:43Z")

</div>

I’d like to join too. @sbmaruf@khalidsaifullaah

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 4:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F7 "2021-06-28T16:56:50Z")

</div>

Let’s officially define this project 🙂

Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something.

---

<div class="post-metadata">

**Author:** ![tghosh93](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tghosh93/32/2130_2.png) [@tghosh93](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftghosh93)%5C
**Post date:** [July 2, 2021, 3:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F8 "2021-07-02T15:07:52Z")

</div>

I would like to join this project @valhalla @patrickvonplaten

---

<div class="post-metadata">

**Author:** ![DeadBeast](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deadbeast/32/29623_2.png) [@DeadBeast](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDeadBeast)%5C
**Post date:** [August 19, 2021, 2:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-bengali%2F7122%2F9 "2021-08-19T02:26:18Z")

</div>

would like to join this project @valhalla
