# PreTrain GPT2 from scratch in Punjabi

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-punjabi%2F7376%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 25, 2021, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-punjabi%2F7376 "2021-06-25T16:06:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hgarg](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hgarg/32/1592_2.png) [@hgarg](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhgarg)%5C
**Post date:** [June 25, 2021, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-punjabi%2F7376%2F1 "2021-06-25T16:06:48Z")

</div>

# GPT2 for Punjabi

Pretrain GPT2 on Punjabi language to create a strong language generation model for Punjbai

## Model

A randomly initialized GPT2 model

## Datasets

One can make use of Kaggle Wikipedia Punjabi Articles dataset - [Punjabi Wikipedia Articles | Kaggle](https://www.kaggle.com/disisbig/punjabi-wikipedia-articles)

## Available training scripts

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py).

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Punjabi language.

---

<div class="post-metadata">

**Author:** ![bhavnicksm](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bhavnicksm/32/3547_2.png) [@bhavnicksm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbhavnicksm)%5C
**Post date:** [June 25, 2021, 6:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-punjabi%2F7376%2F2 "2021-06-25T18:02:59Z")

</div>

Hey @hgarg! 🤗

Great to see someone working on Punjabi. There is definitly a lack of models for Punjabi in the HuggingFace Hub, so this can definitly supplement that.

Great idea and all the best!

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt2-from-scratch-in-punjabi%2F7376%2F3 "2021-06-29T14:39:29Z")

</div>

Cool idea @bhavnicksm - would you like to join? 🙂
