# Pretrain GPT-2 from scratch in Thai

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt-2-from-scratch-in-thai%2F8463%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [July 18, 2021, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt-2-from-scratch-in-thai%2F8463 "2021-07-18T03:52:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sakares](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sakares/32/2601_2.png) [@sakares](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsakares)%5C
**Post date:** [July 18, 2021, 3:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-gpt-2-from-scratch-in-thai%2F8463%2F1 "2021-07-18T03:52:17Z")

</div>

# GPT2 for Thai

The goal is to create a strong language generation model for Thai 🇹🇭  
Since initial code and data are pretty much written by @patrickvonplaten and other huggingface members, it should not be so hard to get the first sense.

## Model

Randomly initialized GPT2 model

## Datasets

We can use OSCAR which is available through `datasets`

## Datasets

A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be used pretty much without any required code changes.  
If there is time left, I’d love to try some private crawling and integrate it datasets.

## Expected Outcome

Understandable Thai text generation model

## Challenges

Lack of data → The OSCAR dataset might be too small (it has \< 20GB of data for Thai). It would be better to collect and accumulate to bigger dataset.

## Links

- [Colab Notebook to train CLM](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)
- [OSCAR](https://oscar-corpus.com/)
- [flax code](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py)

## Team Members

- Sakares Saengkaew ([@sakares](https://hf.co/sakares))
- Wilson Wongso ([@w11wo](https://hf.co/w11wo))
