# Pretrained GPT2 for Tamil

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 30, 2021, 4:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754 "2021-06-30T16:27:50Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frays2pix)%5C
**Post date:** [June 30, 2021, 4:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F1 "2021-06-30T16:27:50Z")

</div>

# GPT2 for Tamil

We aim to create a GPT2 model for tamil language.

## Model

A randomly initialized GPT2 model

## Datasets

OSCAR dataset ([https://oscar-corpus.com/](https://oscar-corpus.com/)) has about 9 GB of training data and Indic-NLP ([IndicCorp | AI4Bharat IndicNLP](https://indicnlp.ai4bharat.org/corpora/)) has 500 million token dataset.

## Available training scripts

A causal language modeling script for Flax is available [here]It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a GPT2 model that is able to generate Tamil language.

---

<div class="post-metadata">

**Author:** ![abinayam](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@abinayam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabinayam)%5C
**Post date:** [July 1, 2021, 3:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F2 "2021-07-01T03:17:51Z")

</div>

@rays2pix : Hi Deepak. I was hoping that someone would propose a project for Tamil. I would like to collaborate on this!

---

<div class="post-metadata">

**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frays2pix)%5C
**Post date:** [July 1, 2021, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F3 "2021-07-01T03:47:59Z")

</div>

That’s nice @abinayam . We can do this if this gets picked . What do you think @patrickvonplaten

---

<div class="post-metadata">

**Author:** ![100worte](https://avatars.discourse-cdn.com/v4/letter/1/7bcc69/32.png) [@100worte](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2F100worte)%5C
**Post date:** [July 1, 2021, 7:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F4 "2021-07-01T07:41:53Z")

</div>

I cannot help much, but I would be excited to see this happen!

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 1, 2021, 9:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F5 "2021-07-01T09:44:40Z")

</div>

Awesome! Let’s officially define it 🙂

---

<div class="post-metadata">

**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frays2pix)%5C
**Post date:** [July 1, 2021, 12:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F6 "2021-07-01T12:47:15Z")

</div>

Great. @abinayam let’s discuss on discord #gpt-tamil @100worte .

---

<div class="post-metadata">

**Author:** ![rays2pix](https://avatars.discourse-cdn.com/v4/letter/r/94ad74/32.png) [@rays2pix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frays2pix)%5C
**Post date:** [July 1, 2021, 12:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F7 "2021-07-01T12:56:24Z")

</div>

Link for discord [Flax-HuggingFace-Community-Week](https://discord.gg/K2fnCGfv)

---

<div class="post-metadata">

**Author:** ![ghosh-r](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghosh-r/32/2183_2.png) [@ghosh-r](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fghosh-r)%5C
**Post date:** [July 1, 2021, 2:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F8 "2021-07-01T14:18:04Z")

</div>

I cannot contribute to it. But wish you all the best! 😇 💯 🥳

---

<div class="post-metadata">

**Author:** ![adithya1111](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@adithya1111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadithya1111)%5C
**Post date:** [July 1, 2021, 4:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F9 "2021-07-01T16:39:54Z")

</div>

Thanks @rays2pix for introducing me to this post . I would like to collaborate if there is room

---

<div class="post-metadata">

**Author:** ![abinayam](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@abinayam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabinayam)%5C
**Post date:** [July 1, 2021, 4:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F10 "2021-07-01T16:41:38Z")

</div>

@adithya1111: You’re welcome to join the group. Please join the discord channel.

---

<div class="post-metadata">

**Author:** ![cishwarya](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@cishwarya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcishwarya)%5C
**Post date:** [July 11, 2021, 7:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F11 "2021-07-11T19:02:38Z")

</div>

Hi @abinayam though I may not be able to contribute, I would like to join the discussion in the discord channel, could you please share the link?

---

<div class="post-metadata">

**Author:** ![abinayam](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@abinayam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabinayam)%5C
**Post date:** [July 12, 2021, 4:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F12 "2021-07-12T04:52:46Z")

</div>

Hi @cishwarya: Here is the channel name in discord #gpt-tamil.

---

<div class="post-metadata">

**Author:** ![cishwarya](https://avatars.discourse-cdn.com/v4/letter/c/ed655f/32.png) [@cishwarya](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcishwarya)%5C
**Post date:** [July 12, 2021, 6:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F13 "2021-07-12T06:19:45Z")

</div>

Hi @abinayam thanks for your reply. I do not have the invite for the discord group. Could you please send me an invite…thank you so much…

---

<div class="post-metadata">

**Author:** ![abinayam](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@abinayam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabinayam)%5C
**Post date:** [July 12, 2021, 7:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrained-gpt2-for-tamil%2F7754%2F14 "2021-07-12T07:31:43Z")

</div>

Here is the invite for the discord group: [Flax-HuggingFace-Community-Week](https://discord.gg/qYRwzqqE)
