# Pre-train ALBERT from scratch for Persian/Farsi language

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [July 7, 2021, 1:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146 "2021-07-07T13:42:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ghofrani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghofrani/32/2242_2.png) [@ghofrani](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fghofrani)%5C
**Post date:** [July 7, 2021, 1:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F1 "2021-07-07T13:42:50Z")

</div>

## 1. Pre-train Albert from Scratch for the Persian Language

Currently, there are some open-source language models for the Farsi language. We Crawl some new samples as a new dataset which I want to use for pre-training Albert for the farsi language from scratch.

## 2. Language

Persian/Farsi

## 3. Model

ALBERT xLarge

## 4. Datasets

Wikipedia dump  
Common Crawl dump  
random web scraps

## 5. Training scripts

There are already Flax scripts to pre-train Albert that we can easily use:

[transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

## 6. Challenges

Using Dataset pipeline (ETL approach) for new txt crawled file

---

<div class="post-metadata">

**Author:** ![Mojtaba](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mojtaba/32/2222_2.png) [@Mojtaba](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMojtaba)%5C
**Post date:** [July 7, 2021, 2:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F2 "2021-07-07T14:43:20Z")

</div>

Sounds amazing, I’m also interested in participating.

---

<div class="post-metadata">

**Author:** ![ghofrani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghofrani/32/2242_2.png) [@ghofrani](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fghofrani)%5C
**Post date:** [July 7, 2021, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F3 "2021-07-07T14:44:26Z")

</div>

thanks sure, We are waiting for accept

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 7, 2021, 6:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F4 "2021-07-07T18:06:05Z")

</div>

Alright finalizing it!

---

<div class="post-metadata">

**Author:** ![alphareality](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@alphareality](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falphareality)%5C
**Post date:** [July 8, 2021, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F5 "2021-07-08T15:00:41Z")

</div>

I’m interested to join

---

<div class="post-metadata">

**Author:** ![m3hrdadfi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/m3hrdadfi/32/18938_2.png) [@m3hrdadfi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fm3hrdadfi)%5C
**Post date:** [July 10, 2021, 4:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-albert-from-scratch-for-persian-farsi-language%2F8146%2F6 "2021-07-10T16:21:40Z")

</div>

I’d be happy to help you with this project!

@patrickvonplaten, do I need to request TPU access, or can I collaborate using the credential by one of the team members since I have access to the hf-flax group?
