# PreTrain ELECTRA from scratch in Portuguese

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-electra-from-scratch-in-portuguese%2F7273%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 24, 2021, 4:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-electra-from-scratch-in-portuguese%2F7273 "2021-06-24T16:02:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![diiogo](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@diiogo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiiogo)%5C
**Post date:** [June 24, 2021, 4:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-electra-from-scratch-in-portuguese%2F7273%2F1 "2021-06-24T16:02:29Z")

</div>

## ELECTRA for Portuguese

Currently, there is no ELECTRA or ELECTRA Large model that was trained from scratch for Portuguese on the hub: [Hugging Face – The AI community building the future.](https://huggingface.co/models?filter=electra,pt). For this project, the goal is to create a ELECTRA model for just the Portuguese language.

## Model

A randomly initialized ELECTRA model

## Available training scripts

A masked language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py). It can be used pretty much without any required code changes.

## (Optional) Desired project outcome

The desired project output is a strong ELECTRA model in Portuguese.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 2, 2021, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-electra-from-scratch-in-portuguese%2F7273%2F2 "2021-07-02T15:38:11Z")

</div>

ELECTRA actually uses a special pretraining script, which would need to be written during the sprint - if you are very motivated I’m willing to confirm this “single-person” project and give you a TPU VM. Please leave a comment on the official googe sheet in this case: [Confirmed teams for Flax/JAX community week - Google Sheets](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing)

---

<div class="post-metadata">

**Author:** ![Emanuel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/emanuel/32/20406_2.png) [@Emanuel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEmanuel)%5C
**Post date:** [November 8, 2021, 1:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-electra-from-scratch-in-portuguese%2F7273%2F3 "2021-11-08T13:22:56Z")

</div>

Hi Patrick, I am also interested in training Electra for Portuguese. I don’t know if this is currently under development or already has been done, if not, I would like to implement this pretraining script and train the model for Portuguese.
