# PreTrain T5 for Italian 🇮🇹

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 26, 2021, 3:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425 "2021-06-26T15:10:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![gsarti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gsarti/32/15494_2.png) [@gsarti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgsarti)%5C
**Post date:** [June 26, 2021, 3:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425%2F1 "2021-06-26T15:10:15Z")

</div>

# Train T5 from scratch in Italian on mC4

Currently the [Italian Hub](https://huggingface.co/models?filter=it) has only autoencoding (e.g. [`bert-base-italian`](https://huggingface.co/dbmdz/bert-base-italian-cased), [`BERTino`](https://huggingface.co/indigo-ai/BERTino)) and autoregressive (e.g. [`GePpeTto`](https://huggingface.co/LorenzoDeMattei/GePpeTto)) language models. For this project, the goal is to create a strong `Text2TextGeneration` model for seq2seq tasks in Italian.

## Model

We’ll be using a random [T5](https://huggingface.co/transformers/model_doc/t5.html) model with a configuration similar to the one of `t5-base`, since the Flax implementation was recently made available on the Hub.

## Datasets

Recently the [multilingual C4 dataset](https://github.com/allenai/allennlp/discussions/5265) was made available, and it surpasses by a large margin the size of the OSCAR corpus (590GB cleaned vs. 69GB). Given the new [streaming capabilities for large datasets](https://github.com/huggingface/datasets/pull/2375) this should be easily feasible. Another option is to use multiple large datasets (mC4, OSCAR, Italian Wikipedia) with `ìnterleave_datasets` to [mix them](https://huggingface.co/docs/datasets/dataset_streaming.html).

## Training scripts

We can leverage the T5 pretraining script that is [currently being prepared](https://github.com/huggingface/transformers/pull/12355) to train the model, and [`run_summarization_flax.py`](https://github.com/huggingface/transformers/blob/master/examples/flax/summarization/run_summarization_flax.py) to test downstream performances, e.g. on [SQUAD-it](https://huggingface.co/datasets/squad_it) converted in text2text format.

## Challenges

The project involves a pre-training of a large LM, so it will inevitably take some time to reach competitive performances. Most scripts and implementations are very new, and as such it is possible that we’ll incur in some issues to be fixed.

## Desired project outcome

The main project outcome, besides producing a trained T5 model for Italian, is to have fun and get our hands dirty with some :jax:! We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.

## Reads

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683)

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425%2F2 "2021-06-29T14:36:51Z")

</div>

Think this is a cool project - let’s see if we get some more participants 🙂

---

<div class="post-metadata">

**Author:** ![gsarti](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gsarti/32/15494_2.png) [@gsarti](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgsarti)%5C
**Post date:** [July 5, 2021, 6:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425%2F3 "2021-07-05T18:04:14Z")

</div>

Any chance to get some compute for this one? 🙂 Everything should work out of the box with scripts, so I think I can handle it solo if TPUs are available! @patrickvonplaten

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 7, 2021, 12:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-for-italian%2F7425%2F4 "2021-07-07T12:36:56Z")

</div>

Let’s add it!
