# Fine-tune a translation or train from sratch?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%3E%5C
**Category:** Beginners\
**Created:** [April 1, 2022, 1:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362 "2022-04-01T13:27:50Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rickySaka](https://avatars.discourse-cdn.com/v4/letter/r/ecae2f/32.png) [@rickySaka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FrickySaka)%5C
**Post date:** [April 1, 2022, 1:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F1 "2022-04-01T13:27:50Z")

</div>

Hello here. I hope this question wasn’t asked here before ask I first looked around.  
I’m still studying HF Transformers. I tried to fine-tune a machine translation model « Helsinki-NLP/orpus-mt-en-mul » on a local dialect.  
I used en-mul because the target language doesn’t appear on the list of languages this model checkpoint was trained on, like any other in the hub. So I can’t use the pretrained tokenizer as it doesn’t know my text. So please I wanted to know if my only bet is training a BART or MBART from scratch as that’s what I feel like after to research.

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNimaBoscarino)%5C
**Post date:** [April 1, 2022, 11:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F2 "2022-04-01T23:41:20Z")

</div>

Welcome rickySaka! This is a bit outside of my expertise, but I took a shot at researching it too and I think I’ve landed on the same conclusion you have. From [this thread](https://github.com/huggingface/transformers/issues/2691) and [this thread](https://github.com/huggingface/tokenizers/issues/747), my understanding is that you’ll want to train from scratch. Specifically:

> [In case you have a plenty of new words (e.g. technical terms) or even a different language, it might makes sense to start from scratch](https://github.com/huggingface/transformers/issues/2691#issuecomment-587473545)

---

<div class="post-metadata">

**Author:** ![rickySaka](https://avatars.discourse-cdn.com/v4/letter/r/ecae2f/32.png) [@rickySaka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FrickySaka)%5C
**Post date:** [April 2, 2022, 12:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F3 "2022-04-02T00:03:58Z")

</div>

> [@NimaBoscarino](#):
>
> In case you have a plenty of new words (e.g. technical terms) or even a different language, it might makes sense to start from scratch

Thanks @NimaBoscarino. I will have a look at them.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fosanseviero)%5C
**Post date:** [April 5, 2022, 9:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F4 "2022-04-05T09:33:01Z")

</div>

Is there any model translating to a target language that is similar to your target language. Cross-lingual transfer could help leverage some knowledge from a model that translates from english to italian to e.g. english to spanish, even if the two languages are different.

This really depends on the amount of data you have, and you might likely want to go with scratch training as @NimaBoscarino suggested, but wanted to bring that option as well. @lewtun et. al. recent book has a whole chapter about Multilingual NER and the concepts from it can be used as well. The code is open sourced, so you might want to check the " Cross-Lingual Transfer" and “When Does Zero-Shot Transfer Make Sense?” sections of [its notebook](https://github.com/nlp-with-transformers/notebooks/blob/main/04_multilingual-ner.ipynb) or [take a look at the book](https://www.oreilly.com/library/view/natural-language-processing/9781098103231/).

---

<div class="post-metadata">

**Author:** ![rickySaka](https://avatars.discourse-cdn.com/v4/letter/r/ecae2f/32.png) [@rickySaka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FrickySaka)%5C
**Post date:** [April 7, 2022, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F5 "2022-04-07T15:14:49Z")

</div>

Thank you for your suggestions. I will have to check which language that can be close to my target language. I have around 5000 translation pair from this dataset, that’s why training from scratch will be my last solution as the data is really small. I observed interesting results though with a multilingual model(eng-mul) with a validation blue score of 21 after 20 epochs using the model’s tokenizer. Still working on it.  
Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Terence](https://avatars.discourse-cdn.com/v4/letter/t/aca169/32.png) [@Terence](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTerence)%5C
**Post date:** [April 13, 2022, 10:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F6 "2022-04-13T10:27:04Z")

</div>

Hi @rickySaka , I am hoping to benefit from cross-lingual transfer as I want to fine-tune the opus-mt-am-sv (Amharic-Swedish) pretrained model with a 650K sentence pair Amharic-English dataset. If it works out I will post here.

---

<div class="post-metadata">

**Author:** ![rickySaka](https://avatars.discourse-cdn.com/v4/letter/r/ecae2f/32.png) [@rickySaka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FrickySaka)%5C
**Post date:** [April 15, 2022, 10:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-a-translation-or-train-from-sratch%2F16362%2F7 "2022-04-15T10:08:27Z")

</div>

Great. Feel free.  
I discovered that sacreblue score I announced (20, 21) decreased considerably to 18 2 days ago with the same dataset, checkpoint(eng-mul) and tokenizer. I don’t know why though
