# Help with finetuning mBART on an unseen language

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%3E%5C
**Category:** Models\
**Created:** [October 13, 2020, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507 "2020-10-13T15:25:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 13, 2020, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F1 "2020-10-13T15:25:09Z")

</div>

Hi everyone,  
I wanted to know how would we finetune mBART on a summarization task on a different language than that of English. Also, how can we finetune mBART on a translation task where one of the languages is not present in the language code list that mBART has been trained on.  
Appreciate any help!! Thank you.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 13, 2020, 3:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F2 "2020-10-13T15:57:32Z")

</div>

Hi @LaibaMehnaz

DISCLAIMER: I haven’t tried this myself , and as Sam found in his experiments mBART doesn’t always give good results.

for mBart source seq ends with src lang id and tgt seq starts with the tgt seq id. so for summ you can pass the same lang id for both source and tgt lang and then finetune it the same way you finetune any other seq2seq model.

For translation, if the lang is not present, you can try without using any lang id in the sequences.

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 14, 2020, 2:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F3 "2020-10-14T14:01:57Z")

</div>

Hi @valhalla,  
I did try using mBart without any lang id, but it gives me this error:

self.cur\_lang\_code = self.lang\_code\_to\_id[src\_lang]  
KeyError: ‘’

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 14, 2020, 4:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F4 "2020-10-14T16:01:06Z")

</div>

Also, when I am using the same language code on both the sides, the generations are in a totally different script.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 16, 2020, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F5 "2020-10-16T17:19:55Z")

</div>

For this, you’ll need tokenize input and output seq without using `prepare_seq2seq_batch` method or override `prepare_seq2seq_batch` and modify it to not use lang id

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 19, 2020, 1:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F6 "2020-10-19T13:02:10Z")

</div>

I modified the tokenizer to not use the lang id as you suggested, but still the same problem. ROUGE is 0.0, as the generations are in another script.

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 20, 2020, 6:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F7 "2020-10-20T06:16:03Z")

</div>

Also, is it possible because I am using tiny-mbart and not mbart-large-cc25. I was trying out tiny-mbart due to memory constraints.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 20, 2020, 6:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F8 "2020-10-20T06:45:03Z")

</div>

hi @LaibaMehnaz  
`tiny-mbart ` is just meant for testing, it’s randomly initialised model.

You could try to create a smaller student using `make_student.py` [script](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/make_student.py).

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 20, 2020, 7:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F9 "2020-10-20T07:55:55Z")

</div>

Oh, thanks a lot. I will proceed this way and let you know. Thanks again:)

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 20, 2020, 8:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F10 "2020-10-20T08:15:01Z")

</div>

Also, how many encoder and decoder layers would you suggest?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 20, 2020, 11:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F11 "2020-10-20T11:38:38Z")

</div>

hard to say, depends on problem. But you could start with same number of encoder layers and 6 decoder layers distillbart-12-6 performs really well on summarization

---

<div class="post-metadata">

**Author:** ![LaibaMehnaz](https://avatars.discourse-cdn.com/v4/letter/l/41988e/32.png) [@LaibaMehnaz](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLaibaMehnaz)%5C
**Post date:** [October 20, 2020, 11:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F12 "2020-10-20T11:54:31Z")

</div>

Alright, thank you so much.

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZhylkaaa)%5C
**Post date:** [October 23, 2020, 12:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F13 "2020-10-23T00:15:01Z")

</div>

Hi, I am also interested in topic and I am trying to add mBart functionality to another library, but I have encountered strange error: [https://huggingface.co/transformers/model\_doc/mbart.html](https://huggingface.co/transformers/model_doc/mbart.html) states that `prepare_seq2seq_batch` should give me dict with this keys: `[input_ids, attention_mask, decoder_input_ids, decoder_attention_mask]`, but actually it gives me `[input_ids, attention_mask, labels]`. I am a bit confused 🙂  
Is it a bug or me doing something wrong?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 24, 2020, 7:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F14 "2020-10-24T07:28:42Z")

</div>

Hi @Zhylkaaa

The doc is incorrect. The `prepare_seq2seq_batch` returns `[input_ids, attention_mask, labels]`, and it’s not a bug

cc. @sshleifer

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZhylkaaa)%5C
**Post date:** [October 24, 2020, 7:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F15 "2020-10-24T19:33:02Z")

</div>

Hi @valhalla  
thanks for your respond, but how I am supposed to create decoder inputs? because there is difference in lang\_id position  
should I use something like:  
`[lang_id] + prepare_seq2seq_batch(decoder_input)['input_ids'][:-1]` + padding if required?  
Or should I just modify `prepare_seq2seq_batch` throwing away lang\_id for summarisation task? (I am not sure about this modification because my intuition tells me that lang\_id is something like language conditioned [CLS] token, or my intuition is wrong again😁?)  
Thanks!

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZhylkaaa)%5C
**Post date:** [October 27, 2020, 3:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F16 "2020-10-27T03:42:33Z")

</div>

I have read that I can tag @sshleifer for summarisation and BART problems/questions. Sorry if I am wrong.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 29, 2020, 5:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F17 "2020-10-29T17:27:26Z")

</div>

You can keep lang id for summarisation, you could pass the same lang id as `src_lang` and `tgt_lang` to `prepare_seq2seq_batch` method

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZhylkaaa)%5C
**Post date:** [October 29, 2020, 9:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F18 "2020-10-29T21:05:37Z")

</div>

Thank you @valhalla,  
but what about `decoder_input_ids`? because I doesn’t receive this value after I use `prepare_seq2seq_batch`

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 30, 2020, 5:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F19 "2020-10-30T05:45:33Z")

</div>

`finetune.py` and `finetune_trainer.py` will make the right right `deocder_input_ids`, you won’t need to pass them 🙂

---

<div class="post-metadata">

**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZhylkaaa)%5C
**Post date:** [October 30, 2020, 8:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhelp-with-finetuning-mbart-on-an-unseen-language%2F1507%2F20 "2020-10-30T20:41:27Z")

</div>

thanks, actually I’ve been digging through source code and found that `forward` method actually generates` decoder_input_ids` from `labels` through `shift_tokens_right`. Thank you for help, and sorry for being annoying, should have checked source code first 🙂
