# Pre-training for Wav2Vec2-XLSR via Huggingface

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%3E%5C
**Category:** Models\
**Created:** [June 28, 2021, 7:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490 "2021-06-28T07:16:42Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Javen](https://avatars.discourse-cdn.com/v4/letter/j/8491ac/32.png) [@Javen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJaven)%5C
**Post date:** [June 28, 2021, 7:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F1 "2021-06-28T07:16:42Z")

</div>

Hi guys! I note that the most topics are related to fine-tuning a pre-trained model. But if I have got some new unlabeled data, how can I preform the pre-training process via Huggingface?

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [October 11, 2021, 10:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F2 "2021-10-11T22:48:40Z")

</div>

Hey Javen,

We’ve now an official wav2vec2-pretraining example here: [transformers/examples/pytorch/speech-pretraining at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-pretraining)

---

<div class="post-metadata">

**Author:** ![tiena2cva](https://avatars.discourse-cdn.com/v4/letter/t/a87d85/32.png) [@tiena2cva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftiena2cva)%5C
**Post date:** [October 13, 2021, 3:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F3 "2021-10-13T15:48:12Z")

</div>

Hi @patrickvonplaten, I re-run this scripts on [Google colab](https://colab.research.google.com/drive/1IDwie_Te_2GntHay_zZ-oPvriNsacQ8d?usp=sharing). I pass all parameter as same as you recommend on [README](https://github.com/huggingface/transformers/tree/master/examples/pytorch/speech-pretraining#demo) but after some epoch, the loss is not decrease.

```auto
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 5.137e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 8.068e-19
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.952e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 4.017e-19
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.831e-01| ppl: 2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 5.166e-19

```

Can you take a look? what did I miss?

---

<div class="post-metadata">

**Author:** ![ayanas](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@ayanas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fayanas)%5C
**Post date:** [October 17, 2021, 3:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F4 "2021-10-17T15:27:18Z")

</div>

Hi @patrickvonplaten and @tiena2cva,  
Thanks for the new official wav2vec2-pretraining example, this helps a lot!  
I had the same problem as @tiena2cva. Tried to re-run the demo script with the same parameters on my own gpu. After a few epochs the contrastive loss was decreased to zero and the model stopped changing.  
Running inference showed that the quantizer maps all the time steps to the same vector (can be seen at projected\_quantized\_states), which explains the zero contrastive loss.  
I would have thought that the diversity loss weight should be increased, but I used the parameters given in the README file so this behavior is unexpected and may indicate a different problem.

Can you please help?

---

<div class="post-metadata">

**Author:** ![iamaliahmad](https://avatars.discourse-cdn.com/v4/letter/i/ee7513/32.png) [@iamaliahmad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiamaliahmad)%5C
**Post date:** [November 13, 2021, 5:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F5 "2021-11-13T17:57:53Z")

</div>

Hi @patrickvonplaten and @sgugger , I am facing the same issue, constrast\_loss is not changing after 3 backward it goes straight to zero and didn’t change even after many epochs, any idea how can we reproduce the results. Thanks

---

<div class="post-metadata">

**Author:** ![iamaliahmad](https://avatars.discourse-cdn.com/v4/letter/i/ee7513/32.png) [@iamaliahmad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fiamaliahmad)%5C
**Post date:** [November 13, 2021, 5:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F6 "2021-11-13T17:58:55Z")

</div>

Hi, @tiena2cva and @ayanas any update or solution to your problem?

---

<div class="post-metadata">

**Author:** ![Kshitizkhandelwal](https://avatars.discourse-cdn.com/v4/letter/k/c68b51/32.png) [@Kshitizkhandelwal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKshitizkhandelwal)%5C
**Post date:** [November 14, 2022, 1:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F7 "2022-11-14T13:13:05Z")

</div>

hey @tiena2cva . Were you able to pre train your model? If yes, can you share your code?

---

<div class="post-metadata">

**Author:** ![tiena2cva](https://avatars.discourse-cdn.com/v4/letter/t/a87d85/32.png) [@tiena2cva](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftiena2cva)%5C
**Post date:** [November 14, 2022, 4:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F8 "2022-11-14T16:22:43Z")

</div>

Hi @Kshitizkhandelwal, it was a long time ago, let me check in the next day. I have done pre-training wav2vec2 for Vietnamese

---

<div class="post-metadata">

**Author:** ![Kshitizkhandelwal](https://avatars.discourse-cdn.com/v4/letter/k/c68b51/32.png) [@Kshitizkhandelwal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKshitizkhandelwal)%5C
**Post date:** [November 15, 2022, 1:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F9 "2022-11-15T13:41:48Z")

</div>

@tiena2cva If possible please let me know on this?

---

<div class="post-metadata">

**Author:** ![Bauyrjan](https://avatars.discourse-cdn.com/v4/letter/b/4da419/32.png) [@Bauyrjan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBauyrjan)%5C
**Post date:** [November 18, 2022, 7:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F10 "2022-11-18T07:44:47Z")

</div>

Good day! I am consnantly getting an error “realloc of size … failed” and I wanted to know is there any solution to overpass it?

---

<div class="post-metadata">

**Author:** ![mohammadtvk](https://avatars.discourse-cdn.com/v4/letter/m/71e660/32.png) [@mohammadtvk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmohammadtvk)%5C
**Post date:** [January 9, 2023, 8:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F11 "2023-01-09T08:55:15Z")

</div>

I have the same situation here.  
I tried to pretrain wave2vec on 8K and 16K samples. after few steps contrastive loss goes 0, diversity loss shoots up to 1 and perplexity to 2.  
I also tried changing hyper params like learning rate and gumble temp, but no luck  
any updates ?  
@tiena2cva @patrickvonplaten

---

<div class="post-metadata">

**Author:** ![morenolq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morenolq/32/55542_2.png) [@morenolq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmorenolq)%5C
**Post date:** [March 28, 2023, 7:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F12 "2023-03-28T07:04:10Z")

</div>

Do you have any updates? I’m experiencing a similar issue where the diversity loss starts off very high, around 1500, but after the warmup period, it quickly drops down to 0.0 within 10K steps. Conversely, the diversity loss starts at around 300 and gradually increases to 400, then oscillates.

---

<div class="post-metadata">

**Author:** ![OmarEXP](https://avatars.discourse-cdn.com/v4/letter/o/96bed5/32.png) [@OmarEXP](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOmarEXP)%5C
**Post date:** [June 6, 2023, 3:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F13 "2023-06-06T15:50:03Z")

</div>

Hi @mohammadtvk I am experiencing the same problem did you find a solution?

---

<div class="post-metadata">

**Author:** ![mohammadtvk](https://avatars.discourse-cdn.com/v4/letter/m/71e660/32.png) [@mohammadtvk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmohammadtvk)%5C
**Post date:** [June 6, 2023, 6:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F14 "2023-06-06T18:37:37Z")

</div>

unfortunately no. I used fairseq for pretraining

---

<div class="post-metadata">

**Author:** ![morenolq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morenolq/32/55542_2.png) [@morenolq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmorenolq)%5C
**Post date:** [June 9, 2023, 1:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F15 "2023-06-09T13:05:31Z")

</div>

I also switched to fairseq, there are several LR and threshold management things missing in HF I think.

---

<div class="post-metadata">

**Author:** ![bilalhsp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bilalhsp/32/34048_2.png) [@bilalhsp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbilalhsp)%5C
**Post date:** [November 5, 2024, 8:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-for-wav2vec2-xlsr-via-huggingface%2F7490%2F16 "2024-11-05T20:16:09Z")

</div>

> [@Javen](#):
>
> ta, how can I preform the pre-training process via Huggingface?

Hi guys,  
Wav2vec2 paper mentions some customization to the training loop, like gradient scaling, penalty on feature encoder, temperature annealing. I am not sure if they are implemented in the example script or not. Here is a tutorial that lets you customize the training loop by extending the Huggingface Trainer class. The tutorial extends one method of Trainer class but also points to more resources to implement full functionality.

> **[Pretraining wav2vec2 using Huggingface Trainer API](https://medium.com/@bilalhsp/pretraining-wav2vec2-using-huggingface-trainer-api-19fbf2fc6bf2)**
>
> Self-supervised training using audio data.

This might be useful for someone.
