# Domain adaptation for embeddings - fine tuning on MLM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-for-embeddings-fine-tuning-on-mlm%2F96454%3E%5C
**Category:** Beginners\
**Created:** [July 9, 2024, 6:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-for-embeddings-fine-tuning-on-mlm%2F96454 "2024-07-09T18:13:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon73276906](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@anon73276906](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon73276906)%5C
**Post date:** [July 9, 2024, 6:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-for-embeddings-fine-tuning-on-mlm%2F96454%2F1 "2024-07-09T18:13:55Z")

</div>

I would like to create better search functionality for a domain specific language (DSL).

For this, I’m trying to finetune an encoder on the masked language modeling (MLM) objective as described here: [Fine-tuning a masked language model - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/en/chapter7/3). Similar to this question, except I’m leaving the tokenizer as is: [Domain adaptation of Language Model and Tokenizer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-of-language-model-and-tokenizer%2F55680%2F1)

I’ve tried two base-models: all-MiniLM-L6-v2 and roberta-base and finetuned them on about 30k samples from the DSL with 15% masking.

To evaluate the results, I use some semantically equivalent/similar pairs and see how well I can retrieve one from a line-up by encoding the other. Similar to InfoNCE loss, but with rankings instead of probabilities.

With both models I’ve tried, I find that as the MLM loss decreases (on eval, not just train), the actual metrics I care about get worse. In other words, roberta-base is better at searching for similar DSL pairs than a model that’s actually seen the DSL.

I realize I should also train on an InfoNCE style loss at some point (e.g. [Losses — Sentence Transformers documentation](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss)), but there’s not enough data for that at the moment.

Shouldn’t there already be some improvement from training on MLM?

Would appreciate any thoughts/pointers/references!

---

<div class="post-metadata">

**Author:** ![anon73276906](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@anon73276906](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon73276906)%5C
**Post date:** [July 12, 2024, 10:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-for-embeddings-fine-tuning-on-mlm%2F96454%2F2 "2024-07-12T10:55:45Z")

</div>

Found an answer here: [MLM — Sentence Transformers documentation](https://sbert.net/examples/unsupervised_learning/MLM/README.html)

> **Note:** Only running MLM will not yield good sentence embeddings. But you can first tune your favorite transformer model with MLM on your domain specific data. Then you can fine-tune the model with the labeled data you have or using other data sets like [NLI](https://sbert.net/examples/training/nli/README.html), [Paraphrases](https://sbert.net/examples/training/paraphrases/README.html), or [STS](https://sbert.net/examples/training/sts/README.html).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [July 12, 2024, 10:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-for-embeddings-fine-tuning-on-mlm%2F96454%2F3 "2024-07-12T22:56:17Z")

</div>

Closed on July 12, 2024, 10:56pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
