Upload README.md

README.md

@@ -1,218 +1,218 @@
----
-license: apache-2.0  
-base_model: microsoft/MiniLM-L6-v2  
-tags:  
-- transformers  
-- sentence-transformers  
-- sentence-similarity  
-- feature-extraction  
-- text-embeddings-inference  
-- information-retrieval  
-- knowledge-distillation  
-language:
-- en
----
-<div style="display: flex; justify-content: center;">      
-    <div style="display: flex; align-items: center; gap: 10px;">      
-        <img src="logo.webp" alt="MongoDB Logo" style="height: 36px; width: auto; border-radius: 4px;">      
-        <span style="font-size: 32px; font-weight: bold">MongoDB/mdbr-leaf-mt</span>      
-    </div>      
-</div>  
-
-# Content
-
-1. [Introduction](#introduction)
-2. [Technical Report](#technical-report)
-3. [Highlights](#highlights)
-4. [Benchmarks](#benchmark-comparison)
-5. [Quickstart](#quickstart)
-6. [Citation](#citation)
-
-# Introduction
-
-`mdbr-leaf-mt` is a compact high-performance text embedding model designed for classification, clustering, semantic sentence similarity and summarization tasks. 
-
-To enable even greater efficiency, `mdbr-leaf-mt` supports [flexible asymmetric architectures](#asymmetric-retrieval-setup) and is robust to [vector quantization](#vector-quantization) and [MRL truncation](#mrl-truncation).
-
-If you are looking to perform semantic search / information retrieval (e.g. for RAGs), please check out our [`mdbr-leaf-ir`](https://huggingface.co/MongoDB/mdbr-leaf-ir) model, which is specifically trained for these tasks.
-
-> [!Note]  
-> **Note**: this model has been developed by the ML team of MongoDB Research. At the time of writing it is not used in any of MongoDB's commercial product or service offerings.
-
-# Technical Report
-
-A technical report detailing our proposed `LEAF` training procedure will be available soon.
-
-# Highlights  
-
-* **State-of-the-Art Performance**: `mdbr-leaf-mt` achieves new state-of-the-art results for compact embedding models, **ranking #1** on the [public MTEB v2 (Eng) benchmark leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for models with ≤30M parameters.
-* **Flexible Architecture Support**: `mdbr-leaf-mt` supports asymmetric retrieval architectures enabling even greater retrieval results. [See below](#asymmetric-retrieval-setup) for more information.
-* **MRL and Quantization Support**: embedding vectors generated by `mdbr-leaf-mt` compress well when truncated (MRL) and can be stored using more efficient types like `int8` and `binary`.  [See below](#mrl) for more information.
-
-## Benchmark Comparison
-  
-The table below shows the scores for `mdbr-leaf-mt` on the MTEB v2 (English) benchmark, compared to other retrieval models.
-
-`mdbr-leaf-mt` ranks #1 on this benchmark for models with <30M parameters.
-
-| Model                              | Size    | MTEB v2 (Eng) |  
-|------------------------------------|---------|---------------|  
-| OpenAI text-embedding-3-large      | Unknown | 66.43         |  
-| OpenAI text-embedding-3-small      | Unknown | 64.56         |  
-| **mdbr-leaf-mt**                   | 23M     | **63.97**     |  
-| gte-small                          | 33M     | 63.22         |  
-| snowflake-arctic-embed-s           | 32M     | 61.59         |  
-| e5-small-v2                        | 33M     | 61.32         |  
-| granite-embedding-small-english-r2 | 47M     | 61.07         |  
-| all-MiniLM-L6-v2                   | 22M     | 59.03         |  
-
-
-# Quickstart  
-  
-## Sentence Transformers  
-  
-```python  
-from sentence_transformers import SentenceTransformer  
-  
-# Load the model  
-model = SentenceTransformer("MongoDB/mdbr-leaf-mt")  
-  
-# Example queries and documents  
-queries = [
-    "What is machine learning?",  
-    "How does neural network training work?"  
-]  
-  
-documents = [  
-    "Machine learning is a subset of artificial intelligence that focuses on algorithms that can learn from data.",  
-    "Neural networks are trained through backpropagation, adjusting weights to minimize prediction errors."  
-]  
-  
-# Encode queries and documents  
-query_embeddings = model.encode(queries, prompt_name="query")  
-document_embeddings = model.encode(documents)  
-  
-# Compute similarity scores  
-scores = model.similarity(query_embeddings, document_embeddings)  
-
-# Print results
-for i, query in enumerate(queries):
-    print(f"Query: {query}")
-    for j, doc in enumerate(documents):
-        print(f" Similarity: {scores[i, j]:.4f} | Document {j}: {doc[:80]}...")
-
-# Query: What is machine learning?
-#  Similarity: 0.9063 | Document 0: Machine learning is a subset of ...
-#  Similarity: 0.7287 | Document 1: Neural networks are trained ...
-# 
-# Query: How does neural network training work?
-#  Similarity: 0.6725 | Document 0: Machine learning is a subset of ...
-#  Similarity: 0.8287 | Document 1: Neural networks are trained ...
-```
-  
-## Transformers Usage  
-
-See [here](https://huggingface.co/MongoDB/mdbr-leaf-mt/blob/main/transformers_example_mt.ipynb).
-  
-## Asymmetric Retrieval Setup  
-  
-`mdbr-leaf-mt` is *aligned* to [`mxbai-embed-large-v1`](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1), the model it has been distilled from, making the asymmetric system below possible: 
-  
-```python  
-# Use mdbr-leaf-mt for query encoding (real-time, low latency)  
-query_model = SentenceTransformer("MongoDB/mdbr-leaf-mt")  
-query_embeddings = query_model.encode(queries, prompt_name="query")  
-
-# Use a larger model for document encoding (one-time, at index time)  
-doc_model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")  
-document_embeddings = doc_model.encode(documents)  
-  
-# Compute similarities  
-scores = query_model.similarity(query_embeddings, document_embeddings)  
-```
-Retrieval results from asymmetric mode are usually superior to the [standard mode above](#sentence-transformers).
-
-## MRL Truncation
-
-Embeddings have been trained via [MRL](https://arxiv.org/abs/2205.13147) and can be truncated for more efficient storage:
-```python
-from torch.nn import functional as F
-
-query_embeds = model.encode(queries, prompt_name="query", convert_to_tensor=True)
-doc_embeds = model.encode(documents, convert_to_tensor=True)
-
-# Truncate and normalize according to MRL
-query_embeds = F.normalize(query_embeds[:, :256], dim=-1)
-doc_embeds = F.normalize(doc_embeds[:, :256], dim=-1)
-
-similarities = model.similarity(query_embeds, doc_embeds)
-
-print('After MRL:')
-print(f"* Embeddings dimension: {query_embeds.shape[1]}")
-print(f"* Similarities:\n\t{similarities}")
-
-# After MRL:
-# * Embeddings dimension: 256
-# * Similarities:
-# 	  tensor([[0.9164, 0.7219],
-#             [0.6682, 0.8393]], device='cuda:0')
-```
-
-## Vector Quantization
-Vector quantization, for example to `int8` or `binary`, can be performed as follows:
-
-**Note**: For vector quantization to types other than binary, we suggest performing a calibration to determine the optimal ranges, [see here](https://sbert.net/examples/sentence_transformer/applications/embedding-quantization/README.html#scalar-int8-quantization). 
-Good initial values are -1.0 and +1.0.
-```python
-from sentence_transformers.quantization import quantize_embeddings
-import torch
-
-query_embeds = model.encode(queries, prompt_name="query")
-doc_embeds = model.encode(documents)
-
-# Quantize embeddings to int8 using -1.0 and +1.0
-ranges = torch.tensor([[-1.0], [+1.0]]).expand(2, query_embeds.shape[1]).cpu().numpy()
-query_embeds = quantize_embeddings(query_embeds, "int8", ranges=ranges)
-doc_embeds = quantize_embeddings(doc_embeds, "int8", ranges=ranges)
-
-# Calculate similarities; cast to int64 to avoid under/overflow
-similarities = query_embeds.astype(int) @ doc_embeds.astype(int).T
-
-print('After quantization:')
-print(f"* Embeddings type: {query_embeds.dtype}")
-print(f"* Similarities:\n{similarities}")
-
-# After quantization:
-# * Embeddings type: int8
-# * Similarities:
-#   [[2202032 1422868]
-#    [1421197 1845580]]
-```
-
-# Evaluation
-
-Please [see here](https://huggingface.co/MongoDB/mdbr-leaf-mt/blob/main/evaluate_models.ipynb).
-
-# Citation  
-  
-If you use this model in your work, please cite:  
-  
-```bibtex  
-@article{mdb_leaf,  
-  title         = {LEAF: Lightweight Embedding Alignment Knowledge Distillation Framework},  
-  author        = {Robin Vujanic and Thomas Rueckstiess},  
-  year          = {2025}
-  eprint        = {TBD},
-  archiveprefix = {arXiv},
-  primaryclass  = {FILL HERE},
-  url           = {FILL HERE}
-}  
-```  
-  
-# License  
-  
-This model is released under Apache 2.0 License.  
-  
-# Contact  
-  
+---
+license: apache-2.0  
+base_model: microsoft/MiniLM-L6-v2  
+tags:  
+- transformers  
+- sentence-transformers  
+- sentence-similarity  
+- feature-extraction  
+- text-embeddings-inference  
+- information-retrieval  
+- knowledge-distillation  
+language:
+- en
+---
+<div style="display: flex; justify-content: center;">      
+    <div style="display: flex; align-items: center; gap: 10px;">      
+        <img src="logo.webp" alt="MongoDB Logo" style="height: 36px; width: auto; border-radius: 4px;">      
+        <span style="font-size: 32px; font-weight: bold">MongoDB/mdbr-leaf-mt</span>      
+    </div>      
+</div>  
+
+# Content
+
+1. [Introduction](#introduction)
+2. [Technical Report](#technical-report)
+3. [Highlights](#highlights)
+4. [Benchmarks](#benchmark-comparison)
+5. [Quickstart](#quickstart)
+6. [Citation](#citation)
+
+# Introduction
+
+`mdbr-leaf-mt` is a compact high-performance text embedding model designed for classification, clustering, semantic sentence similarity and summarization tasks. 
+
+To enable even greater efficiency, `mdbr-leaf-mt` supports [flexible asymmetric architectures](#asymmetric-retrieval-setup) and is robust to [vector quantization](#vector-quantization) and [MRL truncation](#mrl-truncation).
+
+If you are looking to perform semantic search / information retrieval (e.g. for RAGs), please check out our [`mdbr-leaf-ir`](https://huggingface.co/MongoDB/mdbr-leaf-ir) model, which is specifically trained for these tasks.
+
+> [!Note]  
+> **Note**: this model has been developed by the ML team of MongoDB Research. At the time of writing it is not used in any of MongoDB's commercial product or service offerings.
+
+# Technical Report
+
+A technical report detailing our proposed `LEAF` training procedure is [available here](https://arxiv.org/abs/2509.12539).
+
+# Highlights  
+
+* **State-of-the-Art Performance**: `mdbr-leaf-mt` achieves new state-of-the-art results for compact embedding models, **ranking #1** on the [public MTEB v2 (Eng) benchmark leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for models with ≤30M parameters.
+* **Flexible Architecture Support**: `mdbr-leaf-mt` supports asymmetric retrieval architectures enabling even greater retrieval results. [See below](#asymmetric-retrieval-setup) for more information.
+* **MRL and Quantization Support**: embedding vectors generated by `mdbr-leaf-mt` compress well when truncated (MRL) and can be stored using more efficient types like `int8` and `binary`.  [See below](#mrl) for more information.
+
+## Benchmark Comparison
+  
+The table below shows the scores for `mdbr-leaf-mt` on the MTEB v2 (English) benchmark, compared to other retrieval models.
+
+`mdbr-leaf-mt` ranks #1 on this benchmark for models with <30M parameters.
+
+| Model                              | Size    | MTEB v2 (Eng) |  
+|------------------------------------|---------|---------------|  
+| OpenAI text-embedding-3-large      | Unknown | 66.43         |  
+| OpenAI text-embedding-3-small      | Unknown | 64.56         |  
+| **mdbr-leaf-mt**                   | 23M     | **63.97**     |  
+| gte-small                          | 33M     | 63.22         |  
+| snowflake-arctic-embed-s           | 32M     | 61.59         |  
+| e5-small-v2                        | 33M     | 61.32         |  
+| granite-embedding-small-english-r2 | 47M     | 61.07         |  
+| all-MiniLM-L6-v2                   | 22M     | 59.03         |  
+
+
+# Quickstart  
+  
+## Sentence Transformers  
+  
+```python  
+from sentence_transformers import SentenceTransformer  
+  
+# Load the model  
+model = SentenceTransformer("MongoDB/mdbr-leaf-mt")  
+  
+# Example queries and documents  
+queries = [
+    "What is machine learning?",  
+    "How does neural network training work?"  
+]  
+  
+documents = [  
+    "Machine learning is a subset of artificial intelligence that focuses on algorithms that can learn from data.",  
+    "Neural networks are trained through backpropagation, adjusting weights to minimize prediction errors."  
+]  
+  
+# Encode queries and documents  
+query_embeddings = model.encode(queries, prompt_name="query")  
+document_embeddings = model.encode(documents)  
+  
+# Compute similarity scores  
+scores = model.similarity(query_embeddings, document_embeddings)  
+
+# Print results
+for i, query in enumerate(queries):
+    print(f"Query: {query}")
+    for j, doc in enumerate(documents):
+        print(f" Similarity: {scores[i, j]:.4f} | Document {j}: {doc[:80]}...")
+
+# Query: What is machine learning?
+#  Similarity: 0.9063 | Document 0: Machine learning is a subset of ...
+#  Similarity: 0.7287 | Document 1: Neural networks are trained ...
+# 
+# Query: How does neural network training work?
+#  Similarity: 0.6725 | Document 0: Machine learning is a subset of ...
+#  Similarity: 0.8287 | Document 1: Neural networks are trained ...
+```
+  
+## Transformers Usage  
+
+See [here](https://huggingface.co/MongoDB/mdbr-leaf-mt/blob/main/transformers_example_mt.ipynb).
+  
+## Asymmetric Retrieval Setup  
+  
+`mdbr-leaf-mt` is *aligned* to [`mxbai-embed-large-v1`](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1), the model it has been distilled from, making the asymmetric system below possible: 
+  
+```python  
+# Use mdbr-leaf-mt for query encoding (real-time, low latency)  
+query_model = SentenceTransformer("MongoDB/mdbr-leaf-mt")  
+query_embeddings = query_model.encode(queries, prompt_name="query")  
+
+# Use a larger model for document encoding (one-time, at index time)  
+doc_model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")  
+document_embeddings = doc_model.encode(documents)  
+  
+# Compute similarities  
+scores = query_model.similarity(query_embeddings, document_embeddings)  
+```
+Retrieval results from asymmetric mode are usually superior to the [standard mode above](#sentence-transformers).
+
+## MRL Truncation
+
+Embeddings have been trained via [MRL](https://arxiv.org/abs/2205.13147) and can be truncated for more efficient storage:
+```python
+from torch.nn import functional as F
+
+query_embeds = model.encode(queries, prompt_name="query", convert_to_tensor=True)
+doc_embeds = model.encode(documents, convert_to_tensor=True)
+
+# Truncate and normalize according to MRL
+query_embeds = F.normalize(query_embeds[:, :256], dim=-1)
+doc_embeds = F.normalize(doc_embeds[:, :256], dim=-1)
+
+similarities = model.similarity(query_embeds, doc_embeds)
+
+print('After MRL:')
+print(f"* Embeddings dimension: {query_embeds.shape[1]}")
+print(f"* Similarities:\n\t{similarities}")
+
+# After MRL:
+# * Embeddings dimension: 256
+# * Similarities:
+# 	  tensor([[0.9164, 0.7219],
+#             [0.6682, 0.8393]], device='cuda:0')
+```
+
+## Vector Quantization
+Vector quantization, for example to `int8` or `binary`, can be performed as follows:
+
+**Note**: For vector quantization to types other than binary, we suggest performing a calibration to determine the optimal ranges, [see here](https://sbert.net/examples/sentence_transformer/applications/embedding-quantization/README.html#scalar-int8-quantization). 
+Good initial values are -1.0 and +1.0.
+```python
+from sentence_transformers.quantization import quantize_embeddings
+import torch
+
+query_embeds = model.encode(queries, prompt_name="query")
+doc_embeds = model.encode(documents)
+
+# Quantize embeddings to int8 using -1.0 and +1.0
+ranges = torch.tensor([[-1.0], [+1.0]]).expand(2, query_embeds.shape[1]).cpu().numpy()
+query_embeds = quantize_embeddings(query_embeds, "int8", ranges=ranges)
+doc_embeds = quantize_embeddings(doc_embeds, "int8", ranges=ranges)
+
+# Calculate similarities; cast to int64 to avoid under/overflow
+similarities = query_embeds.astype(int) @ doc_embeds.astype(int).T
+
+print('After quantization:')
+print(f"* Embeddings type: {query_embeds.dtype}")
+print(f"* Similarities:\n{similarities}")
+
+# After quantization:
+# * Embeddings type: int8
+# * Similarities:
+#   [[2202032 1422868]
+#    [1421197 1845580]]
+```
+
+# Evaluation
+
+Please [see here](https://huggingface.co/MongoDB/mdbr-leaf-mt/blob/main/evaluate_models.ipynb).
+
+# Citation  
+  
+If you use this model in your work, please cite:  
+  
+```bibtex  
+@misc{mdbr_leaf,
+      title={LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations}, 
+      author={Robin Vujanic and Thomas Rueckstiess},
+      year={2025},
+      eprint={2509.12539},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2509.12539}, 
+}
+```  
+  
+# License  
+  
+This model is released under Apache 2.0 License.  
+  
+# Contact  
+  
 For questions or issues, please open an issue or pull request. You can also contact the MongoDB ML Research team at [email protected].