wwydmanski/modernbert-bio-v0.1

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'what is the connection between cancer and the cell cycle?',
    'Conclusion. Cancer is unchecked cell growth. Mutations in genes can cause cancer by accelerating cell division rates or inhibiting normal controls on the system, such as cell cycle arrest or programmed cell death. As a mass of cancerous cells grows, it can develop into a tumor.',
    'Biology',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoNQ and NanoMSMARCO
• Evaluated with InformationRetrievalEvaluator

Metric	NanoNQ	NanoMSMARCO
cosine_accuracy@1	0.1	0.12
cosine_accuracy@3	0.18	0.28
cosine_accuracy@5	0.24	0.4
cosine_accuracy@10	0.34	0.52
cosine_precision@1	0.1	0.12
cosine_precision@3	0.06	0.0933
cosine_precision@5	0.048	0.08
cosine_precision@10	0.034	0.052
cosine_recall@1	0.1	0.12
cosine_recall@3	0.15	0.28
cosine_recall@5	0.21	0.4
cosine_recall@10	0.31	0.52
cosine_ndcg@10	0.1934	0.2985
cosine_mrr@10	0.1659	0.2304
cosine_map@100	0.1764	0.2469

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator

Metric	Value
cosine_accuracy@1	0.11
cosine_accuracy@3	0.23
cosine_accuracy@5	0.32
cosine_accuracy@10	0.43
cosine_precision@1	0.11
cosine_precision@3	0.0767
cosine_precision@5	0.064
cosine_precision@10	0.043
cosine_recall@1	0.11
cosine_recall@3	0.215
cosine_recall@5	0.305
cosine_recall@10	0.415
cosine_ndcg@10	0.246
cosine_mrr@10	0.1982
cosine_map@100	0.2117

Training Details

Training Dataset

csv

• Dataset: csv
• Size: 100,006 training samples
• Columns: question, answer, and category
• Approximate statistics based on the first 1000 samples:

  	question	answer	category
  type	string	string	string
  details	min: 8 tokens mean: 11.91 tokens max: 20 tokens	min: 16 tokens mean: 57.49 tokens max: 136 tokens	min: 4 tokens mean: 4.0 tokens max: 4 tokens

• Samples:

  question	answer	category
  how many times a week should you use heat on your hair?	Don't style hair with heat every day. Hot tools can also make hair look crispy and create split ends if overused. Blow out hair 3-5 times a week and try to limit your flat iron/curling iron usage to 1-2 times a week.”	Medicine
  do african violets like to be root bound?	African violets only bloom when they're root bound. When it is time to repot, be sure to use an organic potting soil made specifically for African violets, such as Espoma's African Violet Mix. They flower best in small pots — choose one that's about a third of the diameter of their leaf spread.	Biology
  is pgwp exempt from lmia?	The PGWP is exempt from Labour Market Impact Assessment (LMIA) requirements. The candidate must have attended a recognized post-secondary school, or a secondary school that offers qualifying programs, for at least eight months.	Medicine

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 0.0001
• num_train_epochs: 1
• warmup_ratio: 0.05
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	NanoNQ_cosine_ndcg@10	NanoMSMARCO_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10
0	0	-	0.0388	0.0863	0.0626
0.0763	10	0.5482	-	-	-
0.1527	20	0.1079	-	-	-
0.2290	30	0.1491	-	-	-
0.3053	40	0.1381	-	-	-
0.3817	50	0.0873	0.0909	0.2197	0.1553
0.4580	60	0.133	-	-	-
0.5344	70	0.0539	-	-	-
0.6107	80	0.029	-	-	-
0.6870	90	0.0008	-	-	-
0.7634	100	0.0997	0.1982	0.2657	0.2320
0.8397	110	0.04	-	-	-
0.9160	120	0.0053	-	-	-
0.9924	130	0.0095	-	-	-
1.0	131	-	0.1934	0.2985	0.2460

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0.dev0
• PyTorch: 2.5.1
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}