magnifi/bge-small-en-v1-5-ft-orc_fold5

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'what sectors am i in',
    '[{"get_portfolio(None,True,None)": "portfolio"}, {"factor_contribution(\'portfolio\',\'<DATES>\',\'sector\',None,\'portfolio\')": "portfolio"}]',
    '[{"get_portfolio(None,True,None)": "portfolio"}, {"stress_test(\'portfolio\',\'eurostoxx_600\',None,\'down\')": "stress_test"}]',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8205
cosine_accuracy@3	0.9744
cosine_accuracy@5	0.9872
cosine_accuracy@10	1.0
cosine_precision@1	0.8205
cosine_precision@3	0.3248
cosine_precision@5	0.1974
cosine_precision@10	0.1
cosine_recall@1	0.0228
cosine_recall@3	0.0271
cosine_recall@5	0.0274
cosine_recall@10	0.0278
cosine_ndcg@10	0.2029
cosine_mrr@10	0.8958
cosine_map@100	0.0249

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,541 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 4 tokens mean: 12.78 tokens max: 28 tokens	min: 20 tokens mean: 75.46 tokens max: 269 tokens

• Samples:

  sentence_0	sentence_1
  [TICKER] vs peers based on [ATTRIBUTE]	[{"search('query', 'match_type', '')": "search_results"},{"compare([[''],'search_results'], [''], None)": "comparison_data"}]
  How will the rising UK stock index affect my portfolio?	[{"get_portfolio(None,True,None)": "portfolio"}, {"stress_test('portfolio','ftse_100',None,'up')": "stress_test"}]
  How much did [A_SECTOR] sector move [DATES]?	[{"get_attribute([''],['returns'],'')":"sector_returns"}]

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 6
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand

Epoch	Step	Training Loss	cosine_ndcg@10
0.0129	2	-	0.0999
0.0258	4	-	0.1004
0.0387	6	-	0.1009
0.0516	8	-	0.1031
0.0645	10	-	0.1045
0.0774	12	-	0.1064
0.0903	14	-	0.1097
0.1032	16	-	0.1117
0.1161	18	-	0.1143
0.1290	20	-	0.1175
0.1419	22	-	0.1200
0.1548	24	-	0.1239
0.1677	26	-	0.1261
0.1806	28	-	0.1296
0.1935	30	-	0.1310
0.2065	32	-	0.1322
0.2194	34	-	0.1361
0.2323	36	-	0.1403
0.2452	38	-	0.1426
0.2581	40	-	0.1442
0.2710	42	-	0.1458
0.2839	44	-	0.1474
0.2968	46	-	0.1478
0.3097	48	-	0.1487
0.3226	50	-	0.1482
0.3355	52	-	0.1483
0.3484	54	-	0.1495
0.3613	56	-	0.1497
0.3742	58	-	0.1491
0.3871	60	-	0.1499
0.4	62	-	0.1512
0.4129	64	-	0.1524
0.4258	66	-	0.1540
0.4387	68	-	0.1552
0.4516	70	-	0.1562
0.4645	72	-	0.1570
0.4774	74	-	0.1600
0.4903	76	-	0.1607
0.5032	78	-	0.1614
0.5161	80	-	0.1616
0.5290	82	-	0.1612
0.5419	84	-	0.1626
0.5548	86	-	0.1642
0.5677	88	-	0.1646
0.5806	90	-	0.1653
0.5935	92	-	0.1663
0.6065	94	-	0.1668
0.6194	96	-	0.1678
0.6323	98	-	0.1682
0.6452	100	-	0.1683
0.6581	102	-	0.1689
0.6710	104	-	0.1705
0.6839	106	-	0.1696
0.6968	108	-	0.1713
0.7097	110	-	0.1719
0.7226	112	-	0.1723
0.7355	114	-	0.1730
0.7484	116	-	0.1739
0.7613	118	-	0.1746
0.7742	120	-	0.1751
0.7871	122	-	0.1760
0.8	124	-	0.1773
0.8129	126	-	0.1791
0.8258	128	-	0.1794
0.8387	130	-	0.1792
0.8516	132	-	0.1779
0.8645	134	-	0.1779
0.8774	136	-	0.1775
0.8903	138	-	0.1783
0.9032	140	-	0.1784
0.9161	142	-	0.1795
0.9290	144	-	0.1797
0.9419	146	-	0.1799
0.9548	148	-	0.1815
0.9677	150	-	0.1823
0.9806	152	-	0.1829
0.9935	154	-	0.1839
1.0	155	-	0.1847
1.0065	156	-	0.1841
1.0194	158	-	0.1839
1.0323	160	-	0.1840
1.0452	162	-	0.1844
1.0581	164	-	0.1842
1.0710	166	-	0.1837
1.0839	168	-	0.1829
1.0968	170	-	0.1833
1.1097	172	-	0.1837
1.1226	174	-	0.1841
1.1355	176	-	0.1832
1.1484	178	-	0.1826
1.1613	180	-	0.1828
1.1742	182	-	0.1824
1.1871	184	-	0.1816
1.2	186	-	0.1812
1.2129	188	-	0.1804
1.2258	190	-	0.1814
1.2387	192	-	0.1806
1.2516	194	-	0.1805
1.2645	196	-	0.1810
1.2774	198	-	0.1828
1.2903	200	-	0.1834
1.3032	202	-	0.1823
1.3161	204	-	0.1827
1.3290	206	-	0.1830
1.3419	208	-	0.1832
1.3548	210	-	0.1829
1.3677	212	-	0.1839
1.3806	214	-	0.1857
1.3935	216	-	0.1855
1.4065	218	-	0.1856
1.4194	220	-	0.1862
1.4323	222	-	0.1858
1.4452	224	-	0.1853
1.4581	226	-	0.1862
1.4710	228	-	0.1872
1.4839	230	-	0.1877
1.4968	232	-	0.1878
1.5097	234	-	0.1878
1.5226	236	-	0.1882
1.5355	238	-	0.1882
1.5484	240	-	0.1877
1.5613	242	-	0.1884
1.5742	244	-	0.1885
1.5871	246	-	0.1892
1.6	248	-	0.1888
1.6129	250	-	0.1888
1.6258	252	-	0.1887
1.6387	254	-	0.1884
1.6516	256	-	0.1883
1.6645	258	-	0.1891
1.6774	260	-	0.1887
1.6903	262	-	0.1891
1.7032	264	-	0.1889
1.7161	266	-	0.1888
1.7290	268	-	0.1893
1.7419	270	-	0.1895
1.7548	272	-	0.1897
1.7677	274	-	0.1900
1.7806	276	-	0.1906
1.7935	278	-	0.1901
1.8065	280	-	0.1902
1.8194	282	-	0.1906
1.8323	284	-	0.1903
1.8452	286	-	0.1901
1.8581	288	-	0.1911
1.8710	290	-	0.1915
1.8839	292	-	0.1915
1.8968	294	-	0.1917
1.9097	296	-	0.1914
1.9226	298	-	0.1916
1.9355	300	-	0.1918
1.9484	302	-	0.1919
1.9613	304	-	0.1924
1.9742	306	-	0.1926
1.9871	308	-	0.1922
2.0	310	-	0.1923
2.0129	312	-	0.1917
2.0258	314	-	0.1919
2.0387	316	-	0.1923
2.0516	318	-	0.1923
2.0645	320	-	0.1925
2.0774	322	-	0.1918
2.0903	324	-	0.1927
2.1032	326	-	0.1927
2.1161	328	-	0.1934
2.1290	330	-	0.1928
2.1419	332	-	0.1927
2.1548	334	-	0.1931
2.1677	336	-	0.1931
2.1806	338	-	0.1937
2.1935	340	-	0.1937
2.2065	342	-	0.1938
2.2194	344	-	0.1936
2.2323	346	-	0.1934
2.2452	348	-	0.1935
2.2581	350	-	0.1937
2.2710	352	-	0.1937
2.2839	354	-	0.1941
2.2968	356	-	0.1945
2.3097	358	-	0.1950
2.3226	360	-	0.1947
2.3355	362	-	0.1940
2.3484	364	-	0.1945
2.3613	366	-	0.1951
2.3742	368	-	0.1954
2.3871	370	-	0.1954
2.4	372	-	0.1953
2.4129	374	-	0.1956
2.4258	376	-	0.1952
2.4387	378	-	0.1952
2.4516	380	-	0.1955
2.4645	382	-	0.1951
2.4774	384	-	0.1957
2.4903	386	-	0.1964
2.5032	388	-	0.1965
2.5161	390	-	0.1965
2.5290	392	-	0.1973
2.5419	394	-	0.1976
2.5548	396	-	0.1972
2.5677	398	-	0.1962
2.5806	400	-	0.1966
2.5935	402	-	0.1965
2.6065	404	-	0.1966
2.6194	406	-	0.1964
2.6323	408	-	0.1970
2.6452	410	-	0.1970
2.6581	412	-	0.1977
2.6710	414	-	0.1974
2.6839	416	-	0.1973
2.6968	418	-	0.1965
2.7097	420	-	0.1965
2.7226	422	-	0.1968
2.7355	424	-	0.1975
2.7484	426	-	0.1976
2.7613	428	-	0.1973
2.7742	430	-	0.1974
2.7871	432	-	0.1973
2.8	434	-	0.1971
2.8129	436	-	0.1974
2.8258	438	-	0.1963
2.8387	440	-	0.1955
2.8516	442	-	0.1951
2.8645	444	-	0.1947
2.8774	446	-	0.1946
2.8903	448	-	0.1947
2.9032	450	-	0.1952
2.9161	452	-	0.1956
2.9290	454	-	0.1954
2.9419	456	-	0.1953
2.9548	458	-	0.1956
2.9677	460	-	0.1964
2.9806	462	-	0.1960
2.9935	464	-	0.1960
3.0	465	-	0.1956
3.0065	466	-	0.1953
3.0194	468	-	0.1953
3.0323	470	-	0.1964
3.0452	472	-	0.1973
3.0581	474	-	0.1973
3.0710	476	-	0.1963
3.0839	478	-	0.1965
3.0968	480	-	0.1973
3.1097	482	-	0.1985
3.1226	484	-	0.1988
3.1355	486	-	0.1988
3.1484	488	-	0.1991
3.1613	490	-	0.1990
3.1742	492	-	0.1990
3.1871	494	-	0.1993
3.2	496	-	0.1993
3.2129	498	-	0.1991
3.2258	500	0.3401	0.2005
3.2387	502	-	0.2001
3.2516	504	-	0.2003
3.2645	506	-	0.2009
3.2774	508	-	0.2009
3.2903	510	-	0.2011
3.3032	512	-	0.2007
3.3161	514	-	0.2006
3.3290	516	-	0.2006
3.3419	518	-	0.2005
3.3548	520	-	0.2009
3.3677	522	-	0.2005
3.3806	524	-	0.2005
3.3935	526	-	0.2001
3.4065	528	-	0.1999
3.4194	530	-	0.1999
3.4323	532	-	0.1996
3.4452	534	-	0.1999
3.4581	536	-	0.2007
3.4710	538	-	0.2009
3.4839	540	-	0.2013
3.4968	542	-	0.2012
3.5097	544	-	0.2014
3.5226	546	-	0.2017
3.5355	548	-	0.2022
3.5484	550	-	0.2029
3.5613	552	-	0.2029

Framework Versions

• Python: 3.12.2
• Sentence Transformers: 3.4.1
• Transformers: 4.50.0
• PyTorch: 2.6.0
• Accelerate: 1.5.2
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}