magnifi/bge-small-en-v1-5-ft-orc_fold4

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'do I have direct holding of [TICKER]?',
    '[{"get_portfolio([\'marketValue\'],True,None)": "portfolio"}, {"get_attribute(\'portfolio\',[\'<TICKER>\'],\'<DATES>\')": "portfolio"}, {"calculate(\'portfolio\',[\'marketValue\', \'<TICKER>\'],\'multiply\',\'expo_<TICKER>\')": "portfolio"}, {"sort(\'portfolio\',\'expo_<TICKER>\',\'desc\')": "portfolio"}, {"aggregate(\'portfolio\',\'ticker\',\'expo_<TICKER>\',\'sum\',None)": "port_expo_<TICKER>"}]',
    '[{"newsletter_search(None,[\'<TICKER>\'],\'query\',None,False)": "newsletter_chunks"}, {"get_portfolio([\'weight\'],False,None)": "portfolio"}, {"get_attribute(\'portfolio\',[\'<TICKER>\'],None)": "portfolio"}, {"calculate(\'portfolio\',[\'weight\', \'<TICKER>\'],\'multiply\',\'expo_percentage_<TICKER>\')": "portfolio"}, {"sort(\'portfolio\',\'expo_percentage_<TICKER>\',\'desc\')": "portfolio"}, {"aggregate(\'portfolio\',\'ticker\',\'expo_percentage_<TICKER>\',\'sum\',None)": "total_expo_percentage_<TICKER>"} , {"analyze_impact(\'portfolio\',\'<TICKER>\',\'buy\')": "<TICKER>_buy_impact"}]',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8205
cosine_accuracy@3	0.953
cosine_accuracy@5	0.9786
cosine_accuracy@10	1.0
cosine_precision@1	0.8205
cosine_precision@3	0.3177
cosine_precision@5	0.1957
cosine_precision@10	0.1
cosine_recall@1	0.0228
cosine_recall@3	0.0265
cosine_recall@5	0.0272
cosine_recall@10	0.0278
cosine_ndcg@10	0.202
cosine_mrr@10	0.8905
cosine_map@100	0.0247

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,541 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 4 tokens mean: 12.87 tokens max: 28 tokens	min: 20 tokens mean: 75.47 tokens max: 269 tokens

• Samples:

  sentence_0	sentence_1
  [TICKER] vs peers based on [ATTRIBUTE]	[{"search('query', 'match_type', '')": "search_results"},{"compare([[''],'search_results'], [''], None)": "comparison_data"}]
  How will the rising UK stock index affect my portfolio?	[{"get_portfolio(None,True,None)": "portfolio"}, {"stress_test('portfolio','ftse_100',None,'up')": "stress_test"}]
  How is [A_REGION] performing [DATES]?	[{"get_attribute([''],['returns'],'')":"region_returns"}]

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 6
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand

Epoch	Step	Training Loss	cosine_ndcg@10
0.0129	2	-	0.1020
0.0258	4	-	0.1020
0.0387	6	-	0.1025
0.0516	8	-	0.1033
0.0645	10	-	0.1055
0.0774	12	-	0.1062
0.0903	14	-	0.1085
0.1032	16	-	0.1110
0.1161	18	-	0.1143
0.1290	20	-	0.1188
0.1419	22	-	0.1234
0.1548	24	-	0.1252
0.1677	26	-	0.1275
0.1806	28	-	0.1293
0.1935	30	-	0.1300
0.2065	32	-	0.1323
0.2194	34	-	0.1348
0.2323	36	-	0.1372
0.2452	38	-	0.1397
0.2581	40	-	0.1420
0.2710	42	-	0.1433
0.2839	44	-	0.1439
0.2968	46	-	0.1447
0.3097	48	-	0.1457
0.3226	50	-	0.1460
0.3355	52	-	0.1465
0.3484	54	-	0.1475
0.3613	56	-	0.1484
0.3742	58	-	0.1496
0.3871	60	-	0.1507
0.4	62	-	0.1517
0.4129	64	-	0.1537
0.4258	66	-	0.1543
0.4387	68	-	0.1553
0.4516	70	-	0.1554
0.4645	72	-	0.1555
0.4774	74	-	0.1559
0.4903	76	-	0.1575
0.5032	78	-	0.1589
0.5161	80	-	0.1589
0.5290	82	-	0.1605
0.5419	84	-	0.1614
0.5548	86	-	0.1625
0.5677	88	-	0.1624
0.5806	90	-	0.1640
0.5935	92	-	0.1656
0.6065	94	-	0.1662
0.6194	96	-	0.1676
0.6323	98	-	0.1678
0.6452	100	-	0.1688
0.6581	102	-	0.1689
0.6710	104	-	0.1696
0.6839	106	-	0.1712
0.6968	108	-	0.1708
0.7097	110	-	0.1709
0.7226	112	-	0.1723
0.7355	114	-	0.1732
0.7484	116	-	0.1735
0.7613	118	-	0.1752
0.7742	120	-	0.1764
0.7871	122	-	0.1762
0.8	124	-	0.1764
0.8129	126	-	0.1772
0.8258	128	-	0.1764
0.8387	130	-	0.1766
0.8516	132	-	0.1764
0.8645	134	-	0.1771
0.8774	136	-	0.1775
0.8903	138	-	0.1792
0.9032	140	-	0.1807
0.9161	142	-	0.1793
0.9290	144	-	0.1806
0.9419	146	-	0.1807
0.9548	148	-	0.1809
0.9677	150	-	0.1812
0.9806	152	-	0.1821
0.9935	154	-	0.1830
1.0	155	-	0.1831
1.0065	156	-	0.1838
1.0194	158	-	0.1842
1.0323	160	-	0.1845
1.0452	162	-	0.1844
1.0581	164	-	0.1859
1.0710	166	-	0.1862
1.0839	168	-	0.1858
1.0968	170	-	0.1859
1.1097	172	-	0.1849
1.1226	174	-	0.1859
1.1355	176	-	0.1865
1.1484	178	-	0.1856
1.1613	180	-	0.1856
1.1742	182	-	0.1847
1.1871	184	-	0.1845
1.2	186	-	0.1846
1.2129	188	-	0.1848
1.2258	190	-	0.1840
1.2387	192	-	0.1838
1.2516	194	-	0.1837
1.2645	196	-	0.1844
1.2774	198	-	0.1849
1.2903	200	-	0.1839
1.3032	202	-	0.1838
1.3161	204	-	0.1845
1.3290	206	-	0.1845
1.3419	208	-	0.1848
1.3548	210	-	0.1850
1.3677	212	-	0.1851
1.3806	214	-	0.1846
1.3935	216	-	0.1849
1.4065	218	-	0.1843
1.4194	220	-	0.1849
1.4323	222	-	0.1850
1.4452	224	-	0.1851
1.4581	226	-	0.1853
1.4710	228	-	0.1860
1.4839	230	-	0.1865
1.4968	232	-	0.1871
1.5097	234	-	0.1869
1.5226	236	-	0.1868
1.5355	238	-	0.1867
1.5484	240	-	0.1873
1.5613	242	-	0.1871
1.5742	244	-	0.1876
1.5871	246	-	0.1870
1.6	248	-	0.1884
1.6129	250	-	0.1888
1.6258	252	-	0.1891
1.6387	254	-	0.1887
1.6516	256	-	0.1890
1.6645	258	-	0.1887
1.6774	260	-	0.1884
1.6903	262	-	0.1884
1.7032	264	-	0.1873
1.7161	266	-	0.1881
1.7290	268	-	0.1884
1.7419	270	-	0.1888
1.7548	272	-	0.1888
1.7677	274	-	0.1891
1.7806	276	-	0.1897
1.7935	278	-	0.1910
1.8065	280	-	0.1914
1.8194	282	-	0.1915
1.8323	284	-	0.1914
1.8452	286	-	0.1912
1.8581	288	-	0.1914
1.8710	290	-	0.1905
1.8839	292	-	0.1919
1.8968	294	-	0.1923
1.9097	296	-	0.1922
1.9226	298	-	0.1919
1.9355	300	-	0.1916
1.9484	302	-	0.1916
1.9613	304	-	0.1919
1.9742	306	-	0.1921
1.9871	308	-	0.1928
2.0	310	-	0.1924
2.0129	312	-	0.1923
2.0258	314	-	0.1916
2.0387	316	-	0.1918
2.0516	318	-	0.1912
2.0645	320	-	0.1907
2.0774	322	-	0.1905
2.0903	324	-	0.1907
2.1032	326	-	0.1912
2.1161	328	-	0.1916
2.1290	330	-	0.1913
2.1419	332	-	0.1924
2.1548	334	-	0.1921
2.1677	336	-	0.1925
2.1806	338	-	0.1925
2.1935	340	-	0.1928
2.2065	342	-	0.1924
2.2194	344	-	0.1925
2.2323	346	-	0.1926
2.2452	348	-	0.1927
2.2581	350	-	0.1930
2.2710	352	-	0.1932
2.2839	354	-	0.1933
2.2968	356	-	0.1928
2.3097	358	-	0.1925
2.3226	360	-	0.1928
2.3355	362	-	0.1932
2.3484	364	-	0.1940
2.3613	366	-	0.1939
2.3742	368	-	0.1940
2.3871	370	-	0.1932
2.4	372	-	0.1934
2.4129	374	-	0.1938
2.4258	376	-	0.1939
2.4387	378	-	0.1936
2.4516	380	-	0.1938
2.4645	382	-	0.1934
2.4774	384	-	0.1935
2.4903	386	-	0.1940
2.5032	388	-	0.1947
2.5161	390	-	0.1948
2.5290	392	-	0.1949
2.5419	394	-	0.1953
2.5548	396	-	0.1956
2.5677	398	-	0.1960
2.5806	400	-	0.1960
2.5935	402	-	0.1961
2.6065	404	-	0.1960
2.6194	406	-	0.1961
2.6323	408	-	0.1961
2.6452	410	-	0.1961
2.6581	412	-	0.1975
2.6710	414	-	0.1975
2.6839	416	-	0.1978
2.6968	418	-	0.1971
2.7097	420	-	0.1972
2.7226	422	-	0.1978
2.7355	424	-	0.1974
2.7484	426	-	0.1974
2.7613	428	-	0.1969
2.7742	430	-	0.1973
2.7871	432	-	0.1960
2.8	434	-	0.1957
2.8129	436	-	0.1957
2.8258	438	-	0.1960
2.8387	440	-	0.1964
2.8516	442	-	0.1963
2.8645	444	-	0.1968
2.8774	446	-	0.1976
2.8903	448	-	0.1970
2.9032	450	-	0.1970
2.9161	452	-	0.1966
2.9290	454	-	0.1970
2.9419	456	-	0.1971
2.9548	458	-	0.1978
2.9677	460	-	0.1978
2.9806	462	-	0.1979
2.9935	464	-	0.1983
3.0	465	-	0.1979
3.0065	466	-	0.1979
3.0194	468	-	0.1983
3.0323	470	-	0.1987
3.0452	472	-	0.1988
3.0581	474	-	0.1995
3.0710	476	-	0.1997
3.0839	478	-	0.1997
3.0968	480	-	0.1997
3.1097	482	-	0.2001
3.1226	484	-	0.2002
3.1355	486	-	0.2002
3.1484	488	-	0.2005
3.1613	490	-	0.2000
3.1742	492	-	0.2004
3.1871	494	-	0.2004
3.2	496	-	0.2005
3.2129	498	-	0.2004
3.2258	500	0.3412	0.2005
3.2387	502	-	0.2006
3.2516	504	-	0.2005
3.2645	506	-	0.2006
3.2774	508	-	0.2007
3.2903	510	-	0.2007
3.3032	512	-	0.2011
3.3161	514	-	0.2011
3.3290	516	-	0.2008
3.3419	518	-	0.2008
3.3548	520	-	0.2010
3.3677	522	-	0.2013
3.3806	524	-	0.2011
3.3935	526	-	0.2008
3.4065	528	-	0.2003
3.4194	530	-	0.2003
3.4323	532	-	0.2004
3.4452	534	-	0.2002
3.4581	536	-	0.2000
3.4710	538	-	0.2004
3.4839	540	-	0.2007
3.4968	542	-	0.2006
3.5097	544	-	0.2003
3.5226	546	-	0.2003
3.5355	548	-	0.2003
3.5484	550	-	0.2004
3.5613	552	-	0.2000
3.5742	554	-	0.1999
3.5871	556	-	0.1995
3.6	558	-	0.1998
3.6129	560	-	0.1998
3.6258	562	-	0.1999
3.6387	564	-	0.2006
3.6516	566	-	0.2006
3.6645	568	-	0.2006
3.6774	570	-	0.2003
3.6903	572	-	0.2003
3.7032	574	-	0.2001
3.7161	576	-	0.2005
3.7290	578	-	0.2005
3.7419	580	-	0.2005
3.7548	582	-	0.2005
3.7677	584	-	0.2001
3.7806	586	-	0.2005
3.7935	588	-	0.2006
3.8065	590	-	0.2002
3.8194	592	-	0.1999
3.8323	594	-	0.1999
3.8452	596	-	0.1996
3.8581	598	-	0.1996
3.8710	600	-	0.1996
3.8839	602	-	0.1997
3.8968	604	-	0.1996
3.9097	606	-	0.2000
3.9226	608	-	0.2001
3.9355	610	-	0.2005
3.9484	612	-	0.2002
3.9613	614	-	0.2003
3.9742	616	-	0.2006
3.9871	618	-	0.2006
4.0	620	-	0.2009
4.0129	622	-	0.2012
4.0258	624	-	0.2012
4.0387	626	-	0.2010
4.0516	628	-	0.2010
4.0645	630	-	0.2011
4.0774	632	-	0.2011
4.0903	634	-	0.2011
4.1032	636	-	0.2010
4.1161	638	-	0.2013
4.1290	640	-	0.2013
4.1419	642	-	0.2013
4.1548	644	-	0.2009
4.1677	646	-	0.2009
4.1806	648	-	0.2009
4.1935	650	-	0.2005
4.2065	652	-	0.2001
4.2194	654	-	0.2001
4.2323	656	-	0.2001
4.2452	658	-	0.2001
4.2581	660	-	0.1999
4.2710	662	-	0.1999
4.2839	664	-	0.1999
4.2968	666	-	0.1999
4.3097	668	-	0.2003
4.3226	670	-	0.2006
4.3355	672	-	0.2006
4.3484	674	-	0.2003
4.3613	676	-	0.2004
4.3742	678	-	0.2005
4.3871	680	-	0.2009
4.4	682	-	0.2011
4.4129	684	-	0.2011
4.4258	686	-	0.2014
4.4387	688	-	0.2014
4.4516	690	-	0.2011
4.4645	692	-	0.2009
4.4774	694	-	0.2012
4.4903	696	-	0.2012
4.5032	698	-	0.2013
4.5161	700	-	0.2010
4.5290	702	-	0.2010
4.5419	704	-	0.2010
4.5548	706	-	0.2014
4.5677	708	-	0.2014
4.5806	710	-	0.2014
4.5935	712	-	0.2014
4.6065	714	-	0.2017
4.6194	716	-	0.2016
4.6323	718	-	0.2018
4.6452	720	-	0.2017
4.6581	722	-	0.2016
4.6710	724	-	0.2017
4.6839	726	-	0.2017
4.6968	728	-	0.2019
4.7097	730	-	0.2019
4.7226	732	-	0.2019
4.7355	734	-	0.2019
4.7484	736	-	0.2019
4.7613	738	-	0.2019
4.7742	740	-	0.2019
4.7871	742	-	0.2019
4.8	744	-	0.2020

Framework Versions

• Python: 3.12.2
• Sentence Transformers: 3.4.1
• Transformers: 4.50.0
• PyTorch: 2.6.0
• Accelerate: 1.5.2
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}