SentenceTransformer based on Shuu12121/CodeModernBERT-Crow-v1.1

This is a sentence-transformers model finetuned from Shuu12121/CodeModernBERT-Crow-v1.1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Shuu12121/CodeModernBERT-Crow-v1.1
Maximum Sequence Length: 1024 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'The toggle content, if left empty it will render the default toggle (seen above).',
    "function NavbarToggle() {\n\t    (0, _classCallCheck3['default'])(this, NavbarToggle);\n\t    return (0, _possibleConstructorReturn3['default'])(this, _React$Component.apply(this, arguments));\n\t  }",
    "update = () => {\n\n\t            const timerId = window.requestAnimationFrame( update );\n\t            const elapsed = performance.now() - timestamp;\n\t            const progress = elapsed / duration;\n\t            const opacity = 1.0 - progress > 0 ? 1.0 - progress : 0;\n\t            const radius = progress * canvasWidth * 0.5 / dpr;\n\n\t            context.clearRect( 0, 0, canvasWidth, canvasHeight );\n\t            context.beginPath();\n\t            context.arc( x, y, radius, 0, Math.PI * 2 );\n\t            context.fillStyle = `rgba(${color.r * 255}, ${color.g * 255}, ${color.b * 255}, ${opacity})`;\n\t            context.fill();\n\t            context.closePath();\n\n\t            if ( progress >= 1.0 ) {\n\n\t                window.cancelAnimationFrame( timerId );\n\t                this.updateCanvasArcByProgress( 0 );\n\n\t                /**\n\t                 * Reticle ripple end event\n\t                 * @type {object}\n\t                 * @event Reticle#reticle-ripple-end\n\t                 */\n\t                this.dispatchEvent( { type: 'reticle-ripple-end' } );\n\n\t            }\n\n\t            material.map.needsUpdate = true;\n\n\t        }",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6778, -0.0447],
#         [ 0.6778,  1.0000,  0.0303],
#         [-0.0447,  0.0303,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 2,392,064 training samples
Columns: sentence_0, sentence_1, and label
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label
type string string float
details
min: 8 tokens
mean: 74.35 tokens
max: 1024 tokens

min: 11 tokens
mean: 182.37 tokens
max: 1024 tokens

min: 1.0
mean: 1.0
max: 1.0

	sentence_0	sentence_1	label
type	string	string	float
details	min: 8 tokens mean: 74.35 tokens max: 1024 tokens	min: 11 tokens mean: 182.37 tokens max: 1024 tokens	min: 1.0 mean: 1.0 max: 1.0

Samples:

sentence_0	sentence_1	label
`Set the column title @param column - column number (first column is: 0) @param title - new column title`	setHeader = function(column, newValue) { const obj = this; if (obj.headers[column]) { const oldValue = obj.headers[column].textContent; const onchangeheaderOldValue = (obj.options.columns && obj.options.columns[column] && obj.options.columns[column].title) \|\| ''; if (! newValue) { newValue = getColumnName(column); } obj.headers[column].textContent = newValue; // Keep the title property obj.headers[column].setAttribute('title', newValue); // Update title if (!obj.options.columns) { obj.options.columns = []; } if (!obj.options.columns[column]) { obj.options.columns[column] = {}; } obj.options.columns[column].title = newValue; setHistory.call(obj, { action: 'setHeader', column: column, oldValue: oldValue, newValue: newValue }); // On onchange header dispatch.c...	`1.0`
Elsewhere this is known as a "Weak Value Map". Whereas a std JS WeakMap is weak on its keys, this map is weak on its values. It does not retain these values strongly. If a given value disappears, then the entries for it disappear from every weak-value-map that holds it as a value. Just as a WeakMap only allows gc-able values as keys, a weak-value-map only allows gc-able values as values. Unlike a WeakMap, a weak-value-map unavoidably exposes the non-determinism of gc to its clients. Thus, both the ability to create one, as well as each created one, must be treated as dangerous capabilities that must be closely held. A program with access to these can read side channels though gc that do not* rely on the ability to measure duration. This is a separate, and bad, timing-independent side channel. This non-determinism also enables code to escape deterministic replay. In a blockchain context, this could cause validators to differ from each other, preventing consensus, and thus preventing ...	makeFinalizingMap = (finalizer, opts) => { const { weakValues = false } = opts \|\| {}; if (!weakValues \|\| !WeakRef \|\| !FinalizationRegistry) { /** @type Map / const keyToVal = new Map(); return Far('fakeFinalizingMap', { clearWithoutFinalizing: keyToVal.clear.bind(keyToVal), get: keyToVal.get.bind(keyToVal), has: keyToVal.has.bind(keyToVal), set: (key, val) => { keyToVal.set(key, val); }, delete: keyToVal.delete.bind(keyToVal), getSize: () => keyToVal.size, }); } /* @type Map> */ const keyToRef = new Map(); const registry = new FinalizationRegistry(key => { // Because this will delete the current binding of key, we need to // be sure that it is not called because a previous binding was collected. // We do this with the unregister in set below, assuming that // unregister immediately suppresses the finalization of the thing // it unregisters. TODO If this is...	`1.0`
Creates a function that memoizes the result of func. If resolver is provided, it determines the cache key for storing the result based on the arguments provided to the memoized function. By default, the first argument provided to the memoized function is used as the map cache key. The func is invoked with the this binding of the memoized function. Note: The cache is exposed as the cache property on the memoized function. Its creation may be customized by replacing the _.memoize.Cache constructor with one whose instances implement the Map method interface of delete, get, has, and set. @static @memberOf _ @since 0.1.0 @category Function @param {Function} func The function to have its output memoized. @param {Function} [resolver] The function to resolve the cache key. @returns {Function} Returns the new memoized function. @example var object = { 'a': 1, 'b': 2 }; var othe...	`function memoize(func, resolver) { if (typeof func != 'function' \|\| (resolver && typeof resolver != 'function')) { throw new TypeError(FUNC_ERROR_TEXT); } var memoized = function() { var args = arguments, key = resolver ? resolver.apply(this, args) : args[0], cache = memoized.cache; if (cache.has(key)) { return cache.get(key); } var result = func.apply(this, args); memoized.cache = cache.set(key, result); return result; }; memoized.cache = new (memoize.Cache \|\| MapCache); return memoized; }`	`1.0`

Loss: CachedMultipleNegativesSymmetricRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 128,
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 2048
per_device_eval_batch_size: 2048
fp16: True
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 2048
per_device_eval_batch_size: 2048
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.4281	500	0.3784
0.8562	1000	0.1367
1.2842	1500	0.0707
1.7123	2000	0.0456
2.1404	2500	0.0344
2.5685	3000	0.0143
2.9966	3500	0.0136

Framework Versions

Python: 3.10.12
Sentence Transformers: 5.1.0
Transformers: 4.55.3
PyTorch: 2.7.0+cu128
Accelerate: 1.7.0
Datasets: 3.6.0
Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}