|
|
--- |
|
|
base_model: minishlab/potion-base-2m |
|
|
datasets: |
|
|
- Intel/polite-guard |
|
|
library_name: model2vec |
|
|
license: mit |
|
|
model_name: enguard/tiny-guard-2m-en-general-politeness-binary-intel |
|
|
tags: |
|
|
- static-embeddings |
|
|
- text-classification |
|
|
- model2vec |
|
|
--- |
|
|
|
|
|
# enguard/tiny-guard-2m-en-general-politeness-binary-intel |
|
|
|
|
|
This model is a fine-tuned Model2Vec classifier based on [minishlab/potion-base-2m](https://huggingface.co/minishlab/potion-base-2m) for the general-politeness-binary found in the [Intel/polite-guard](https://huggingface.co/datasets/Intel/polite-guard) dataset. |
|
|
|
|
|
|
|
|
|
|
|
## Installation |
|
|
|
|
|
```bash |
|
|
pip install model2vec[inference] |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from model2vec.inference import StaticModelPipeline |
|
|
|
|
|
model = StaticModelPipeline.from_pretrained( |
|
|
"enguard/tiny-guard-2m-en-general-politeness-binary-intel" |
|
|
) |
|
|
|
|
|
|
|
|
# Supports single texts. Format input as a single text: |
|
|
text = "Example sentence" |
|
|
|
|
|
model.predict([text]) |
|
|
model.predict_proba([text]) |
|
|
|
|
|
``` |
|
|
|
|
|
## Why should you use these models? |
|
|
|
|
|
- Optimized for precision to reduce false positives. |
|
|
- Extremely fast inference: up to x500 faster than SetFit. |
|
|
|
|
|
## This model variant |
|
|
|
|
|
Below is a quick overview of the model variant and core metrics. |
|
|
|
|
|
| Field | Value | |
|
|
|---|---| |
|
|
| Classifies | general-politeness-binary | |
|
|
| Base Model | [minishlab/potion-base-2m](https://huggingface.co/minishlab/potion-base-2m) | |
|
|
| Precision | 0.9843 | |
|
|
| Recall | 0.9889 | |
|
|
| F1 | 0.9866 | |
|
|
|
|
|
### Confusion Matrix |
|
|
|
|
|
| True \ Predicted | FAIL | PASS | |
|
|
| --- | --- | --- | |
|
|
| **FAIL** | 2504 | 28 | |
|
|
| **PASS** | 40 | 7628 | |
|
|
|
|
|
<details> |
|
|
<summary><b>Full metrics (JSON)</b></summary> |
|
|
|
|
|
```json |
|
|
{ |
|
|
"FAIL": { |
|
|
"precision": 0.9842767295597484, |
|
|
"recall": 0.9889415481832543, |
|
|
"f1-score": 0.9866036249014972, |
|
|
"support": 2532.0 |
|
|
}, |
|
|
"PASS": { |
|
|
"precision": 0.9963427377220481, |
|
|
"recall": 0.9947835159102765, |
|
|
"f1-score": 0.9955625163142783, |
|
|
"support": 7668.0 |
|
|
}, |
|
|
"accuracy": 0.9933333333333333, |
|
|
"macro avg": { |
|
|
"precision": 0.9903097336408982, |
|
|
"recall": 0.9918625320467653, |
|
|
"f1-score": 0.9910830706078877, |
|
|
"support": 10200.0 |
|
|
}, |
|
|
"weighted avg": { |
|
|
"precision": 0.9933475286370538, |
|
|
"recall": 0.9933333333333333, |
|
|
"f1-score": 0.9933386032694584, |
|
|
"support": 10200.0 |
|
|
} |
|
|
} |
|
|
``` |
|
|
</details> |
|
|
|
|
|
|
|
|
<details> |
|
|
<summary><b>Sample Predictions</b></summary> |
|
|
|
|
|
| Text | True Label | Predicted Label | |
|
|
|------|------------|-----------------| |
|
|
| I appreciate your interest in our vegetarian options. I can provide you with a list of our current dishes that cater to your dietary preferences. | PASS | PASS | |
|
|
| I understand you're concerned about the ski lessons, and I'll look into the options for rescheduling. | PASS | PASS | |
|
|
| Our technical skills course will cover the essential topics in data analysis, including data visualization and statistical modeling. The course materials will be available on our learning platform. | PASS | PASS | |
|
|
| Our buffet hours are from 11 AM to 9 PM. Please note that we have a limited selection of options available during the lunch break. | PASS | PASS | |
|
|
| I'll look into your policy details and see what options are available to you. | PASS | PASS | |
|
|
| I appreciate your interest in our vegetarian options. I can provide you with a list of our current dishes that cater to your dietary preferences. | PASS | PASS | |
|
|
</details> |
|
|
|
|
|
|
|
|
<details> |
|
|
<summary><b>Prediction Speed Benchmarks</b></summary> |
|
|
|
|
|
| Dataset Size | Time (seconds) | Predictions/Second | |
|
|
|--------------|----------------|---------------------| |
|
|
| 1 | 0.0002 | 5108.77 | |
|
|
| 1000 | 0.0542 | 18439.74 | |
|
|
| 10000 | 0.6208 | 16108.79 | |
|
|
</details> |
|
|
|
|
|
|
|
|
## Other model variants |
|
|
|
|
|
Below is a general overview of the best-performing models for each dataset variant. |
|
|
|
|
|
| Classifies | Model | Precision | Recall | F1 | |
|
|
| --- | --- | --- | --- | --- | |
|
|
| general-politeness-binary | [enguard/tiny-guard-2m-en-general-politeness-binary-intel](https://huggingface.co/enguard/tiny-guard-2m-en-general-politeness-binary-intel) | 0.9843 | 0.9889 | 0.9866 | |
|
|
| general-politeness-multiclass | [enguard/tiny-guard-2m-en-general-politeness-multiclass-intel](https://huggingface.co/enguard/tiny-guard-2m-en-general-politeness-multiclass-intel) | 0.9875 | 0.9704 | 0.9789 | |
|
|
| general-politeness-binary | [enguard/tiny-guard-4m-en-general-politeness-binary-intel](https://huggingface.co/enguard/tiny-guard-4m-en-general-politeness-binary-intel) | 0.9831 | 0.9878 | 0.9854 | |
|
|
| general-politeness-multiclass | [enguard/tiny-guard-4m-en-general-politeness-multiclass-intel](https://huggingface.co/enguard/tiny-guard-4m-en-general-politeness-multiclass-intel) | 0.9896 | 0.9783 | 0.9839 | |
|
|
| general-politeness-binary | [enguard/tiny-guard-8m-en-general-politeness-binary-intel](https://huggingface.co/enguard/tiny-guard-8m-en-general-politeness-binary-intel) | 0.9828 | 0.9905 | 0.9866 | |
|
|
| general-politeness-multiclass | [enguard/tiny-guard-8m-en-general-politeness-multiclass-intel](https://huggingface.co/enguard/tiny-guard-8m-en-general-politeness-multiclass-intel) | 0.9873 | 0.9795 | 0.9833 | |
|
|
| general-politeness-binary | [enguard/small-guard-32m-en-general-politeness-binary-intel](https://huggingface.co/enguard/small-guard-32m-en-general-politeness-binary-intel) | 0.9858 | 0.9889 | 0.9874 | |
|
|
| general-politeness-multiclass | [enguard/small-guard-32m-en-general-politeness-multiclass-intel](https://huggingface.co/enguard/small-guard-32m-en-general-politeness-multiclass-intel) | 0.9897 | 0.9862 | 0.9879 | |
|
|
| general-politeness-binary | [enguard/medium-guard-128m-xx-general-politeness-binary-intel](https://huggingface.co/enguard/medium-guard-128m-xx-general-politeness-binary-intel) | 0.9831 | 0.9901 | 0.9866 | |
|
|
| general-politeness-multiclass | [enguard/medium-guard-128m-xx-general-politeness-multiclass-intel](https://huggingface.co/enguard/medium-guard-128m-xx-general-politeness-multiclass-intel) | 0.9881 | 0.9870 | 0.9876 | |
|
|
|
|
|
## Resources |
|
|
|
|
|
- Awesome AI Guardrails: <https://github.com/enguard-ai/awesome-ai-guardails> |
|
|
- Model2Vec: https://github.com/MinishLab/model2vec |
|
|
- Docs: https://minish.ai/packages/model2vec/introduction |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model, please cite Model2Vec: |
|
|
|
|
|
``` |
|
|
@software{minishlab2024model2vec, |
|
|
author = {Stephan Tulkens and {van Dongen}, Thomas}, |
|
|
title = {Model2Vec: Fast State-of-the-Art Static Embeddings}, |
|
|
year = {2024}, |
|
|
publisher = {Zenodo}, |
|
|
doi = {10.5281/zenodo.17270888}, |
|
|
url = {https://github.com/MinishLab/model2vec}, |
|
|
license = {MIT} |
|
|
} |
|
|
``` |
