---
library_name: transformers
tags:
- nli
- bert
- natural-language-inference
language:
- ru
metrics:
- accuracy
- f1
- precision
- recall
base_model:
- cointegrated/rubert-tiny2
pipeline_tag: text-classification
model-index:
- name: rubert-tiny-nli-terra-v0
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: TERRA
      type: NLI
      split: validation
    metrics:
    - type: accuracy
      value: 0.6710097719869706
      name: Accuracy
    - type: f1
      value: 0.6576271186440678
      name: F1
    - type: precision
      value: 0.6830985915492958
      name: Precision
    - type: recall
      value: 0.6339869281045751
      name: Recall
---

**⚠️ Disclaimer: This model is in the early stages of development and may produce low-quality predictions. For better results, consider using the recommended Russian natural language inference models available [here](https://huggingface.co/cointegrated).**

# RuBERT-tiny-nli v1

This model is a second attempt to fine-tune the [RuBERT-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) model for a two-way natural language inference task, utilizing the Russian [Textual Entailment Recognition](https://russiansuperglue.com/tasks/task_info/TERRa) dataset. This model uses a custom classifier head with two dense layers. The performance is currently limited.


## Usage
How to run the model for NLI:

```python
# !pip install transformers sentencepiece --quiet
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = 'Marwolaeth/rubert-tiny-nli-terra-v0'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
if torch.cuda.is_available():
    model.cuda()

# An example from the base model card
premise1 = 'Сократ - человек, а все люди смертны.'
hypothesis1 = 'Сократ никогда не умрёт.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise1, hypothesis1, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.61647415, 'entailment': 0.38352585}

# An example concerning sentiments
premise2 = 'Я ненавижу желтые занавески'
hypothesis2 = 'Мне нравятся желтые занавески'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise2, hypothesis2, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.520086, 'entailment': 0.47991407}
```

## Model Performance Metrics

The following metrics summarize the performance of the model on the test dataset:

| Metric                           | Value                     |
|----------------------------------|---------------------------|
| **Validation Loss**              | 0.6546                    |
| **Validation Accuracy**          | 67.10%                    |
| **Validation F1 Score**          | 65.76%                    |
| **Validation Precision**         | 68.31%                    |
| **Validation Recall**            | 63.40%                    |
| **Validation Runtime***          | 0.2546 seconds            |
| **Samples per Second***          | 1 205.80                    |
| **Steps per Second***            | 7.86                     |

*Using T4 GPU with Google Colab