HH-BTRewardModel-roberta

Bradley-Terry reward model trained on Anthropic/hh-rlhf dataset, using log-sigmoid loss:

$$\mathcal{L}=-\log \sigma (r_{\text{chosen}}-r_{\text{rejected}})=\log (1+\exp (-(r_{\text{chosen}}-r_{\text{rejected}})))\backslash mathcal\{L\}\; =\; -\backslash log\; \backslash sigma(r\_\{\backslash text\{chosen\}\}\; -\; r\_\{\backslash text\{rejected\}\})\; =\; \backslash log(1\; +\; \backslash exp(-(r\_\{\backslash text\{chosen\}\}\; -\; r\_\{\backslash text\{rejected\}\})))$$

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("JERRYPAN617/HH-BTRewardModel-roberta")
model = AutoModelForSequenceClassification.from_pretrained("JERRYPAN617/HH-BTRewardModel-roberta")

text = "Your text here"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    reward = model(**inputs).logits.item()