|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
tags: |
|
|
- clip |
|
|
- multimodal |
|
|
- contrastive-learning |
|
|
- cultural-heritage |
|
|
- reevaluate |
|
|
- information-retrieval |
|
|
datasets: |
|
|
- xuemduan/reevaluate-image-text-pairs |
|
|
model-index: |
|
|
- name: REEVALUATE CLIP Fine-tuned Models |
|
|
results: |
|
|
- task: |
|
|
type: image-text-retrieval |
|
|
name: Image-Text Retrieval |
|
|
dataset: |
|
|
name: Cultural Heritage Hybrid Dataset |
|
|
type: xuemduan/reevaluate-image-text-pairs |
|
|
metrics: |
|
|
- name: I2T R@1 |
|
|
type: recall@1 |
|
|
value: <TOBE_FILL_IN> |
|
|
- name: I2T R@5 |
|
|
type: recall@5 |
|
|
value: <TOBE_FILL_IN> |
|
|
- name: T2I R@1 |
|
|
type: recall@1 |
|
|
value: <TOBE_FILL_IN> |
|
|
--- |
|
|
|
|
|
|
|
|
# Domain-Adaptive CLIP for Multimodal Retrieval |
|
|
|
|
|
The fine-tuned CLIP (Vit-L/14) used in **Knowledge-Enhanced Multimodal Retrieval** |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
## 📦 Available Models |
|
|
|
|
|
| Model | Description | Data Type | |
|
|
|--------|--------------|-----------| |
|
|
| `reevaluate-clip` | Fine-tuned on images, query texts, and description texts | Image+Text | |
|
|
--- |
|
|
|
|
|
## 🧾 Dataset |
|
|
|
|
|
The models were trained and evaluated on the **REEVLAUATE Image-Text Pair Dataset**, which contains **43,500 image–text pairs** derived from Wikidata and Pilot Museums. |
|
|
|
|
|
Each artefact is described by: |
|
|
- `Image`: artefact image |
|
|
- `Description text`: BLIP-generated natural language portion + meatadata portion |
|
|
- `Query text`: User query-like text |
|
|
|
|
|
Dataset: [xuemduan/reevaluate-image-text-pairs](https://huggingface.co/datasets/xuemduan/reevaluate-image-text-pairs) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 Usage |
|
|
|
|
|
```python |
|
|
from transformers import CLIPProcessor, CLIPModel |
|
|
from PIL import Image |
|
|
|
|
|
model = CLIPModel.from_pretrained("xuemduan/reevaluate-clip") |
|
|
processor = CLIPProcessor.from_pretrained("xuemduan/reevaluate-clip") |
|
|
|
|
|
image = Image.open("artefact.jpg") |
|
|
text = "yellow flower paintings" |
|
|
|
|
|
image_embeds = model.get_image_features(**processor(images=image, return_tensors="pt")) |
|
|
text_embeds = model.get_text_features(**processor(text=[text], return_tensors="pt")) |
|
|
|
|
|
# normalize |
|
|
image_embeds = image_embeds / image_embeds.norm(dim=-1, keepdim=True) |
|
|
text_embeds = text_embeds / text_embeds.norm(dim=-1, keepdim=True) |
|
|
|
|
|
similarity = (image_embeds @ text_embeds.T) |
|
|
print(similarity) |
|
|
|
