README.md

---
language:
- en
license: open-mdw
model_name: ActIO-UI-7B-SFT
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
pipeline_tag: image-text-to-text
tags:
- Multimodal
- VLM
- Computer-Use-Agent
- Web-Agent
- GUI
- Grounding
- GUI Subtask
library_name: transformers
---

<h1 style="
  font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Helvetica,Arial,sans-serif;
  font-size:48px;
  font-weight:700;
  line-height:1.25;
  text-align:center;
  margin:0 0 24px;">
  ActIO-UI: Vision-Language Models for GUI Subtasks
</h1>

<div style="
  display:flex;
  justify-content:center;
  gap:12px;
  flex-wrap:wrap;
  margin-bottom:28px;">
  
  <a href="?????(homepage)" style="
     display:inline-block;
     padding:8px 24px;
     background:#2b2b2b;
     color:#ffffff;
     border-radius:36px;
     text-decoration:none;
     font-weight:600;
     font-size:16px;">
    🌐 Website (Coming Soon!)
  </a>
  <a href="https://arxiv.org/abs/2510.09872" style="
     display:inline-block;
     padding:8px 24px;
     background:#2b2b2b;
     color:#ffffff;
     border-radius:36px;
     text-decoration:none;
     font-weight:600;
     font-size:16px;">
    📝 Paper
  </a>
  <a href="?????(repository)" style="
     display:inline-block;
     padding:8px 24px;
     background:#2b2b2b;
     color:#ffffff;
     border-radius:36px;
     text-decoration:none;
     font-weight:600;
     font-size:16px;">
    💻 Code (Coming Soon!)
  </a>
</div>



# Introduction

ActIO-UI-7B-SFT is a model trained for web GUI subtask execution.
We introduce it in the paper [WARC-Bench: Web Archive based Benchmark for GUI Subtask Executions](https://arxiv.org/abs/2510.09872).
Its base model is [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct).
We define a GUI subtask as a short-horizon web interaction task that correspond to simple natural language instructions within a larger web browsing workflow.
Some examples of subtasks include choosing the correct date in a date picker or scrolling in a container to extract information.
ActIO-UI-7B-SFT achieves state-of-the-art (SOTA) results for open-source models on WARC-Bench, our GUI subtask benchmark.
It also exhibits improved long-horizon GUI task and grounding capabilities when compared with its base model. We provide a performance summary of the ActIO-UI model family below.

<div align="center">

| **Model**             | **WARC-Bench (test split)** | **WebArena (no map)** | **Miniwob++** | **ScreenSpot V2** |
|------------------------|------------------------:|----------------------:|--------------:|------------------:|
| ActIO-UI-7B-SFT | 27.33 % | 13.40 % | 29.60 % | 80.03 % |
| ActIO-UI-7B-RLVR | 29.17 % | 7.31 % | 36.27 % | 75.81 % |
| ActIO-UI-72B-SFT | 48.33 % | 23.05 % | 45.87 % | 84.12 % |
| ActIO-UI-72B-RLVR | 52.33 % | 26.80 % | 59.20 % | 82.44 % |

</div>

ActIO-UI is developed by [Orby AI](https://www.orby.ai/), a [Uniphore](https://www.uniphore.com/) company. Please direct all correspondence to [Sanjari Srivastava](mailto:[email protected]) and [Peng Qi](mailto:[email protected]) (first_name.last_name at uniphore.com).



# Model Family

- [ActIO-UI-7B-SFT](https://huggingface.co/Uniphore/actio-ui-7b-sft): a 7B model trained with supervised finetuning (SFT) using distilled subtask data.
- [ActIO-UI-7B-RLVR](?????(model_link)): a 7B model trained with Reinforcement Learning with Verifiable Rewards (RLVR) over the ActIO-UI-7B-SFT checkpoint.
- [ActIO-UI-72B-SFT](?????(model_link)): a 72B model trained with SFT using distilled subtask data.
- [ActIO-UI-72B-RLVR](?????(model_link)): a 72B model trained with RLVR over the ActIO-UI-72B-SFT checkpoint.



# Performance

## GUI Subtask (WARC-Bench)

ActIO-UI models are specifically trained to solve GUI subtask problems. Both the 7B and 72B ActIO-UI RLVR models achieves SOTA performance for their respective sub-categories of 7B or 72B open-source models.

<div align="center">

| **Model** | **Dev [synthetic]** | **Dev [real]** | **Dev [total]** | **Test** |
|---|---:|---:|---:|---:|
| _Closed-source_ |  |  |  |  |
| OpenAI computer-use-preview *(2025-03-11)*<sup>CUA</sup> | 62.17 | 49.44 | 58.96 | 33.83 |
| GPT-4o *(2024-11-20)* | 7.87 | 14.51 | 9.54 | 9.17 |
| GPT-5 *(2025-08-07)* | 72.66 | 61.67 | 69.89 | 51.33 |
| Claude Sonnet 4.0 *(2025-05-14)*<sup>CUA</sup> | 79.92 | 76.11 | 78.96 | 47.17 |
| Claude Sonnet 3.7 *(2025-02-19)* | 82.96 | 78.89 | 81.93 | 59.83 |
| Claude Sonnet 4.0 *(2025-05-14)* | <u>84.27</u> | <u>**81.67**</u> | <u>83.61</u> | <u>**64.83**</u> |
| _Open-source — Small_ |  |  |  |  |
| Qwen2.5-VL 7B | 16.85 | 11.67 | 15.54 | 4.67 |
| UI-Tars 1.5 7B<sup>CUA</sup> | 44.01 | 26.55 | 39.66 | 10.33 |
| OpenCUA 7B<sup>CUA*</sup> | 48.03 | 41.67 | 46.43 | 14.00 |
| **ActIO-UI-7B-SFT** | 70.60 | <u>54.49</u> | 66.54 | 27.33 |
| **ActIO-UI-7B-RLVR** | <u>78.09</u> | 54.44 | <u>72.13</u> | <u>29.17</u> |
| _Open-source — Large_ |  |  |  |  |
| OpenCUA 32B<sup>CUA*</sup> | 51.12 | 41.67 | 48.74 | 17.50 |
| Qwen2.5-VL 72B | 64.23 | 51.67 | 61.06 | 37.33 |
| **ActIO-UI-72B-SFT** | 78.23 | 68.89 | 75.88 | 48.33 |
| **ActIO-UI-72B-RLVR** | <u>**87.64**</u> | <u>78.33</u> | <u>**84.31**</u> | <u>52.33</u> |

<sub>Trajectory-level success rates on WARC-Bench. Small VLMs (7B params) are grouped under “Open-source — Small”. Results are divided into closed (top) vs. open-source (bottom). CUA = evaluated with provider’s computer-use agent; others use our Subtask Vision Agent (SVA) design. Rows marked with <sup>*</sup> denote averages across fewer than 3 runs; all others are averages of 3 runs. <strong>Bold</strong> = best per benchmark. <u>Underline</u> = best inside its sector (closed; open-source small; open-source large).</sub>

</div>


## Other Benchmarks

To access generalizability of GUI subtask execution as a model capability, we compare the performance of ActIO-UI over GUI subtasks (WARC-Bench), long-horizon tasks (WebArena), short-horizon tasks (Miniwob++), and GUI visual grounding (ScreenSpot V2). Without access to any long-horizon and grounding data in its training dataset, our models show improved performance over their base models (except for the grounding performance when compared to Qwen 2.5 VL 72B).

<div align="center">

| **Model**             | **WARC-Bench (test split)** | **WebArena (no map)** | **Miniwob++** | **ScreenSpot V2** |
|------------------------|------------------------:|----------------------:|--------------:|------------------:|
| _Public Models_ |  |  |  |  |
| Qwen 2.5 VL 7B | 4.67 % | 3.07 % | 12.53 % | 51.62 % |
| Qwen 2.5 VL 72B | 37.33 % | 15.68 % | 53.87 % | **88.05 %** |
| GPT-5 *(2025-08-07)* | 51.33 % | 34.06 % | 52.27 % | 26.39 % |
| Claude 4 Sonnet | **64.83 %** | **37.96 %** | **71.73 %** | 85.06 % |
| _Ours_ |  |  |  |  |
| **ActIO-UI-7B-SFT** | 27.33 % | 13.40 % | 29.60 % | 80.03 % |
| **ActIO-UI-7B-RLVR** | 29.17 % | 7.31 % | 36.27 % | 75.81 % |
| **ActIO-UI-72B-SFT** | 48.33 % | 23.05 % | 45.87 % | 84.12 % |
| **ActIO-UI-72B-RLVR** | 52.33 % | 26.80 % | 59.20 % | 82.44 % |

<sub>Each number is an average of 3 runs. **Bold** = best per benchmark. We omit the OpenStreetMap portion of the WebArena benchmark due to technical difficulties with the online map service. For Screenspot V2, we limit the agent action space to \texttt{click(...)} only. For each evaluation, we use the same prompt format built inside our Subtask Vision Agent</sub>

</div>



# Usage

## Image Input Size

To maintain optimal model performance, each input image should be set at **1280 (pixel width) \\(\times\\) 720 (pixel height)**.


## Setup

To run the code snippets below, we recommend that you install everything in `requirements.txt` in a python environment.
```bash
python -m venv ./venv
source venv/bin/activate
pip install -r requirements.txt
```


## Sanity test

Note that this is only a sanity test for ensuring model is working properly.
For replicating the evaluation result or using the model for your own project, please refer to our code repository on [GitHub](?????(repository)).

The following code snippet is also available in the attached sanity.py. So you can just run it with
```{bash}
python sanity.py
```

```{python}
import base64
import sys
import torch
from transformers import AutoTokenizer, AutoModelForVision2Seq, AutoProcessor
from PIL import Image


def encode_image(image_path: str) -> str:
    """Encode image to base64 string for model input."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode()


def load_model(
    model_path: str,
) -> tuple[AutoModelForVision2Seq, AutoTokenizer, AutoProcessor]:
    """Load OpenCUA model, tokenizer, and image processor."""
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    model = AutoModelForVision2Seq.from_pretrained(
        model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
    )
    image_processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

    return model, tokenizer, image_processor


def create_grounding_messages(image_path: str, instruction: str) -> list[dict]:
    """Create chat messages for GUI grounding task."""
    system_prompt = (
        "You are a GUI agent. You are given a task and a screenshot of the screen. "
        "You need to perform a series of pyautogui actions to complete the task."
    )

    messages = [
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Please perform the following task by providing the action and the coordinates: "
                    + instruction,
                },
                {
                    "type": "image",
                    "image": f"data:image/png;base64,{encode_image(image_path)}",
                },
            ],
        },
    ]
    return messages


def run_inference(
    model: AutoModelForVision2Seq,
    tokenizer: AutoTokenizer,
    image_processor: AutoProcessor,
    messages: list[dict],
    image_path: str,
) -> str:
    """Run inference on the model."""
    # Prepare text from messages
    text = image_processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    # Open image
    image = Image.open(image_path).convert("RGB")

    # Process inputs using the processor
    inputs = image_processor(
        text=[text], images=[image], padding=True, return_tensors="pt"
    )

    # Move inputs to model device
    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    # Generate response
    with torch.no_grad():
        generated_ids = model.generate(
            **inputs,
            max_new_tokens=2048,
            do_sample=False,
        )

    # Decode output (skip the input tokens)
    generated_ids_trimmed = [
        out_ids[len(in_ids) :]
        for in_ids, out_ids in zip(inputs["input_ids"], generated_ids)
    ]
    output_text = image_processor.batch_decode(
        generated_ids_trimmed,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]

    return output_text


def main():
    """Main function to run the sanity check."""
    # Configuration
    model_path = "Uniphore/actio-ui-7b-sft"  # or other model variants
    image_path = "screenshot.png"
    instruction = "Click on the submit button"

    # Check if custom instruction provided
    if len(sys.argv) > 1:
        instruction = " ".join(sys.argv[1:])

    print(f"Loading model from: {model_path}")
    try:
        model, tokenizer, image_processor = load_model(model_path)
        print("✓ Model loaded successfully")
    except Exception as e:
        print(f"✗ Error loading model: {e}")
        return 1

    print(f"Processing image: {image_path}")
    print(f"Instruction: {instruction}")

    try:
        messages = create_grounding_messages(image_path, instruction)
        result = run_inference(model, tokenizer, image_processor, messages, image_path)

        print("\n" + "=" * 60)
        print("MODEL OUTPUT:")
        print("=" * 60)
        print(result)
        print("=" * 60)
        return 0
    except Exception as e:
        print(f"✗ Error during inference: {e}")
        import traceback

        traceback.print_exc()
        return 1


if __name__ == "__main__":
    sys.exit(main())
```



# Legal

##  License
This project is licensed under the Open Model, Data, & Weights License Agreement (OpenMDW). See the LICENSE file in the root folder for details.

## Prohibited Uses
The model may not be used for any purpose or activity that violates applicable laws or regulations in any jurisdiction.
Use for illegal, unethical, or harmful activities is strictly prohibited.

## Disclaimer
ActIO-UI are intended for research and educational purposes only.

The authors, contributors, and copyright holders are not responsible for any illegal, unethical, or harmful use of the Software, nor for any direct or indirect damages resulting from such use.
Use of the name, logo, or trademarks of "ActIO", "ActIO-UI" "WARC-Bench", or "Uniphore" does not imply any endorsement or affiliation unless separate written permission is obtained.
Users are solely responsible for ensuring their use complies with applicable laws and regulations.



# Citation
If you find our work useful, please cite with

```
@misc{srivastava2025warcbenchwebarchivebased,
      title={WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions}, 
      author={Sanjari Srivastava and Gang Li and Cheng Chang and Rishu Garg and Manpreet Kaur and Charlene Y. Lee and Yuezhang Li and Yining Mao and Ignacio Cases and Yanan Xie and Peng Qi},
      year={2025},
      eprint={2510.09872},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.09872}, 
}
```