# ClawBench

> ClawBench is a comprehensive benchmark for evaluating AI browser agents on 153 real-world everyday online tasks across 144 live websites and 8 categories. It captures 5 layers of behavioral data (session replay, screenshots, HTTP traffic, agent reasoning traces, and browser actions), includes human ground-truth for every task, and scores with an agentic evaluator providing step-level traceable diagnostics. The best-performing model (Claude Sonnet 4.6) achieves only 33.3% success rate, revealing a large gap between current AI agents and human-level web task completion.

## Links

- [Paper](https://arxiv.org/abs/2604.08523): ClawBench: Can AI Agents Complete Everyday Online Tasks? (arXiv:2604.08523)
- [PDF](https://arxiv.org/pdf/2604.08523): Full paper PDF
- [Website](https://claw-bench.com): Interactive leaderboard, task browser, trace viewer, and agent demo gallery
- [GitHub](https://github.com/TIGER-AI-Lab/ClawBench): Source code — framework, evaluators, test driver, and Chrome extension
- [Dataset](https://huggingface.co/datasets/NAIL-Group/ClawBench): 153 tasks in Parquet format on Hugging Face
- [Hugging Face Papers](https://huggingface.co/papers/2604.08523): Community discussion page
- [PyPI](https://pypi.org/project/clawbench-eval/): Install with `pip install clawbench-eval`

## Key Facts

- 153 tasks across 144 live websites in 8 categories
- Categories: Daily Life, Finance, Work & Office, Development, Academic, Travel, Social, Pets
- 5 layers of behavioral data: session replay (rrweb), screenshots, HTTP traffic, agent reasoning traces, browser actions
- Human ground-truth recorded for every task
- Agentic evaluator with VLM, LLM, and Human-Agent evaluation modes providing step-level traceable diagnostics
- Request interceptor prevents irreversible real-world actions (payments, form submissions) during evaluation
- 7 models evaluated: Claude Sonnet 4.6, GLM-5, Gemini 3 Flash, Claude Haiku 4.5, GPT-5.4, Kimi K2.5, Gemini 3.1 Flash Lite
- Apache 2.0 license
- COLM 2026 submission
- 21 authors from 11 institutions

## Leaderboard — Overall Success Rate

| Model | Provider | Overall |
|-------|----------|---------|
| Claude Sonnet 4.6 | Anthropic | 33.3% |
| GLM-5 | Zhipu AI | 24.2% |
| Gemini 3 Flash | Google | 19.0% |
| Claude Haiku 4.5 | Anthropic | 18.3% |
| Kimi K2.5 | Moonshot AI | 15.0% |
| GPT-5.4 | OpenAI | 6.5% |
| Gemini 3.1 Flash Lite | Google | 3.3% |

## Leaderboard — Per-Category Success Rate (%)

| Model | Daily | Finance | Work | Dev | Academic | Travel | Social | Pets |
|-------|-------|---------|------|-----|----------|--------|--------|------|
| Claude Sonnet 4.6 | 44.2 | 50.0 | 19.0 | 11.1 | 50.0 | 23.1 | 38.9 | 18.2 |
| GLM-5 | 30.8 | 16.7 | 38.1 | 16.7 | 28.6 | 0.0 | 16.7 | 18.2 |
| Gemini 3 Flash | 15.4 | 33.3 | 23.8 | 22.2 | 28.6 | 30.8 | 11.1 | 0.0 |
| Claude Haiku 4.5 | 15.4 | 33.3 | 19.0 | 27.8 | 21.4 | 7.7 | 16.7 | 18.2 |
| Kimi K2.5 | 15.4 | 16.7 | 14.3 | 11.1 | 14.3 | 15.4 | 16.7 | 9.1 |
| GPT-5.4 | 9.6 | 0.0 | 0.0 | 11.1 | 7.1 | 7.7 | 0.0 | 9.1 |
| Gemini 3.1 Flash Lite | 1.9 | 0.0 | 0.0 | 5.6 | 14.3 | 0.0 | 0.0 | 9.1 |

## Task Categories (21 Metaclasses)

| Category | Tasks | Example Platforms |
|----------|-------|-------------------|
| daily-life | 21 | Uber Eats, Instacart, Zillow, AMC Theatres, Costco |
| entertainment-hobbies | 15 | Goodreads, Eventbrite, Fandango, Ticketmaster |
| creation-init | 13 | ClickUp, Typeform, Ghost, Squarespace, Canva |
| office-secretary-tasks | 9 | Trello, Calendly, Purelymail, Notion |
| rating-voting | 10 | TripAdvisor, Glassdoor, Yelp, G2 |
| education-learning | 9 | Coursera, LeetCode, Blinkist, Udemy |
| travel | 9 | Google Flights, Hipcamp, Airbnb, Booking.com |
| beauty-personal-care | 9 | TaskRabbit, Booksy, Soko Glam, MAC Cosmetics |
| pet-animal-care | 8 | Rover, Petfinder, Chewy, Embark |
| job-search-hr | 8 | Indeed, Greenhouse, ZipRecruiter, LinkedIn |
| academia-research | 5 | Zotero, Overleaf, Google Scholar |
| finance-investment | 6 | Insureon, Trezor, Robinhood |
| government-civic | 5 | LegalNature, USA.gov |
| nonprofit-charity | 4 | Idealist, GoFundMe |
| personal-management | 4 | EveryDollar, Mint |
| health-wellness | 4 | Curology, Jefit |
| home-garden | 3 | HomeDepot, Lowe's |
| automotive | 3 | CarGurus, AutoTrader |
| food-cooking | 3 | King Arthur Baking, HelloFresh |
| real-estate | 2 | Zillow, Redfin |
| sports-fitness | 3 | Nike, Peloton |

## Evaluation Framework

ClawBench provides three complementary evaluation modes:

### VLM Evaluator (Vision-Language Model)
Uses a multimodal model to examine screenshots captured during task execution. The evaluator receives the task instruction, screenshots at each step, and a structured rubric, then determines whether the agent successfully completed each required action. This evaluator excels at catching visual errors like wrong form fields, incorrect selections, or navigation mistakes.

### LLM Evaluator (Language Model)
Analyzes the agent's reasoning traces, browser action logs, and intercepted HTTP requests. It verifies that the agent took the correct sequence of actions, submitted the right data in forms, and reached the expected final state. This evaluator is particularly strong at catching data entry errors and incorrect API calls.

### Human-Agent Evaluator
Combines automated analysis with human-recorded ground-truth traces. For each task, the human recording provides the expected sequence of actions, timing, and final state. The evaluator compares the agent's trace against this baseline to identify deviations.

All three evaluators produce step-level traceable diagnostics, meaning they report exactly which step succeeded or failed and why. The final pass/fail decision uses majority voting across the three evaluators.

## Request Interceptor

A key innovation in ClawBench is the request interceptor mechanism that enables safe evaluation on live production websites. For each task, an `eval_schema` defines:

- `url_pattern`: A regex matching the final HTTP request (e.g., checkout, form submission, booking confirmation)
- `method`: The HTTP method to intercept (typically POST)

When the agent's browser sends a request matching this pattern, the interceptor:
1. Captures the full request (URL, headers, body) as evidence
2. Blocks the request before it reaches the server
3. Returns a synthetic success response to the browser

This allows evaluating whether the agent correctly filled out forms, selected the right items, and initiated the proper checkout flow — without actually completing purchases, submitting applications, or making reservations.

## Data Layers

Each task execution produces 5 layers of data:

1. **Session Replay (rrweb)**: Full DOM-level recording that can be replayed in a browser, capturing every visual change
2. **Screenshots**: Captured at each agent action step for visual verification
3. **HTTP Traffic**: All network requests and responses, including the intercepted final request
4. **Agent Reasoning Traces**: The agent's chain-of-thought, tool calls, and decision-making process
5. **Browser Actions**: Low-level browser events (clicks, keystrokes, navigation, scrolling) with timestamps

## Installation

```bash
pip install clawbench-eval
clawbench info
```

## Dataset Usage

```python
from datasets import load_dataset

ds = load_dataset("NAIL-Group/ClawBench", split="test")
print(f"Tasks: {len(ds)}")
print(ds[0])
```

## Authors

Yuxuan Zhang (UBC, Vector Institute, Etude AI), Yubo Wang (Vector Institute, U Waterloo), Yipeng Zhu (UBC), Penghui Du (Etude AI), Junwen Miao (CMU), Xuan Lu (SJTU), Wendong Xu (UniPat AI), Yunzhuo Hao (ZJU), Songcheng Cai (U Waterloo), Xiaochen Wang (HKUST), Huaisong Zhang (Tsinghua), Xian Wu (Etude AI), Yi Lu (U Waterloo), Minyi Lei (U Waterloo), Kai Zou (Netmind.ai), Huifeng Yin (UniPat AI), Ping Nie (U Waterloo), Liang Chen (UniPat AI), Dongfu Jiang (Vector Institute, U Waterloo), Wenhu Chen (Vector Institute, U Waterloo), Kelsey R. Allen (UBC, Vector Institute)

## Citation

```bibtex
@article{zhang2026clawbench,
  title={ClawBench: Can AI Agents Complete Everyday Online Tasks?},
  author={Zhang, Yuxuan and Wang, Yubo and Zhu, Yipeng and Du, Penghui and Miao, Junwen and Lu, Xuan and Xu, Wendong and Hao, Yunzhuo and Cai, Songcheng and Wang, Xiaochen and Zhang, Huaisong and Wu, Xian and Lu, Yi and Lei, Minyi and Zou, Kai and Yin, Huifeng and Nie, Ping and Chen, Liang and Jiang, Dongfu and Chen, Wenhu and Allen, Kelsey R.},
  journal={arXiv preprint arXiv:2604.08523},
  year={2026}
}
```