# OSWorld hard-20: selection rule (fixed before any agent ran)

Written 2026-10-20, before any agent run on these tasks. The rule, the seed and the
resulting task list are published with the scores; nothing below is changed after results exist.

## Pool

OSWorld-Verified `test_all` at xlang-ai/OSWorld `b138d348256078fa634fc3b73567a7337c793e6b`
(the tree cua-bench pins), restricted to the hard desktop domains:
`multi_apps`, `libreoffice_calc`, `libreoffice_writer`, `libreoffice_impress`, `gimp`, `thunderbird`.

## Exclusions (mechanical, from the task files alone)

1. Every task in `test_small`. Those 39 tasks were used while developing computer-cli 0.1.11/0.1.12,
   so they are not a fair test.
2. Tasks cua-bench marks `container_fidelity: degraded` (they need systemd, GNOME settings daemons,
   audio and similar, which the container variant lacks).
3. Infeasible tasks (evaluator `infeasible`): the harness has no way for an agent to declare one.
4. Tasks that read or write Google Drive (`googledrive` anywhere in the task): they need OSWorld's
   OAuth settings.

## Draw

Sort the remaining task ids, shuffle them with Python's `random.Random(20261020)`, and walk that order.
A task is kept when a no-agent baseline (`cb run task … --noop`) completes setup and evaluation and
scores 0. A task that errors there cannot be run by any tool in this container and is skipped; the
walk continues with the next id. Stop at 20 kept tasks.

## Environment

Cua's own image `ghcr.io/trycua/bench-osworld@sha256:e55cf3a3…504439` with one change for every tool
alike: Chrome and VS Code start with `--no-sandbox` (`Dockerfile` in this folder). In the stock image
Docker's default seccomp profile stops both at launch ("Failed to move to new namespace"), so every
Chrome and VS Code task failed for every agent.

## Arms, trials, model

- Ours: `computer mcp --host vnc://127.0.0.1:5900` from the published `@phnx-labs/computer-cli-linux-x64@0.1.12`.
- Cua: `cua mcp --sandbox <box> --permissions computer:all` from cua-bench at trycua/cua `dfe1323`.
- Each arm gets its vendor's published skill text appended to the system prompt.
- Claude Code headless, model `claude-sonnet-5-5`, `--max-turns 100`, 30-minute timeout, 3 trials per
  task per arm, every run on a fresh container. Score: OSWorld's own evaluator for the task.