What the agent does
The agent starts with how the suite is configured and run, because a timeout plugin is a dependency, a config key and a CI matrix question. In a project laid out like pallets/itsdangerous, pyproject.toml keeps the test tools in a tests dependency group (freezegun and pytest, locked in uv.lock at 1.5.2 and 8.3.5), runs pytest through tox with uv-venv-lock-runner and the flags -v --tb=short --basetemp={env_tmp_dir}, and sets filterwarnings = ["error"] under [tool.pytest.ini_options] so any warning fails a test. .github/workflows/tests.yaml runs uv run --locked tox run -e py3.13 and friends on Ubuntu, plus a Windows row, a Mac row and PyPy. The 37 tests across five files in tests/test_itsdangerous/ take fractions of a second, and test_timed.py freezes the clock with freezegun in an autouse fixture.
It then adds the plugin where the repository keeps such things. pytest-timeout joins the tests group in pyproject.toml, uv lock updates uv.lock, which the uv-lock pre-commit hook checks, and timeout goes under [tool.pytest.ini_options] next to filterwarnings, with a value well above the suite's longest test so it only fires for a hang. The tox commands need no change, since pytest loads the plugin by its entry point. Nothing is added to the workflow: --locked installs the new pin from the lockfile.
Two things come from reading rather than from the plugin's README. The CI matrix has a Windows row, and the plugin's signal method does not exist there, so the agent sets no timeout_method and lets the plugin pick signal on POSIX and thread on Windows, and says in the pull request why the config is silent on it. And freezegun patches time.time and time.monotonic in test_timed.py, which could make a naive timer think no time passes; the plugin's alarm is a real SIGALRM, which freezegun does not touch, so a hang inside a frozen test still trips. The agent proves that rather than asserting it.
In its sandbox the agent runs uv sync and uv run --locked tox run -e py3.13 to see the suite green with the plugin loaded and the timeout in effect. It then writes a throwaway test that calls time.sleep for longer than the limit inside the FreezeMixin fixture, runs it once to read the Failed: Timeout line with its stack, and deletes it, quoting that output in the description so the reviewer sees the plugin fire under the frozen clock. A @pytest.mark.timeout on one test is left as an example only in the description, since no test here needs its own limit.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test timeouts to this pytest suite with pytest-timeout. Read pyproject.toml, uv.lock, .pre-commit-config.yaml, .github/workflows/tests.yaml and tests/test_itsdangerous/test_timed.py first. Add pytest-timeout to the tests dependency group, run uv lock, and set a timeout under [tool.pytest.ini_options] well above the slowest test. Do not set timeout_method, because the CI matrix includes Windows; say why in the pull request. Do not change the tox commands or the workflow. Run uv sync and uv run --locked tox run -e py3.13 and quote the result. Then write a temporary test that sleeps past the limit inside the freezegun fixture from test_timed.py, run it to capture the Timeout failure, delete it, and quote that output in the description. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request with two files changed, pyproject.toml and uv.lock: one line in the tests group, one timeout key in the pytest options, and the lock entries for the plugin. No test file changes and no workflow change. The description quotes the tox run with the plugin listed in pytest's header line and all tests passing, then the Failed: Timeout output from the temporary sleeping test under the frozen clock, with the stack pointing at the sleep.
The run's record keeps both pytest runs and the uv lock output, so a reviewer can confirm the lockfile change is only the plugin and its own dependencies.
Before merging, a reviewer sets the limit to taste, since the agent picked a number far above the suite's longest test, and decides whether to add timeout_func_only so slow fixtures are not counted against a test's own time.
Run it on every nightly
The timeout earns its keep on the runs nobody watches. A schedule trigger with cron set to a quiet hour in the trigger's timezone starts a run that executes the suite on the latest dependency versions, the way the tox config's update-requirements environment refreshes uv.lock, and opens a pull request only when a test hangs or fails, quoting the Timeout line with the test's name and stack.
A hang that used to hold a CI runner until the job limit now ends in seconds with a named test. The run's needs_review state, when the suite does not pass, is the signal to read; a green night produces no pull request at all.
The use case this job follows, with its automation and trigger: Write the missing tests.
Why run it on Rush?
A hung test is the one failure CI reports worst: the job sits until its own limit, the log ends mid-line, and nobody knows which test it was. Adding the plugin is a one-line change that is easy to get wrong on a mixed matrix, where a signal method set by hand breaks every Windows row. In a Rush sandbox the agent reads the matrix, keeps the config portable, and proves the alarm fires under the suite's frozen clock before anyone merges it.
The two pytest runs stay with the run and the pull request quotes them, so the reviewer reads the Timeout failure with its stack rather than taking the plugin's behaviour on faith. Across every repository a team maintains, the job runs in parallel, one sandbox and one pull request each.
Frequently asked questions
Will freezegun stop the timeout from firing?
No. freezegun patches the clock functions a test reads, but the plugin's default method on Linux and Mac arms a real alarm signal, which keeps running on wall-clock time. The pull request includes the output of a sleeping test under the FreezeMixin fixture failing with Timeout to show it.
Why not set timeout_method explicitly?
Because the CI matrix includes Windows, where the signal method does not exist and pytest would error on collection. Left unset, the plugin uses signal where it can and thread on Windows, so one config serves every row.
Does this slow the suite down?
No. The plugin arms a timer per test and cancels it when the test ends; the suite's runtime is unchanged. The limit only matters when a test does not end.
Can one slow test have its own limit?
Yes, with @pytest.mark.timeout(seconds) on that test, or timeout_func_only = true in the config so fixture setup is excluded from the count. Nothing here needs either, so the pull request leaves them as notes in the description.
More pytest jobs
The same job on other stacks
More jobs like this: Write the missing tests
- Add Unit Tests to a dbt Project (dbt)
- Add Visual Regression Tests With Playwright (Playwright)