What the agent does
The agent reads how the app is tested before it touches the workflow. In a multipage app shaped like streamlit/llm-examples, Chatbot.py is the entry script and five more pages live under pages/, from 1_File_Q&A.py to 5_Chat_with_user_feedback.py. The tests are in app_test.py and use AppTest from streamlit.testing.v1: test_Chatbot patches openai.resources.chat.Completions.create, runs AppTest.from_file("Chatbot.py"), sets the sidebar key input and a chat message, and asserts on the rendered chat messages; test_Langchain_Quickstart does the same for pages/3_Langchain_Quickstart.py with langchain.llms.OpenAI.__call__ patched. So two of the six scripts have a test and four have none, which is exactly what a coverage report will say, and the agent writes that expectation down before measuring.
Then it reads .github/workflows/app-testing.yml. The job checks out the repository, sets up Python 3.11 and runs streamlit/streamlit-app-action@v0.0.3 with app-path: Chatbot.py, ruff: true and pytest-args: -v --junit-xml=test-results.xml, then pmeier/pytest-results-action@v0.6.0 turns the JUnit file into a job summary. That action, not the repository, installs dependencies and runs pytest, so the agent fetches its action.yml to see which requirements files it installs and whether pytest-cov can be added through requirements-dev.txt and the pytest-args input. If the action installs only requirements.txt, the agent replaces the one step with explicit steps that do the same work in the open: pip install -r requirements.txt -r requirements-dev.txt, ruff check . with the line length of 105 from .ruff.toml, and pytest with coverage flags, so nothing the workflow did before is lost.
Coverage of a Streamlit script works because AppTest executes the script in the test process, so coverage.py traces Chatbot.py and the page files like any module. The agent adds pytest-cov to requirements-dev.txt next to pytest, black and mypy, writes a .coveragerc that measures the repository root and omits app_test.py itself, and runs pytest --cov=. --cov-report=term-missing --cov-report=xml in the sandbox with the same patches the tests use, so no API key is needed. It reads the per-file table: the two tested scripts near the top, the four untested pages at zero, and the lines in Chatbot.py that only run when a key is present. From that run it picks the threshold for --cov-fail-under, set a little under the measured total so the check fails on a regression and not on noise, and states the measured number in the pull request rather than a round one.
In the workflow it writes the coverage table into $GITHUB_STEP_SUMMARY beside the existing test summary, uploads coverage.xml as a workflow artifact, and keeps permissions: contents: read as the file already declares. It runs ruff check . and ruff format --check . locally because .pre-commit-config.yaml pins ruff hooks and the workflow enables ruff, then confirms the tests still pass under coverage before pushing.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add test coverage to this Streamlit app's CI. Read app_test.py, Chatbot.py, every file under pages/, requirements.txt, requirements-dev.txt, .ruff.toml, .pre-commit-config.yaml and .github/workflows/app-testing.yml first. Fetch the action.yml of the streamlit-app-action version the workflow pins and find out which requirements files it installs and how pytest-args reaches pytest; if coverage cannot be added through it, replace that step with explicit install, ruff and pytest steps that do the same work. Add pytest-cov to requirements-dev.txt, add a .coveragerc that omits the test file, run pytest with --cov=. --cov-report=term-missing --cov-report=xml in the sandbox, and read the per-file table. Set --cov-fail-under a few points below the total you measured and say the measured number in the pull request. Write the coverage table to the step summary, upload coverage.xml as an artifact, keep the existing JUnit summary and ruff checks, run ruff check and ruff format --check, and open a pull request that lists the untested pages by name.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch with four kinds of change: pytest-cov added to requirements-dev.txt, a new .coveragerc, the workflow edited so pytest runs with coverage, writes its table to the job summary and uploads coverage.xml, and a threshold that fails the job under the number the sandbox measured. If the pinned streamlit-app-action could not carry the coverage flags, the description explains why the step was replaced and shows that ruff and the JUnit summary still run.
The description quotes the per-file coverage table from the sandbox run, names the four page scripts with no test, and gives the measured total and the threshold chosen from it. It does not claim a CI run passed; the workflow runs on the pull request itself, and the checks on the pull request are the proof.
A reviewer checks three things: that the threshold is below the measured total so the first run passes, that app_test.py is omitted from the measurement so the suite does not inflate its own number, and that the workflow still fails on a ruff error. Then they decide whether the untested pages are a follow-up for the same session or a separate run with a prompt that asks for AppTest tests.
Run it on every Streamlit repository without a coverage check
Start one run per app: a POST /o/{org}/sessions with this prompt and the repository in resources, with a request_id per repository so a retry returns the run it already started. Each run measures its own suite and sets its own threshold, so an app with one tested page and an app with full AppTest coverage get different numbers written from their own output, not a shared default.
Give the API key a webhook URL and Rush posts run.completed or run.failed when each turn ends, with session_url for the full record; a run that stops because the tests fail under coverage is reported the same way, with the failing table in its transcript. When the pull request is merged and the team later adds tests for the other pages, a follow-up message on the parked session can raise the threshold from the new measurement.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
The question this job answers is a measurement: how much of the app the AppTest suite runs. On a laptop that means installing the dev requirements, patching the two LLM clients, running pytest with coverage and reading the table, then reading a third-party action's source to see whether it can carry the flags. The agent does all of it in its own sandbox with no API keys, because the tests already mock the clients, and quotes every output it based a decision on.
The record is what makes the threshold trustworthy. The run's transcript holds the pytest --cov output, the action.yml it fetched, and the ruff commands, so a reviewer can see the measured total the threshold was derived from instead of taking a round number on faith. The pull request lands on its own branch and the workflow proves itself on that branch before anyone merges.
Frequently asked questions
Does coverage work for Streamlit scripts that are not importable modules?
Yes. AppTest.from_file executes the script inside the pytest process, so coverage.py sees every line it runs. Pages that no test loads show zero, which is useful information rather than a tooling gap.
Will it write tests for the untested pages?
Not in this run. The prompt asks for a coverage check, and the pull request names the pages with no test so the team can decide. A follow-up message on the same session, or a second run that asks for AppTest tests for File Q&A and the search page, does that work with the coverage check already in place.
Do the tests need an OpenAI or Anthropic key in CI?
No. app_test.py patches openai.resources.chat.Completions.create and langchain.llms.OpenAI.call, so nothing leaves the runner. The agent keeps those patches and adds no secret to the workflow.
Will it keep the streamlit-app-action step?
If the pinned version installs requirements-dev.txt and passes pytest-args through to pytest, yes, and the change is two lines. If it does not, the agent replaces the step with explicit pip, ruff and pytest steps that do the same work, and says so in the pull request so the maintainer can see nothing was dropped.
Can it upload to a coverage service instead?
Ask for it in the prompt. By default the report goes to the job summary and a workflow artifact so no new account or token is needed. Adding a Codecov or Coveralls upload step is a small change on top, and the token is something the repository owner adds as a secret.