All jobs

Add Structured Logging to an aiohttp App

Yes. An agent gives an aiohttp app JSON logs: a request id middleware, a JSON access logger, a JSON formatter for every logger, tests under pytest-aiohttp, and a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

An aiohttp app logs from three places, the access log, the error path and whatever logging.basicConfig set up, and the agent reads all three before changing any. In the polls demo under demos/polls of aio-libs/aiohttp-demos, aiohttpdemo_polls/main.py builds the app in init_app: it stores the parsed config under config_key, sets up aiohttp_jinja2 with a package loader, appends pg_context from db.py to cleanup_ctx to open an async SQLAlchemy engine against Postgres and dispose it on shutdown, and calls setup_routes, setup_static_routes and setup_middlewares. Its main() calls logging.basicConfig(level=logging.DEBUG) and web.run_app with the host and port from config/polls.yaml, so every logger, including SQLAlchemy's, writes plain text at debug level. middlewares.py has one middleware, built by create_error_middleware, which renders 404.html and 500.html through overrides and on any other exception calls request.protocol.logger.exception("Error handling request"), a line with no path, no method and no way to tie it to a user's report. requirements-dev.txt pins aiohttp==3.13.5, aiohttp-jinja2==1.6, asyncpg==0.32.0, SQLalchemy==2.0.54, pytest==9.1.1 and pytest-aiohttp==1.1.1.

The config is typed. settings.py parses a YAML file through trafaret_config with the schema in utils.py, a T.Dict of a postgres block plus host and port, and typedefs.py declares config_key and db_key as web.AppKeys, so an unknown key in the YAML is an error rather than a silent default. pytest.ini turns every warning into an error with filterwarnings = error, sets asyncio_mode = auto and points testpaths at tests/; tests/conftest.py builds the app with config/polls_test.yaml, creates the test database through init_db.py's setup_db as the postgres superuser, and the cli fixture wraps it in pytest-aiohttp's aiohttp_client. tests/test_integration.py has four tests, for the index, a results page, a 404 and a vote.

The agent adds a module aiohttpdemo_polls/logging.py with three parts. A JsonFormatter subclass of logging.Formatter whose format emits one json.dumps line with time, level, logger, message and, when the record carries them, request_id, method, path, status and duration_ms, with exc_info rendered into an exception field. A request_id_middleware decorated with @web.middleware that reads X-Request-ID from the request or makes a uuid4, stores it under a web.AppKey-style request key, sets it on the response header, and binds it to the record through a contextvars.ContextVar that the formatter reads, so a logger.exception deep in a view carries the id without passing it around. And an AccessLogger subclass of aiohttp.abc.AbstractAccessLogger whose log(request, response, time) emits the method, path, status, duration in milliseconds, remote address and the same id, replacing aiohttp's default access line. setup_middlewares registers the new middleware before the error one so the error line carries the id, main() replaces basicConfig with a configure_logging(level) that installs the formatter on the root handler, and web.run_app gets access_log_class=AccessLogger. A log_level key is added to the trafaret schema as optional with a default of INFO, and to both YAML files, because the schema would reject an unknown key.

To run the suite the agent needs a Postgres as .github/workflows/ci.yml provides through a service container. The sandbox has none, so it installs PostgreSQL from apt, sets the postgres role's password to postgres as init_db.py's ADMIN_DB_URL expects, installs requirements-dev.txt and runs pytest tests from demos/polls, where pytest.ini's filterwarnings = error means a deprecated logging call in the new module fails the run instead of printing. It adds tests beside the existing four: a response carries an X-Request-ID header and echoes one sent in, and with caplog a request to a missing poll produces a record that parses as JSON with status 404, path and the same request_id as the header. It also runs flake8 aiohttpdemo_polls as the demo's Makefile flake target does, since the root CI runs make lint across every demo.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add structured JSON logging to the aiohttp polls app under demos/polls. Read aiohttpdemo_polls/main.py, middlewares.py, settings.py, utils.py, typedefs.py, routes.py, views.py, db.py, config/polls.yaml, config/polls_test.yaml, pytest.ini, tests/conftest.py, tests/test_integration.py, requirements-dev.txt, the Makefile and the root .github/workflows/ci.yml first. Add aiohttpdemo_polls/logging.py with a JSON logging.Formatter, a request id middleware that reads or makes X-Request-ID, sets it on the response and binds it with a ContextVar so every record carries it, and an AbstractAccessLogger subclass that logs method, path, status, duration_ms, remote and request_id as JSON. Register the middleware before the error middleware, replace logging.basicConfig with a configure_logging call driven by a new optional log_level key added to the trafaret schema and both YAML files, and pass the access logger to web.run_app. Install PostgreSQL in the sandbox with the postgres password set to postgres, run pytest tests from demos/polls and flake8 aiohttpdemo_polls, add tests for the header and for a JSON record on a 404, and open a pull request with an example access line and an example error line.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds demos/polls/aiohttpdemo_polls/logging.py and changes main.py, middlewares.py, utils.py, config/polls.yaml, config/polls_test.yaml and tests/test_integration.py. Its description pastes one access line and one error line as the formatter writes them, with the same request_id in both, and the pytest and flake8 output from the sandbox. requirements-dev.txt gains nothing; the formatter, the middleware and the access logger use the standard library and aiohttp's own base class.

The repository's CI workflow runs make lint and PYTHONASYNCIODEBUG=1 make test against Postgres and Redis service containers, and make test runs pytest demos/polls among the other demos, so the new tests run on the pull request alongside the four existing ones. The run's record keeps the sandbox output so a reviewer can compare it with the check.

Before merging, a reviewer decides whether to trust an incoming X-Request-ID from any client, which log_level production should use now that the debug default is gone, and whether SQLAlchemy's engine logger should be quieter than the app's. Views, templates, routes and the database layer are unchanged.

Run it on every aiohttp service with plain-text logs

An organisation with several aiohttp services starts one run per repository from its own server: a POST /o/{org}/sessions each, with the repository in resources and a request_id that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own main.py and middleware list, so a service that already uses structlog gets the access logger and request id wired into its processors, and one that runs under Gunicorn with aiohttp.GunicornWebWorker gets the access log class set through the worker's access_log_class instead of web.run_app.

A team that files logging work as Linear issues runs this prompt from an automation in the Project with a linear trigger on issues labelled observability. The trigger adds the issue after the prompt, so an issue that asks for a particular field set or a log shipper's expected key names reaches the agent with those names, and the pull request links back to the issue.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The pieces are small and the failure is only visible when a request goes wrong: a middleware registered after the error handler leaves the exception line without an id, a formatter that touches a missing attribute raises inside logging and is swallowed, and under filterwarnings = error a deprecated call fails the suite in CI but not under a plain python -m aiohttpdemo_polls. The agent runs the suite against a real Postgres in its own sandbox with the same pytest.ini, so what arrives has produced the lines the pull request shows.

Every command stays with the run, from the PostgreSQL install to the caplog assertion, so a reviewer sees the JSON lines before any deploy. The same prompt runs across every aiohttp service at once, each in its own sandbox and with its own pull request.

Frequently asked questions

Why not add structlog or python-json-logger?

Because logging.Formatter and aiohttp.abc.AbstractAccessLogger cover this app's needs with no new pin in requirements-dev.txt, and the demo's CI installs every demo's requirements into one environment. A team that already standardises on structlog can ask for it in the prompt and the agent wires the same middleware and access logger into its processor chain.

How does a log line deep in a view get the request id?

The middleware sets a contextvars.ContextVar for the duration of the handler, and the formatter reads it when it renders any record. aiohttp runs each request's handler in its own task, so the variable does not leak between concurrent requests.

Does this change what SQLAlchemy logs?

The engine's logger keeps its level but its records go through the JSON formatter like every other, so echo output becomes JSON lines with logger set to sqlalchemy.engine.Engine. The new log_level key in the YAML sets the root level; the pull request shows how to set the engine's logger lower on its own.

Will the tests need a running database?

Yes, as they already do: tests/conftest.py creates and drops a database through init_db.py as the postgres superuser. The agent installs PostgreSQL in its sandbox for the run; CI provides it as a service container and nothing about that changes.

More aiohttp jobs

The same job on other stacks

Keep reading