# Add OpenTelemetry Tracing to an aiohttp App

Yes. An agent instruments your aiohttp server and its SQLAlchemy engine with OpenTelemetry, exports spans over OTLP, tests them in memory, and opens a pull request.

[Run this on Rush](https://cloud.getrush.ai/run/add-opentelemetry/aiohttp/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the app is built and tested before it adds a tracer, because an instrumentation that starts before the application is created, or that emits a deprecation warning under a test suite that treats warnings as errors, breaks the build instead of adding spans. In an app laid out like the `polls` demo in aio-libs/aiohttp-demos, `requirements-dev.txt` pins `aiohttp` 3.13.5, `aiohttp-jinja2` 1.6, `asyncpg` 0.32.0, `trafaret_config` 2.0.2, `SQLAlchemy` 2.0.54, `pytest` 9.1.1, `pytest-aiohttp` 1.1.1 and `flake8`. `aiohttpdemo_polls/main.py` has `init_app`, which creates a `web.Application`, stores the config under a `web.AppKey`, sets up Jinja2, appends `pg_context` to `cleanup_ctx`, and calls `setup_routes`, `setup_static_routes` and `setup_middlewares`; `main` then calls `web.run_app` with the host and port from `config/polls.yaml`. `settings.py` parses that YAML through `trafaret_config` against the `TRAFARET` schema in `utils.py`, which allows exactly `postgres`, `host` and `port` and refuses any other key. `db.py` builds a SQLAlchemy 2 async engine with `create_async_engine` and an `async_sessionmaker` inside `pg_context`, and `views.py` opens a session per request for `index`, `poll`, `results` and `vote`. `middlewares.py` renders the `404.html` and `500.html` templates and logs unhandled exceptions. `pytest.ini` sets `asyncio_mode = auto`, `filterwarnings = error` and `xfail_strict`, and `tests/conftest.py` builds the app with `init_app` against `config/polls_test.yaml` and a real PostgreSQL. The repository's `ci.yml` runs `flake8 demos`, then every demo's tests with `PYTHONASYNCIODEBUG=1` against PostgreSQL and Redis services, and builds the Sphinx docs with spelling checks.

Tracing becomes a module and a config block. The agent adds `opentelemetry-api`, `opentelemetry-sdk`, `opentelemetry-instrumentation-aiohttp-server`, `opentelemetry-instrumentation-sqlalchemy` and `opentelemetry-exporter-otlp-proto-http` to `requirements-dev.txt`, pinned like their neighbours, and extends `TRAFARET` with an optional `tracing` key holding `enabled`, `service_name` and `endpoint`, because the schema rejects keys it does not know and a new key in `polls.yaml` alone would fail config parsing. `aiohttpdemo_polls/tracing.py` has one function, `setup_tracing(config)`, that builds a `Resource` with `service.name`, a `TracerProvider` with a `BatchSpanProcessor` around an `OTLPSpanExporter` pointed at the configured endpoint, and calls `AioHttpServerInstrumentor().instrument(tracer_provider=provider)`, which the instrumentation's documentation names as the entry point that traces `aiohttp.web.Application` requests. It returns the provider so a caller can shut it down.

Where each call sits follows the documentation and the app's own lifecycle. `main` calls `setup_tracing` before `init_app`, since the instrumentor patches application creation. In `pg_context`, right after `create_async_engine`, the agent calls `SQLAlchemyInstrumentor().instrument(engine=engine.sync_engine)`, the form that works for an async engine, so every `select` in `views.py` becomes a child span of its request span, and the context's teardown calls `uninstrument` and flushes the provider so the last spans leave before `engine.dispose()`. Static files under `/static/` are excluded with the `OTEL_PYTHON_AIOHTTP_SERVER_EXCLUDED_URLS` variable the documentation lists, set in the config rather than the environment. Request spans are named `GET /poll/{question_id}` from the matched route, which is what the instrumentation's `get_default_span_name` produces when a resource is resolved, so one route is one name regardless of the id.

Tests use the SDK's in-memory exporter, so no collector is needed. A new `tests/test_tracing.py` builds the app with tracing on and an `InMemorySpanExporter` in place of OTLP, requests `/poll/1/results` through the `aiohttp_client` fixture, and asserts one server span with that route name, a child span from SQLAlchemy, and the HTTP status attribute set; a second test asserts that `/static/style.css` produces no span. Because `filterwarnings = error` is on, any deprecation warning the OpenTelemetry packages raise on the pinned Python fails the test, so the agent pins versions that are silent and says which in the pull request. It runs `flake8 demos` and `pytest demos/polls` against a PostgreSQL it starts in its sandbox, and pastes the tails into the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add OpenTelemetry tracing to this aiohttp app. Read requirements-dev.txt, aiohttpdemo_polls/main.py, settings.py, utils.py, db.py, routes.py, views.py, middlewares.py, config/polls.yaml, pytest.ini, tests/conftest.py and the repository's CI workflow first. Add pinned opentelemetry-api, opentelemetry-sdk, opentelemetry-instrumentation-aiohttp-server, opentelemetry-instrumentation-sqlalchemy and opentelemetry-exporter-otlp-proto-http to requirements-dev.txt. Extend the trafaret schema with an optional tracing block holding enabled, service_name and endpoint, and add it to polls.yaml. Write aiohttpdemo_polls/tracing.py with setup_tracing(config) that builds a Resource and TracerProvider with a BatchSpanProcessor and OTLP HTTP exporter and calls AioHttpServerInstrumentor().instrument, called from main before init_app. Instrument the SQLAlchemy engine's sync_engine inside pg_context after it is created and uninstrument and flush on teardown. Exclude the static route. Write tests with InMemorySpanExporter that assert a server span named for the matched route with a SQLAlchemy child span and the status attribute, and that static files produce no span; keep pytest's filterwarnings=error passing. Run flake8 and pytest for this demo against PostgreSQL in the sandbox and open a pull request with the output. Do not change other demos or the CI workflow.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds `aiohttpdemo_polls/tracing.py` and `tests/test_tracing.py`, changes `main.py`, `db.py` and `utils.py`, adds the `tracing` block to `config/polls.yaml` and `config/polls_test.yaml`, and pins five packages in `requirements-dev.txt`. Its description lists each span the test saw for one request to the results page, with its name, its parent and its attributes, and pastes the tails of `flake8 demos` and `pytest demos/polls` from the sandbox.

The repository's `CI` workflow runs on the pull request as before, since `make install` reads the demo's requirements file and `make test` runs `pytest demos/polls`, so the reviewer sees the new tests pass against the workflow's PostgreSQL service with `PYTHONASYNCIODEBUG=1` on. No span leaves the test process; the OTLP exporter is only built when `tracing.enabled` is true and no in-memory exporter is supplied.

Before merging, a reviewer sets the endpoint for the collector the team runs, decides whether `vote` should carry the question and choice ids as span attributes, and decides whether the 500 handler in `middlewares.py` should record the exception on the active span so a traced error shows the traceback rather than only the status.

## Run it on every aiohttp service

A team with several aiohttp services sends this body once per repository from its own server, each a `POST /o/{org}/sessions` with that repository in `resources` and a `request_id` that is a UUID v5 of the repository's full name, so a retried call returns the run already started. Each run reads its own app: a service on `asyncpg` without SQLAlchemy gets the asyncpg instrumentation instead, one that already configures a `TracerProvider` for its outgoing `ClientSession` gets the server instrumentor added to that provider, and one whose config is environment variables rather than YAML gets `OTEL_EXPORTER_OTLP_ENDPOINT` read directly.

Once spans flow, a slow trace is the next event. A Grafana contact point can send a webhook with a custom `Authorization: Bearer` header, so an alert on request span duration can call a Rush `webhook` trigger directly with the trigger's secret, with an `Idempotency-Key` header on the trigger's URL made from the alert's fingerprint so a repeated notification does not start a second run. The automation's prompt asks for the slow route's handler and its queries to be read and a comment naming the likely cause.

## Why run it on Rush?

OpenTelemetry on aiohttp is a few calls in the right order and a few traps specific to the app: an instrumentor that must run before the application exists, a config schema that refuses unknown keys, an async engine that has to be instrumented through its sync engine, and a test suite that fails on any warning. The agent reads the lifecycle and the test configuration first and runs the suite in its own sandbox against a real database, so what reaches the pull request has passed the repository's own rules.

The span list and the test output stay with the run and are pasted into the pull request, so the review is of traces the app produced. Several services get the same instrumentation at once, each with its own engine, config format and route list.

## Frequently asked questions

### Why extend the config schema instead of using environment variables?

Because this app validates `polls.yaml` with a `trafaret` schema that refuses unknown keys, and its tests build the app from a second YAML file. A `tracing` block in the schema keeps one source of configuration and lets the test config turn the exporter off. The endpoint can still be overridden by `OTEL_EXPORTER_OTLP_ENDPOINT`, which the SDK reads.

### Will the tests need a collector?

No. The tests pass an `InMemorySpanExporter` into `setup_tracing`, read the finished spans from it and assert on their names and attributes. The OTLP exporter is only built when tracing is enabled and no exporter is supplied.

### Why instrument the sync engine?

Because SQLAlchemy's async engine wraps a sync engine, and the OpenTelemetry SQLAlchemy instrumentation attaches to that one. Instrumenting `engine.sync_engine` inside `pg_context` is what makes each `select` in the views a child of the request span.

### What about the warnings-as-errors setting?

It stays. `pytest.ini` turns every warning into a failure, so the agent pins OpenTelemetry versions that raise none on the pinned Python and says which in the pull request. A future upgrade that starts warning will fail the suite, which is the point of the setting.

## More aiohttp jobs

- [Add Pagination to an aiohttp App](https://getrush.ai/run/add-pagination/aiohttp/)
- [Add Prometheus Metrics to an aiohttp App](https://getrush.ai/run/add-prometheus-metrics/aiohttp/)
- [Add Rate Limiting to an aiohttp App](https://getrush.ai/run/add-rate-limiting/aiohttp/)

## The same job on other stacks

- [Add OpenTelemetry Tracing to an ASP.NET Core API](https://getrush.ai/run/add-opentelemetry/aspnet-core/)
- [Add OpenTelemetry Tracing to a chi API](https://getrush.ai/run/add-opentelemetry/chi/)
- [Add OpenTelemetry Tracing to a Django REST Framework API](https://getrush.ai/run/add-opentelemetry/django-rest-framework/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
