# Add OpenTelemetry Tracing to a chi API

Yes. An agent wraps your chi router and your database queries in OpenTelemetry spans, exports them over OTLP and opens a pull request with a trace to read.

[Run this on Rush](https://cloud.getrush.ai/run/add-opentelemetry/chi/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how a request moves through the service before it adds a span anywhere. In an API laid out like dhax/go-base, `go.mod` says `go 1.25.0` and depends on `go-chi/chi/v5` v5.4.0, `uptrace/bun` v1.2.15 with its `pgdriver` and `bundebug` packages, `sirupsen/logrus`, `spf13/viper` and `spf13/cobra`; `go.opentelemetry.io/otel` v1.41.0 is already there as an indirect dependency of bun, which decides the SDK version the agent pins. `New` in `api/api.go` builds the router with `middleware.Recoverer`, `middleware.RequestID`, `middleware.Timeout(15 * time.Second)`, the logrus request logger from `logging/logger.go`, `render.SetContentType` and an optional CORS handler, then mounts `/auth`, `/admin` and `/api` and serves `/healthz` and the SPA. `NewServer` and `Start` in `api/server.go` run `ListenAndServe` and call `Shutdown` on `os.Interrupt`. `DBConn` in `database/postgres.go` opens Postgres through `pgdriver`, checks the connection with a `SELECT 1`, and adds `bundebug.NewQueryHook` when `db_debug` is set, so a query hook is a pattern this codebase already has.

It finds the one thing that would make database spans useless if left alone: every store method passes `context.Background()`. `List`, `Create`, `Get`, `Update` and `Delete` in `database/admAccountStore.go`, and `Get`, `Update`, `Delete`, `UpdateToken` and `DeleteToken` in `database/accountStore.go`, each call `Scan(context.Background())` or `Exec(context.Background())`, so a bun query hook would start every SQL span as a root with no parent. The agent threads a `context.Context` through: each store method takes `ctx` as its first argument, the `AccountStore` interface in `api/admin/accounts.go` and its siblings in `api/app` and `auth/pwdless` change with them, handlers pass `r.Context()`, and `MockAuthStore` in `auth/pwdless/mockAuthStore.go` gets the same signatures so `api_test.go` keeps compiling. It also notes in the pull request that `middleware.Timeout` cancels that context after 15 seconds, so a cancelled query now shows up as a span with an error rather than a silent hang.

Then it instruments the two edges. On the router it adds `otelchi.Middleware("go-base", otelchi.WithChiRoutes(r))` from `github.com/riandyrn/otelchi` right after `middleware.RequestID` and before the request logger, so each span is named by its chi route pattern, `/admin/accounts/{accountID}` rather than `/admin/accounts/42`; chi panics on a `Use` after a `Mount`, so the placement in `New` matters and the agent keeps it above the mounts. In `NewLogEntry` it adds the trace id from `trace.SpanContextFromContext(r.Context())` as a `trace_id` field, so the existing `request started` and `request complete` lines link to the trace. On the database it adds `bunotel.NewQueryHook(bunotel.WithDBName("postgres"))` from `github.com/uptrace/bun/extra/bunotel` beside the `bundebug` hook in `DBConn`, leaving `WithFormattedQueries` off so parameter values never reach a span. A new `tracing` package builds the provider: an OTLP HTTP exporter from `otlptracehttp`, which reads `OTEL_EXPORTER_OTLP_ENDPOINT` and the other standard variables itself, a resource with `service.name` from a new viper key `tracing_service_name`, a batch span processor, `otel.SetTracerProvider` and a `TraceContext` plus `Baggage` propagator. `Start` in `api/server.go` flushes and shuts the provider down after `srv.Shutdown`, and `cmd/serve.go` gets defaults for `tracing_enabled` (off) and the service name, with matching lines in `.env.example` and `docker-compose.yml`.

To see it work without a collector, the agent also wires the `stdouttrace` exporter behind `tracing_exporter=stdout`. In its sandbox it installs Postgres with `apt-get`, runs `go run main.go migrate`, starts `go run main.go serve` with tracing on and the stdout exporter, requests `/healthz` and `/auth/login`, and reads the spans it prints: a server span named by the route with the bun spans beneath it carrying `db.name` and the statement. It runs `go build ./...`, `go vet ./...` and `go test -v ./...`, the same command `.github/workflows/test.yml` runs, and regenerates `routes.md` with `go run main.go gendoc -r` so the middleware list there shows the new entry.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add OpenTelemetry tracing to this chi API. Read go.mod, api/api.go, api/server.go, cmd/serve.go, cmd/root.go, database/postgres.go, every file in database/, api/admin/, api/app/ and auth/pwdless/, logging/logger.go, .env.example and docker-compose.yml first. Add a tracing package that builds a TracerProvider with an OTLP HTTP exporter (reading the standard OTEL_EXPORTER_OTLP_* variables), a stdouttrace exporter selectable with a tracing_exporter=stdout setting for local use, a resource with service.name from a tracing_service_name setting, a batch processor and the TraceContext and Baggage propagators; make it a no-op when tracing_enabled is false, and shut it down after srv.Shutdown in Start. Add otelchi.Middleware with WithChiRoutes to the router after RequestID and before the request logger, keeping it above every Mount, and add the trace id to the logrus request entry as trace_id. Add bunotel.NewQueryHook with WithDBName in DBConn beside the bundebug hook, without formatted queries. Every store method currently uses context.Background(); change each store method and its interface to take a ctx first argument, pass r.Context() from the handlers, and update MockAuthStore so the existing tests compile. Add defaults for the new settings in cmd/serve.go and lines in .env.example and docker-compose.yml. Install Postgres in your sandbox, run the migrations, start the server with the stdout exporter, request /healthz and /auth/login, and paste the spans into the pull request with the output of go build ./..., go vet ./... and go test -v ./.... Regenerate routes.md with go run main.go gendoc -r. Change nothing about authentication, routes or the response bodies.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds the `tracing` package and edits `api/api.go`, `api/server.go`, `cmd/serve.go`, `database/postgres.go`, the store files and interfaces under `database/`, `api/admin/`, `api/app/` and `auth/pwdless/`, `logging/logger.go`, `.env.example`, `docker-compose.yml`, `routes.md`, `go.mod` and `go.sum`. Its description pastes the `go test -v ./...` output, the spans the stdout exporter printed for one `POST /auth/login`, with the `/auth/login` server span and the bun span for the account lookup under it, and the `request complete` log line that now carries `trace_id`, so a reviewer can match the three without checking the branch out.

Nothing changes for a deployment that leaves `TRACING_ENABLED` unset: the provider is never built, the middleware still runs but records against the SDK's no-op provider, and the responses are byte for byte what they were. The context change is the one behavior difference, and the description calls it out: a request that hits the 15 second `middleware.Timeout` now cancels its queries instead of letting them finish after the client has gone.

Before merging, a reviewer decides the sampler (the default is always-on, which is fine for a boilerplate and expensive for a busy API), confirms that the statement attribute without parameter values is enough for their debugging, and chooses whether the collector endpoint belongs in `docker-compose.yml` for local development or only in production configuration.

## Run it on every service without traces

An engineer who has just wired up a collector starts this from the terminal with this body, one `POST /o/{org}/sessions` per chi service, each with its own `resources` URL, and reads the pull requests when they come in. The run reads each service's real router, stores and config loading, so the middleware position, the context threading and the setting names match that codebase rather than a generic snippet.

Send a `request_id`, a UUID, with each body so a retried call returns the run it already started. A key runs at most `max_concurrent` sessions at once, 4 by default, and a request past that answers `202` with `status: "queued"` and starts when a slot frees. Give the key a `webhook_url` and Rush posts `run.completed` for each run the key started, with `data.session_url` to read for the pull request link.

## Why run it on Rush?

Tracing changes are easy to get subtly wrong in ways a build does not catch: a `Use` after a `Mount` that panics at startup, SQL spans that never attach to the request because the store used `context.Background()`, an exporter that silently drops everything because the endpoint variable was misspelled. The agent proves each one in its own sandbox by starting Postgres, running the migrations, starting the server and reading the spans it emits, and the pull request carries that evidence rather than a claim.

The transcript keeps every command and its output, so the reviewer sees the exact span tree and the test run without reproducing the setup. A team with several Go services traces them in parallel, each run in its own sandbox with its own database, and nothing merges until a person reads the diff.

## Frequently asked questions

### Why does it change the store interfaces instead of just adding the query hook?

Because bunotel reads the parent span from the context passed to `Scan` and `Exec`, and every store method in this codebase passes `context.Background()`. With the hook alone, each SQL span would be its own root trace with no link to the request that caused it. Threading `r.Context()` through the stores is what makes the database spans children of the HTTP span, and it also means a timed-out request cancels its query.

### Will the spans contain query parameter values or user data?

No. The bun hook is added without `WithFormattedQueries`, so the statement attribute holds the query with placeholders and `db.name`, not the email addresses or tokens bound to it. The chi middleware records the route pattern, method and status. If you want full statements in a development environment, that is one option to flip, and the pull request says where.

### Which exporter does it use?

OTLP over HTTP from the `otlptracehttp` package, which reads `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS` and the other standard variables itself, so it works with any collector or vendor that speaks OTLP. The `stdouttrace` exporter is there for local runs and for the evidence in the pull request; it is selected only when `TRACING_EXPORTER=stdout`.

### Do the existing tests still pass?

The `auth/pwdless` tests build a router with the real logging middleware and a `MockAuthStore`. The agent updates the mock's method signatures along with the interface, so `go test -v ./...` runs the same tests it did before, and the pull request pastes that output. There is no test against a real database in this repository, which is why the agent runs the server against Postgres in its sandbox instead.

### What happens when tracing is turned off?

With `TRACING_ENABLED` unset or false, no provider is built and no exporter is started. The middleware and query hook still run but record into OpenTelemetry's no-op provider, which is how the SDK is designed to behave, so the cost is a few allocations per request and nothing leaves the process.

## More chi jobs

- [Add Prometheus Metrics to a chi API](https://getrush.ai/run/add-prometheus-metrics/chi/)
- [Add Rate Limiting to a Go chi API](https://getrush.ai/run/add-rate-limiting/chi/)
- [Add Request ID Logging to a chi App](https://getrush.ai/run/add-request-id-logging/chi/)

## The same job on other stacks

- [Add OpenTelemetry Tracing to a Django REST Framework API](https://getrush.ai/run/add-opentelemetry/django-rest-framework/)
- [Add OpenTelemetry Tracing to an Echo API](https://getrush.ai/run/add-opentelemetry/echo/)
- [Add OpenTelemetry Tracing to an Express Service](https://getrush.ai/run/add-opentelemetry/express/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
