All jobs

Add OpenTelemetry Tracing to a chi API

Yes. An agent wraps your chi router and your database queries in OpenTelemetry spans, exports them over OTLP and opens a pull request with a trace to read.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how a request moves through the service before it adds a span anywhere. In an API laid out like dhax/go-base, go.mod says go 1.25.0 and depends on go-chi/chi/v5 v5.4.0, uptrace/bun v1.2.15 with its pgdriver and bundebug packages, sirupsen/logrus, spf13/viper and spf13/cobra; go.opentelemetry.io/otel v1.41.0 is already there as an indirect dependency of bun, which decides the SDK version the agent pins. New in api/api.go builds the router with middleware.Recoverer, middleware.RequestID, middleware.Timeout(15 * time.Second), the logrus request logger from logging/logger.go, render.SetContentType and an optional CORS handler, then mounts /auth, /admin and /api and serves /healthz and the SPA. NewServer and Start in api/server.go run ListenAndServe and call Shutdown on os.Interrupt. DBConn in database/postgres.go opens Postgres through pgdriver, checks the connection with a SELECT 1, and adds bundebug.NewQueryHook when db_debug is set, so a query hook is a pattern this codebase already has.

It finds the one thing that would make database spans useless if left alone: every store method passes context.Background(). List, Create, Get, Update and Delete in database/admAccountStore.go, and Get, Update, Delete, UpdateToken and DeleteToken in database/accountStore.go, each call Scan(context.Background()) or Exec(context.Background()), so a bun query hook would start every SQL span as a root with no parent. The agent threads a context.Context through: each store method takes ctx as its first argument, the AccountStore interface in api/admin/accounts.go and its siblings in api/app and auth/pwdless change with them, handlers pass r.Context(), and MockAuthStore in auth/pwdless/mockAuthStore.go gets the same signatures so api_test.go keeps compiling. It also notes in the pull request that middleware.Timeout cancels that context after 15 seconds, so a cancelled query now shows up as a span with an error rather than a silent hang.

Then it instruments the two edges. On the router it adds otelchi.Middleware("go-base", otelchi.WithChiRoutes(r)) from github.com/riandyrn/otelchi right after middleware.RequestID and before the request logger, so each span is named by its chi route pattern, /admin/accounts/{accountID} rather than /admin/accounts/42; chi panics on a Use after a Mount, so the placement in New matters and the agent keeps it above the mounts. In NewLogEntry it adds the trace id from trace.SpanContextFromContext(r.Context()) as a trace_id field, so the existing request started and request complete lines link to the trace. On the database it adds bunotel.NewQueryHook(bunotel.WithDBName("postgres")) from github.com/uptrace/bun/extra/bunotel beside the bundebug hook in DBConn, leaving WithFormattedQueries off so parameter values never reach a span. A new tracing package builds the provider: an OTLP HTTP exporter from otlptracehttp, which reads OTEL_EXPORTER_OTLP_ENDPOINT and the other standard variables itself, a resource with service.name from a new viper key tracing_service_name, a batch span processor, otel.SetTracerProvider and a TraceContext plus Baggage propagator. Start in api/server.go flushes and shuts the provider down after srv.Shutdown, and cmd/serve.go gets defaults for tracing_enabled (off) and the service name, with matching lines in .env.example and docker-compose.yml.

To see it work without a collector, the agent also wires the stdouttrace exporter behind tracing_exporter=stdout. In its sandbox it installs Postgres with apt-get, runs go run main.go migrate, starts go run main.go serve with tracing on and the stdout exporter, requests /healthz and /auth/login, and reads the spans it prints: a server span named by the route with the bun spans beneath it carrying db.name and the statement. It runs go build ./..., go vet ./... and go test -v ./..., the same command .github/workflows/test.yml runs, and regenerates routes.md with go run main.go gendoc -r so the middleware list there shows the new entry.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add OpenTelemetry tracing to this chi API. Read go.mod, api/api.go, api/server.go, cmd/serve.go, cmd/root.go, database/postgres.go, every file in database/, api/admin/, api/app/ and auth/pwdless/, logging/logger.go, .env.example and docker-compose.yml first. Add a tracing package that builds a TracerProvider with an OTLP HTTP exporter (reading the standard OTEL_EXPORTER_OTLP_* variables), a stdouttrace exporter selectable with a tracing_exporter=stdout setting for local use, a resource with service.name from a tracing_service_name setting, a batch processor and the TraceContext and Baggage propagators; make it a no-op when tracing_enabled is false, and shut it down after srv.Shutdown in Start. Add otelchi.Middleware with WithChiRoutes to the router after RequestID and before the request logger, keeping it above every Mount, and add the trace id to the logrus request entry as trace_id. Add bunotel.NewQueryHook with WithDBName in DBConn beside the bundebug hook, without formatted queries. Every store method currently uses context.Background(); change each store method and its interface to take a ctx first argument, pass r.Context() from the handlers, and update MockAuthStore so the existing tests compile. Add defaults for the new settings in cmd/serve.go and lines in .env.example and docker-compose.yml. Install Postgres in your sandbox, run the migrations, start the server with the stdout exporter, request /healthz and /auth/login, and paste the spans into the pull request with the output of go build ./..., go vet ./... and go test -v ./.... Regenerate routes.md with go run main.go gendoc -r. Change nothing about authentication, routes or the response bodies.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds the tracing package and edits api/api.go, api/server.go, cmd/serve.go, database/postgres.go, the store files and interfaces under database/, api/admin/, api/app/ and auth/pwdless/, logging/logger.go, .env.example, docker-compose.yml, routes.md, go.mod and go.sum. Its description pastes the go test -v ./... output, the spans the stdout exporter printed for one POST /auth/login, with the /auth/login server span and the bun span for the account lookup under it, and the request complete log line that now carries trace_id, so a reviewer can match the three without checking the branch out.

Nothing changes for a deployment that leaves TRACING_ENABLED unset: the provider is never built, the middleware still runs but records against the SDK's no-op provider, and the responses are byte for byte what they were. The context change is the one behavior difference, and the description calls it out: a request that hits the 15 second middleware.Timeout now cancels its queries instead of letting them finish after the client has gone.

Before merging, a reviewer decides the sampler (the default is always-on, which is fine for a boilerplate and expensive for a busy API), confirms that the statement attribute without parameter values is enough for their debugging, and chooses whether the collector endpoint belongs in docker-compose.yml for local development or only in production configuration.

Run it on every service without traces

An engineer who has just wired up a collector starts this from the terminal with this body, one POST /o/{org}/sessions per chi service, each with its own resources URL, and reads the pull requests when they come in. The run reads each service's real router, stores and config loading, so the middleware position, the context threading and the setting names match that codebase rather than a generic snippet.

Send a request_id, a UUID, with each body so a retried call returns the run it already started. A key runs at most max_concurrent sessions at once, 4 by default, and a request past that answers 202 with status: "queued" and starts when a slot frees. Give the key a webhook_url and Rush posts run.completed for each run the key started, with data.session_url to read for the pull request link.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Tracing changes are easy to get subtly wrong in ways a build does not catch: a Use after a Mount that panics at startup, SQL spans that never attach to the request because the store used context.Background(), an exporter that silently drops everything because the endpoint variable was misspelled. The agent proves each one in its own sandbox by starting Postgres, running the migrations, starting the server and reading the spans it emits, and the pull request carries that evidence rather than a claim.

The transcript keeps every command and its output, so the reviewer sees the exact span tree and the test run without reproducing the setup. A team with several Go services traces them in parallel, each run in its own sandbox with its own database, and nothing merges until a person reads the diff.

Frequently asked questions

Why does it change the store interfaces instead of just adding the query hook?

Because bunotel reads the parent span from the context passed to Scan and Exec, and every store method in this codebase passes context.Background(). With the hook alone, each SQL span would be its own root trace with no link to the request that caused it. Threading r.Context() through the stores is what makes the database spans children of the HTTP span, and it also means a timed-out request cancels its query.

Will the spans contain query parameter values or user data?

No. The bun hook is added without WithFormattedQueries, so the statement attribute holds the query with placeholders and db.name, not the email addresses or tokens bound to it. The chi middleware records the route pattern, method and status. If you want full statements in a development environment, that is one option to flip, and the pull request says where.

Which exporter does it use?

OTLP over HTTP from the otlptracehttp package, which reads OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS and the other standard variables itself, so it works with any collector or vendor that speaks OTLP. The stdouttrace exporter is there for local runs and for the evidence in the pull request; it is selected only when TRACING_EXPORTER=stdout.

Do the existing tests still pass?

The auth/pwdless tests build a router with the real logging middleware and a MockAuthStore. The agent updates the mock's method signatures along with the interface, so go test -v ./... runs the same tests it did before, and the pull request pastes that output. There is no test against a real database in this repository, which is why the agent runs the server against Postgres in its sandbox instead.

What happens when tracing is turned off?

With TRACING_ENABLED unset or false, no provider is built and no exporter is started. The middleware and query hook still run but record into OpenTelemetry's no-op provider, which is how the SDK is designed to behave, so the cost is a few allocations per request and nothing leaves the process.

More chi jobs

The same job on other stacks

Keep reading