# Add OpenTelemetry Tracing to GORM Queries in a Go Service

Yes. An agent registers GORM's OpenTelemetry plugin, threads the request context into every query so SQL spans sit under the HTTP span, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-opentelemetry/gorm/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads how the database handle is built and used before it registers anything. In a service shaped like gotify/server, `New` in `database/database.go` builds a `gorm.Config` with a `logger.New` that writes through zerolog with a 200 millisecond `SlowThreshold`, opens MySQL, Postgres or SQLite from the `dialect` string, caps the pool with `SetMaxOpenConns(10)` (1 for SQLite), runs `AutoMigrate` for the user, application, message, client and plugin models, and returns a `GormDatabase` whose only field is `DB *gorm.DB`. `go.mod` pins `gorm.io/gorm` v1.31.2, `gin-gonic/gin` v1.12.0 and `rs/zerolog`, and already carries `go.opentelemetry.io/otel` v1.45.0 as an indirect dependency, which is the SDK version the agent pins its direct imports to. `serve` in `app.go` sets up the zerolog console writer, calls `database.New`, then `router.Create` and `runner.Run`, and `Run` in `runner/runner.go` serves an `http.Server` and shuts it down with a 5 second deadline on `SIGINT` or `SIGTERM`.

It then finds what would make the plugin useless on its own: none of the `GormDatabase` methods take a context. `GetMessagesByUserSince` in `database/message.go` runs `d.DB.Joins(...).Where(...).Order(...).Limit(limit).Find(&messages)` straight off the shared handle, and `database/application.go`, `client.go`, `user.go` and `plugin.go` do the same, so GORM's plugin would start every SQL span from `context.Background()` as a root with no parent request. The agent makes the change the plugin needs: each `GormDatabase` method takes `ctx context.Context` first and starts from `d.DB.WithContext(ctx)`, the per-handler interfaces in `api/` that name those methods (`MessageDatabase`, `ApplicationDatabase`, `ClientDatabase`, `UserDatabase`, `HealthDatabase` and the plugin's), and the `Database` interface in `auth/authentication.go` that the token checks use, change with them, handlers pass `ctx.Request.Context()`, and the places with no request, the five minute ticker in `router.Create` that calls `UpdateClientTokensLastUsedAndExpiresAt` and `CleanupExpiredClients`, the plugin manager and the startup code in `database.New`, pass `context.Background()`. Most of the diff is this one mechanical change, and the pull request says so up front.

With the context in place, the instrumentation is short. In `database.New` it adds `db.Use(tracing.NewPlugin(tracing.WithoutQueryVariables(), tracing.WithoutMetrics()))` from `gorm.io/plugin/opentelemetry/tracing`, which hooks GORM's Create, Query, Update, Delete, Row and Raw callbacks and records the statement without its bound values, so a message body or a password hash never lands in a span; `WithoutMetrics` keeps it to tracing, because pool statistics are a separate decision. In `router.Create` it adds `otelgin.Middleware("gotify")` from the contrib module before `accessLogger`, filtered so `/health` and the long-lived `/stream` websocket are not traced, and `accessLogger` gains a `trace_id` field read from the request's span context, so the existing `HTTP` log line links to the trace. A new `tracing` package builds the provider: an OTLP HTTP exporter from `otlptracehttp`, which reads `OTEL_EXPORTER_OTLP_ENDPOINT` and the other standard variables itself, a `stdouttrace` exporter for local use, a resource with `service.name` set to `gotify`, a batch processor and the `TraceContext` and `Baggage` propagators. Configuration follows the pattern in `config/`: a `Tracing` struct with `Enabled` and `Exporter`, keys `GOTIFY_TRACING_ENABLED` and `GOTIFY_TRACING_EXPORTER` in `keys.go`, `parseBool` and `parseString` lines in `Get`, and a commented block in `gotify-server.env.example`. `serve` builds the provider after the config is read and defers its shutdown.

To prove it, the agent starts the server in its sandbox on SQLite with `GOTIFY_TRACING_ENABLED=true` and the stdout exporter, creates an application as the default admin, posts a message with its token and reads the spans it prints: a `POST /message` server span with the GORM child spans for the token lookup and the insert beneath it. It runs `go test --race ./...`, the Go half of the `make test` target `.github/workflows/build.yml` runs, after updating the call sites in `database/*_test.go`, `api/*_test.go` and `router/router_test.go` to pass a context, and runs `golangci-lint run` as `make check-go` does. No route or swagger annotation changes, so `make check-swagger` is unaffected, and the agent says so rather than regenerating `docs/spec.json` for nothing.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add OpenTelemetry tracing to GORM in this Go service. Read go.mod, app.go, database/database.go, every file in database/, api/ and router/, runner/runner.go, config/config.go, config/keys.go, config/parse.go, gotify-server.env.example, Makefile and .github/workflows/build.yml first. Register gorm.io/plugin/opentelemetry/tracing in database.New with WithoutQueryVariables and WithoutMetrics. The GormDatabase methods take no context, so change each one to take a ctx first argument and start from d.DB.WithContext(ctx); update the database interfaces in api/ and auth/authentication.go, pass ctx.Request.Context() from the gin handlers, pass context.Background() from the router ticker, the plugin manager and startup, and update every test call site. Add otelgin.Middleware before accessLogger in router.Create, skipping /health and /stream, and add the trace id to the accessLogger line as trace_id. Add a tracing package that builds a TracerProvider with an OTLP HTTP exporter reading the standard OTEL_EXPORTER_OTLP_* variables, a stdouttrace exporter for local use, a resource with service.name gotify, a batch processor and the TraceContext and Baggage propagators; add GOTIFY_TRACING_ENABLED and GOTIFY_TRACING_EXPORTER to config following the existing parse pattern, default off, documented in gotify-server.env.example; build the provider in serve and defer its shutdown. Start the server on SQLite with tracing on and the stdout exporter, create an application, post a message with its token, and paste the spans into the pull request with the output of go test --race ./... and golangci-lint run. Change no route, response body or swagger annotation.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

The pull request adds the `tracing` package and edits `app.go`, `database/database.go`, the other files under `database/`, the handlers and interfaces under `api/`, `router/router.go`, `config/config.go`, `config/keys.go`, `gotify-server.env.example`, the test files whose calls gained a context, and `go.mod` and `go.sum`. Its description opens with the one sentence that matters for review, that the context argument is a mechanical change across every database method, then pastes the `go test --race ./...` output, the `golangci-lint run` result, and the spans the stdout exporter printed for one `POST /message`, with the server span and its GORM children, so the parent and child relationship is visible rather than asserted.

A deployment that leaves `GOTIFY_TRACING_ENABLED` unset gets no provider and no exporter: the middleware and plugin record into the SDK's no-op provider and the responses are unchanged. The one behavior difference the description names is that a query now carries the request's context, so a client that disconnects mid-request cancels its query where before it ran to completion.

Before merging, a reviewer chooses the sampler, since the default always-on is right for a home server and expensive for a busy one, confirms that statements without bound values are enough for their debugging, and decides whether the Docker image in `docker/Dockerfile` should set a default `OTEL_EXPORTER_OTLP_ENDPOINT` or leave it to the deployment.

## Run it on every service without traces

An engineer who has just stood up a collector starts this from the terminal with this body and goes back to what they were doing. One `POST /o/{org}/sessions` per Go service, each with its own `resources` URL, and the run reads that service's real database layer, so where the context has to be threaded and which interfaces change come from the code rather than from a template.

Send a `request_id`, a UUID, with each body so a retried call returns the run it already started. A key runs at most `max_concurrent` sessions at once, 4 by default, and a request past that answers `202` with `status: "queued"` and starts when a slot frees. A `webhook_url` on the key brings `run.completed` for each run the key started, with `data.session_url` to read for the pull request link.

## Why run it on Rush?

The plugin is one line; the work is everything around it. Whether SQL spans attach to the request depends on a context that this codebase never passed, and the only way to know the trace is right is to start the service, send a request and read the spans. The agent does that in its own sandbox, with the real binary on a real SQLite file, and the pull request carries the span tree it saw rather than a description of what should happen.

The transcript keeps every command and its output, so the reviewer reads the test run and the spans without reproducing the setup. A team with several Go services on GORM traces them in parallel, each in its own sandbox, and nothing merges until a person has read a diff that, in this case, touches every database method and deserves that read.

## Frequently asked questions

### Why does the diff touch every database method?

GORM's plugin reads the parent span from the statement's context, which is `context.Background()` unless the query was built with `WithContext`. Every `GormDatabase` method here runs straight off the shared handle, so without the change each SQL span would be a root trace with no link to the request. Adding a `ctx` argument is the smallest change that makes the spans mean something, and it is mechanical, which the pull request says in its first line.

### Will span attributes include message bodies or tokens?

No. The plugin is registered with `WithoutQueryVariables`, so the statement attribute holds the SQL with placeholders and not the values bound to it. The gin middleware records route, method and status. The access log already masks `token=` in query strings, and the agent keeps that behavior.

### Why skip /stream and /health?

`/stream` is the websocket clients hold open for hours; a span for it would stay open as long and be useless when it closed. `/health` is polled by the Docker healthcheck and load balancers and would dominate the trace volume with nothing to show. Both are left out with the middleware's filter, and the filter is one list to edit.

### Which exporter does it use?

OTLP over HTTP from `otlptracehttp`, configured through the standard `OTEL_EXPORTER_OTLP_*` environment variables, so any collector or vendor that speaks OTLP works without code changes. `GOTIFY_TRACING_EXPORTER=stdout` selects the `stdouttrace` exporter for local runs, which is what the pull request's evidence was produced with.

### Does the swagger check in CI still pass?

Yes. `make check-swagger` regenerates `docs/spec.json` from the `swagger:` comments and fails if it differs from the committed file. The change adds no route and no annotation, so the generated spec is identical, and the agent leaves `docs/spec.json` untouched rather than committing a no-op regeneration.

## More GORM jobs

- [Add Optimistic Locking to a GORM Model in a Gin API](https://getrush.ai/run/add-optimistic-locking/gorm/)
- [Add Prometheus Metrics to a GORM Service from a Linear Issue](https://getrush.ai/run/add-prometheus-metrics/gorm/)
- [Add Query Timeouts to a GORM App](https://getrush.ai/run/add-query-timeouts/gorm/)

## The same job on other stacks

- [Add OpenTelemetry Tracing to a Koa API](https://getrush.ai/run/add-opentelemetry/koa/)
- [Add OpenTelemetry Tracing to a Ktor Server](https://getrush.ai/run/add-opentelemetry/ktor/)
- [Add OpenTelemetry Tracing to a Litestar App](https://getrush.ai/run/add-opentelemetry/litestar/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
