All jobs

Add OpenTelemetry Tracing to GORM Queries in a Go Service

Yes. An agent registers GORM's OpenTelemetry plugin, threads the request context into every query so SQL spans sit under the HTTP span, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads how the database handle is built and used before it registers anything. In a service shaped like gotify/server, New in database/database.go builds a gorm.Config with a logger.New that writes through zerolog with a 200 millisecond SlowThreshold, opens MySQL, Postgres or SQLite from the dialect string, caps the pool with SetMaxOpenConns(10) (1 for SQLite), runs AutoMigrate for the user, application, message, client and plugin models, and returns a GormDatabase whose only field is DB *gorm.DB. go.mod pins gorm.io/gorm v1.31.2, gin-gonic/gin v1.12.0 and rs/zerolog, and already carries go.opentelemetry.io/otel v1.45.0 as an indirect dependency, which is the SDK version the agent pins its direct imports to. serve in app.go sets up the zerolog console writer, calls database.New, then router.Create and runner.Run, and Run in runner/runner.go serves an http.Server and shuts it down with a 5 second deadline on SIGINT or SIGTERM.

It then finds what would make the plugin useless on its own: none of the GormDatabase methods take a context. GetMessagesByUserSince in database/message.go runs d.DB.Joins(...).Where(...).Order(...).Limit(limit).Find(&messages) straight off the shared handle, and database/application.go, client.go, user.go and plugin.go do the same, so GORM's plugin would start every SQL span from context.Background() as a root with no parent request. The agent makes the change the plugin needs: each GormDatabase method takes ctx context.Context first and starts from d.DB.WithContext(ctx), the per-handler interfaces in api/ that name those methods (MessageDatabase, ApplicationDatabase, ClientDatabase, UserDatabase, HealthDatabase and the plugin's), and the Database interface in auth/authentication.go that the token checks use, change with them, handlers pass ctx.Request.Context(), and the places with no request, the five minute ticker in router.Create that calls UpdateClientTokensLastUsedAndExpiresAt and CleanupExpiredClients, the plugin manager and the startup code in database.New, pass context.Background(). Most of the diff is this one mechanical change, and the pull request says so up front.

With the context in place, the instrumentation is short. In database.New it adds db.Use(tracing.NewPlugin(tracing.WithoutQueryVariables(), tracing.WithoutMetrics())) from gorm.io/plugin/opentelemetry/tracing, which hooks GORM's Create, Query, Update, Delete, Row and Raw callbacks and records the statement without its bound values, so a message body or a password hash never lands in a span; WithoutMetrics keeps it to tracing, because pool statistics are a separate decision. In router.Create it adds otelgin.Middleware("gotify") from the contrib module before accessLogger, filtered so /health and the long-lived /stream websocket are not traced, and accessLogger gains a trace_id field read from the request's span context, so the existing HTTP log line links to the trace. A new tracing package builds the provider: an OTLP HTTP exporter from otlptracehttp, which reads OTEL_EXPORTER_OTLP_ENDPOINT and the other standard variables itself, a stdouttrace exporter for local use, a resource with service.name set to gotify, a batch processor and the TraceContext and Baggage propagators. Configuration follows the pattern in config/: a Tracing struct with Enabled and Exporter, keys GOTIFY_TRACING_ENABLED and GOTIFY_TRACING_EXPORTER in keys.go, parseBool and parseString lines in Get, and a commented block in gotify-server.env.example. serve builds the provider after the config is read and defers its shutdown.

To prove it, the agent starts the server in its sandbox on SQLite with GOTIFY_TRACING_ENABLED=true and the stdout exporter, creates an application as the default admin, posts a message with its token and reads the spans it prints: a POST /message server span with the GORM child spans for the token lookup and the insert beneath it. It runs go test --race ./..., the Go half of the make test target .github/workflows/build.yml runs, after updating the call sites in database/*_test.go, api/*_test.go and router/router_test.go to pass a context, and runs golangci-lint run as make check-go does. No route or swagger annotation changes, so make check-swagger is unaffected, and the agent says so rather than regenerating docs/spec.json for nothing.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add OpenTelemetry tracing to GORM in this Go service. Read go.mod, app.go, database/database.go, every file in database/, api/ and router/, runner/runner.go, config/config.go, config/keys.go, config/parse.go, gotify-server.env.example, Makefile and .github/workflows/build.yml first. Register gorm.io/plugin/opentelemetry/tracing in database.New with WithoutQueryVariables and WithoutMetrics. The GormDatabase methods take no context, so change each one to take a ctx first argument and start from d.DB.WithContext(ctx); update the database interfaces in api/ and auth/authentication.go, pass ctx.Request.Context() from the gin handlers, pass context.Background() from the router ticker, the plugin manager and startup, and update every test call site. Add otelgin.Middleware before accessLogger in router.Create, skipping /health and /stream, and add the trace id to the accessLogger line as trace_id. Add a tracing package that builds a TracerProvider with an OTLP HTTP exporter reading the standard OTEL_EXPORTER_OTLP_* variables, a stdouttrace exporter for local use, a resource with service.name gotify, a batch processor and the TraceContext and Baggage propagators; add GOTIFY_TRACING_ENABLED and GOTIFY_TRACING_EXPORTER to config following the existing parse pattern, default off, documented in gotify-server.env.example; build the provider in serve and defer its shutdown. Start the server on SQLite with tracing on and the stdout exporter, create an application, post a message with its token, and paste the spans into the pull request with the output of go test --race ./... and golangci-lint run. Change no route, response body or swagger annotation.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

The pull request adds the tracing package and edits app.go, database/database.go, the other files under database/, the handlers and interfaces under api/, router/router.go, config/config.go, config/keys.go, gotify-server.env.example, the test files whose calls gained a context, and go.mod and go.sum. Its description opens with the one sentence that matters for review, that the context argument is a mechanical change across every database method, then pastes the go test --race ./... output, the golangci-lint run result, and the spans the stdout exporter printed for one POST /message, with the server span and its GORM children, so the parent and child relationship is visible rather than asserted.

A deployment that leaves GOTIFY_TRACING_ENABLED unset gets no provider and no exporter: the middleware and plugin record into the SDK's no-op provider and the responses are unchanged. The one behavior difference the description names is that a query now carries the request's context, so a client that disconnects mid-request cancels its query where before it ran to completion.

Before merging, a reviewer chooses the sampler, since the default always-on is right for a home server and expensive for a busy one, confirms that statements without bound values are enough for their debugging, and decides whether the Docker image in docker/Dockerfile should set a default OTEL_EXPORTER_OTLP_ENDPOINT or leave it to the deployment.

Run it on every service without traces

An engineer who has just stood up a collector starts this from the terminal with this body and goes back to what they were doing. One POST /o/{org}/sessions per Go service, each with its own resources URL, and the run reads that service's real database layer, so where the context has to be threaded and which interfaces change come from the code rather than from a template.

Send a request_id, a UUID, with each body so a retried call returns the run it already started. A key runs at most max_concurrent sessions at once, 4 by default, and a request past that answers 202 with status: "queued" and starts when a slot frees. A webhook_url on the key brings run.completed for each run the key started, with data.session_url to read for the pull request link.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

The plugin is one line; the work is everything around it. Whether SQL spans attach to the request depends on a context that this codebase never passed, and the only way to know the trace is right is to start the service, send a request and read the spans. The agent does that in its own sandbox, with the real binary on a real SQLite file, and the pull request carries the span tree it saw rather than a description of what should happen.

The transcript keeps every command and its output, so the reviewer reads the test run and the spans without reproducing the setup. A team with several Go services on GORM traces them in parallel, each in its own sandbox, and nothing merges until a person has read a diff that, in this case, touches every database method and deserves that read.

Frequently asked questions

Why does the diff touch every database method?

GORM's plugin reads the parent span from the statement's context, which is context.Background() unless the query was built with WithContext. Every GormDatabase method here runs straight off the shared handle, so without the change each SQL span would be a root trace with no link to the request. Adding a ctx argument is the smallest change that makes the spans mean something, and it is mechanical, which the pull request says in its first line.

Will span attributes include message bodies or tokens?

No. The plugin is registered with WithoutQueryVariables, so the statement attribute holds the SQL with placeholders and not the values bound to it. The gin middleware records route, method and status. The access log already masks token= in query strings, and the agent keeps that behavior.

Why skip /stream and /health?

/stream is the websocket clients hold open for hours; a span for it would stay open as long and be useless when it closed. /health is polled by the Docker healthcheck and load balancers and would dominate the trace volume with nothing to show. Both are left out with the middleware's filter, and the filter is one list to edit.

Which exporter does it use?

OTLP over HTTP from otlptracehttp, configured through the standard OTEL_EXPORTER_OTLP_* environment variables, so any collector or vendor that speaks OTLP works without code changes. GOTIFY_TRACING_EXPORTER=stdout selects the stdouttrace exporter for local runs, which is what the pull request's evidence was produced with.

Does the swagger check in CI still pass?

Yes. make check-swagger regenerates docs/spec.json from the swagger: comments and fails if it differs from the committed file. The change adds no route and no annotation, so the generated spec is identical, and the agent leaves docs/spec.json untouched rather than committing a no-op regeneration.

More GORM jobs

The same job on other stacks

Keep reading