Add OpenTelemetry Tracing to a Phoenix App

The agent reads what the app already measures before it adds a tracer. In a Phoenix app like hexpm/hexpm, mix.exs requires Elixir 1.20 and pulls phoenix ~> 1.6, phoenix_live_view ~> 1.0, bandit ~> 1.0, ecto_sql, postgrex, oban ~> 2.23, broadway, finch, sentry ~> 13.0, prom_ex ~> 1.11, telemetry_metrics, telemetry_poller and logger_json ~> 7.0. HexpmWeb.Telemetry is a supervisor with a telemetry_poller and metric definitions for phoenix.endpoint.stop.duration, phoenix.router_dispatch.stop.duration and the hexpm.repo_base.query.* events, which tells the agent the repo is Hexpm.RepoBase, and Hexpm.PromEx ships plugins for Phoenix, Ecto, DbConnection, outbound HTTP, email and Oban. Hexpm.Application.start/2 attaches HexpmWeb.BanditTelemetry and Hexpm.Emails.Telemetry before the children start. Metrics and error reporting exist; what is missing is a trace that joins a request, its queries and the jobs it enqueues.

It adds the dependencies to mix.exs with mix deps.get: opentelemetry_api, opentelemetry, opentelemetry_exporter, and the instrumentation libraries that match what the app runs, opentelemetry_bandit, opentelemetry_phoenix, opentelemetry_ecto, opentelemetry_oban and opentelemetry_finch. In Hexpm.Application.start/2, next to the two attach() calls already there, it adds the setup calls: Bandit and Phoenix with the Bandit adapter so one span covers the request from the socket to the controller, Ecto on the [:hexpm, :repo_base] event prefix the existing metrics already name, and Oban so a job carries the trace of the request that enqueued it. The releases entry in mix.exs gets the application ordering the exporter documents, since the Dockerfile builds a MIX_ENV=prod release.

Configuration follows the file layout the app uses. config/config.exs turns on the batch span processor; config/test.exs sets the traces exporter to none so mix test is unchanged; config/dev.exs uses the stdout exporter so a developer sees a span per request in the console; and config/runtime.exs, which already reads every production value with System.fetch_env!("HEXPM_..."), reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS the same way and sets the OTLP exporter to http_protobuf. The service name comes from the release, and the agent adds the current trace and span ids to the Logger metadata so the JSON lines logger_json writes line up with the trace in the backend.

Then it proves the change compiles and runs. .github/workflows/main.yml runs mix compile --warnings-as-errors and mix format --check-formatted with a Postgres 18 service and the test alias that creates, loads and migrates the database, so the agent starts Postgres in its sandbox, runs the same three commands, and boots the app in dev to make one request to the endpoint and read the span the stdout exporter prints, with the Ecto child spans under it. It touches no controller, no PromEx plugin and no Sentry configuration: the existing metrics keep working beside the new traces.

What comes back is a pull request that touches mix.exs, mix.lock, lib/hexpm/application.ex and the five files under config, and nothing else. The description lists each dependency added, quotes the setup block as it sits beside the existing attach() calls, and names the two environment variables production will need, with the lines in runtime.exs that read them.

The proof is in the description too: the output of mix compile --warnings-as-errors, mix format --check-formatted and mix test from the branch, and the span tree the stdout exporter printed for one request in dev, with the HTTP span at the root, the Phoenix router span under it and the Ecto query spans under that. The CI workflow runs the same compile and format checks on the pull request.

A reviewer checks that the Ecto prefix matches Hexpm.RepoBase, that test.exs really disables the exporter so the suite does not try to reach a collector, and that the release ordering in mix.exs is right for the Dockerfile build. Deploying means setting the endpoint and headers for the tracing backend; until then the application starts and the exporter logs that it has nowhere to send spans.

An organisation that runs several Phoenix services starts one run per repository with POST /o/{org}/sessions, this prompt and the repository in resources. Each run reads that service's own mix.exs, application.ex and config and picks the instrumentation libraries that match what it runs, so a service without Oban gets no Oban setup, and each opens its own pull request.

A webhook_url on the API key receives run.completed for each one with its session_url, so a script can collect the pull request links as the runs finish, and the key's max_concurrent limit decides how many services are being instrumented at once.

Adding tracing to an app this size is mostly reading: which adapter the endpoint runs on, what the repo's telemetry prefix is, where production configuration is read, and how the release is built. Getting one of those wrong compiles fine and traces nothing. In a Rush sandbox the agent reads the files, makes the change, and then boots the app and looks at a real span, with the exporter output kept in the transcript next to the diff that produced it.

The pull request carries the same checks CI runs, mix compile --warnings-as-errors among them, so a reviewer sees that the change passes the project's own bar before opening the diff. And because the sandbox had Postgres running, mix test ran for real rather than being skipped as too much trouble on a laptop.

The app already has PromEx metrics and Sentry. Does this replace them?

Will the test suite try to send spans somewhere?

Which tracing backend does it send to?

Do Oban jobs show up under the request that enqueued them?