# Drop Health Check Spans from OpenTelemetry Traces

Yes. An agent finds every probe and synthetic check that produces spans, filters them at the SDK where it can and in the Collector where it must, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/drop-health-check-spans/opentelemetry/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository that holds the services and the Collector config and traces each probe to the spans it creates. In a system like open-telemetry/opentelemetry-demo, `compose.yaml` gives most services a TCP health check, which opens no HTTP request and so produces no span, but three paths do. The `image-provider` check runs `wget --spider` against `/status` every 5 seconds, and `src/image-provider/nginx.conf.template` turns `otel_trace on` for the whole server with `otel_span_name image-provider`, so every probe becomes a span; the Collector's own `nginx` receiver scrapes the same `/status` every 10 seconds. The Collector's `http_check/frontend-proxy` receiver requests the Envoy front proxy's root, and `src/frontend-proxy/envoy.tmpl.yaml` has tracing on with `spawn_upstream_span: true`, so each check becomes a full trace through the proxy and the Next.js frontend. `src/shipping/src/main.rs` wraps its actix app in `RequestTracing` and registers a `/health` route, so any HTTP probe there is traced too. The Go services already show the other half of the picture: `src/checkout/main.go` and `src/product-catalog/main.go` build their gRPC servers with `otelgrpc.WithFilter(filters.Not(filters.HealthCheck()))`, so `grpc_health_probe` calls produce nothing.

It then fixes each source where the fix is cheapest. In the nginx template it adds a `location /status` directive of `otel_trace off`, so the probe and the scrape are answered without a span while the image routes stay traced. In the shipping service it moves the `/health` route outside the `RequestTracing` middleware by splitting the app into a traced scope and a bare route, which keeps the quote and ship handlers instrumented and gives the probe nothing to record. For the Envoy check it changes nothing in Envoy: the request is a real GET and the frontend cannot tell it from a user, so the agent gives the `http_check` receiver a header the Collector can recognise and drops those traces in the pipeline. In `src/otel-collector/otelcol-config.yml` it adds a `filter/healthchecks` processor with OTTL span conditions matching that header's user agent attribute, `url.path` of `/status` or `/health`, and `rpc.service` of `grpc.health.v1.Health` as a backstop for any service whose SDK filter is missing, with `error_mode: ignore`, and places it in the traces pipeline before `span_metrics`, so the RED metrics the connector derives stop counting probes as traffic.

The agent is careful about what a filter does to a trace. A dropped span orphans its children, so the processor matches only root probe spans and the Envoy-originated check, never a database span under a real request, and the pull request shows the condition list with the reason for each. It leaves `compose.yaml` probes alone, since the health checks themselves are correct; only their telemetry changes.

It proves the config and the code before opening anything. It runs `otelcol-contrib validate --config` with the base file and `otelcol-config-extras.yml` layered the way `compose.yaml` layers them, with the `otelcol-contrib` `0.160.0` release binary, the version of the image the `.env` pins, since the sandbox has no Docker, runs `cargo check` in `src/shipping` and the nginx template through `nginx -t` with the environment variables substituted, and adds a case to `test/telemetry/test_traces.py` that queries Jaeger for spans named `image-provider` with a `/status` path and expects none, next to the existing check that each service has at least one trace.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Stop health checks and synthetic probes from producing spans in this system. Read the compose file, every service's health check, each service's instrumentation setup and the OpenTelemetry Collector config first, and list every probe that creates a span with the file that proves it. Fix each at the SDK or server where possible: turn tracing off for the probe location in nginx, keep the health route outside the tracing middleware in the Rust service, and use the gRPC health filter already used by the Go services as the model. For probes that cannot be told apart at the source, add a filter processor to the Collector traces pipeline before the span metrics connector, matching only root probe spans by path, service or a header you add to the Collector's own HTTP check. Validate the Collector config with the otelcol-contrib release binary at the version of the pinned collector image, compile or lint each changed service, add a telemetry test asserting no probe spans reach Jaeger, and open a pull request with the probe table and the command output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request that changes the nginx template, the shipping service's router setup, the Collector config and the telemetry tests. Its description holds a table of every probe: the file that defines it, the file that instruments it, where the fix went and why, with the Go services listed as already correct so a reviewer sees the whole picture.

The run's record keeps the Collector validation with the pinned release, the `cargo check` and `nginx -t` output, and the new test's assertion, so a reviewer can read the exact OTTL conditions and what they were checked against instead of trusting a summary.

Before merging, a reviewer starts the stack, lets the probes run, and checks Jaeger for the `image-provider`, `shipping` and `frontend-proxy` services to confirm only user traffic remains, then watches the span metrics dashboard to see request rates fall by the probe volume and nothing else.

## Run it on every observability repository

An SRE hands this off from the terminal once per repository that carries a Collector config: one `POST /o/{org}/sessions` with this body and the repository in `resources`, and the key's `webhook_url` reports the finished run with its pull request. A retry sends the same `request_id`, a UUID, so a second request returns the first run.

Each run reads that repository's own probes and instrumentation, so a system whose services are probed over gRPC gets SDK filters, and one whose Collector scrapes an HTTP endpoint gets the receiver header and the pipeline filter. The table is rebuilt every time from the files.

## Why run it on Rush?

Health check spans are a tracing problem spread across every language in the system: an nginx directive, a Rust middleware, a Go filter, a Collector pipeline. Finding them means reading each service's health check and its instrumentation together, and the fix in one place can orphan spans or distort metrics in another. The agent does that reading in its own sandbox and validates every changed file with the tool that owns it.

On Rush the validation output, the compile results and the new test sit in the run's record next to the pull request, and the same request can be sent to every repository that ships a Collector config, each run producing its own probe table rather than one engineer's memory of where the noise comes from.

## Frequently asked questions

### Why not filter everything in the Collector?

Because spans the SDK never creates cost nothing to export, and a Collector filter that matches by path can catch a real request to a similar path. The agent filters at the source where the server or SDK supports it and uses the Collector only for probes it cannot tell apart earlier, such as the Collector's own HTTP check.

### Will dropping spans break my traces?

Dropping a parent span orphans its children, so the processor matches only spans that start a probe, such as the Envoy entry span for the Collector's check or a root span on the status path. The pull request lists each condition and the reason it cannot match a user request.

### What happens to the span metrics?

They improve. The filter sits before the `span_metrics` connector in the traces pipeline, so request counts and latency histograms stop including probe calls that return in microseconds and were pulling the averages down.

### Does it change the health checks themselves?

No. The probes in the compose file and the health services in each process stay as they are. Only the telemetry they produce changes, and the Go services, which already filter gRPC health calls, are left untouched and cited as the pattern.

## More OpenTelemetry jobs

- [Investigate a Microservices Outage from OpenTelemetry Data](https://getrush.ai/run/investigate-outage-with-grafana/opentelemetry/)
- [Investigate a Latency Spike from Traces](https://getrush.ai/run/investigate-trace-latency/opentelemetry/)
- [Write Tests for the OpenTelemetry Spans a Node.js Service Emits](https://getrush.ai/run/write-tracing-tests/opentelemetry/)

## More jobs like this: Hand off a long task

- [Enable TypeScript Strict Mode in a discord.js Bot](https://getrush.ai/run/enable-strict-mode/discord-js/)
- [Enable mypy Strict Mode in an Existing Python Library](https://getrush.ai/run/enable-strict-mode/mypy/)
- [Turn On TypeScript Strict Mode and Fix the Errors](https://getrush.ai/run/enable-strict-mode/typescript/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
