# Add Tail-Based Sampling to an OpenTelemetry Collector Pipeline

Yes. An agent adds tail_sampling to your Collector config so errors and slow traces are kept, leaves span metrics unsampled, validates it and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-tail-sampling/opentelemetry/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and reads the Collector configuration in full before adding a processor. In a system like open-telemetry/opentelemetry-demo, the base file `src/otel-collector/otelcol-config.yml` has one `traces` pipeline: receiver `otlp`, processors `resource_detection`, `memory_limiter`, `transform/sanitize_spans`, `gen_ai_normalizer`, `transform/redact_sensitive_data` and `redaction`, and exporters `debug` and `span_metrics`, which is a connector that turns every span into request, error and duration metrics for the `metrics` pipeline. Three more files layer on top through repeated `--config` flags in `compose.yaml` and `compose.observability.yaml`: `otelcol-config-observability.yml` sets the traces exporters to `debug`, `otlp_grpc/jaeger` and `span_metrics`, and both it and `otelcol-config-extras.yml` warn in their comments that the Collector replaces arrays when merging, so a pipeline's exporter list must be repeated in every layer that touches it. There is no sampling anywhere.

That connector decides the design. A `tail_sampling` processor placed in the existing pipeline would also sample what reaches `span_metrics`, and the rate and error metrics would be computed from a biased subset. So the agent splits the pipeline with the `forward` connector: `traces` keeps its receiver and processors and exports to `span_metrics` and `forward`, and a new `traces/sampled` pipeline receives from `forward`, runs `tail_sampling` and then `batch`, and exports to `debug`. In the observability layer, `traces/sampled` exports to `debug` and `otlp_grpc/jaeger` while `traces` is repeated unchanged, and the comment blocks in both layered files that list the upstream exporter names are updated so a fork following them does not reattach Jaeger to the unsampled pipeline.

The policies come from what the demo already marks. `compose.yaml` sets `service.criticality` in `OTEL_RESOURCE_ATTRIBUTES` per service, `critical` for `checkout` and `high` for `cart` among others, so one `string_attribute` policy keeps every trace that touches a `critical` service. A `status_code` policy keeps traces with an `ERROR` span, a `latency` policy keeps traces over a threshold the reviewer sets, and a `probabilistic` policy keeps a share of the rest, with `decision_wait` and `num_traces` chosen against the `400M` memory limit and `GOMEMLIMIT=160MiB` the compose file gives the Collector, since tail sampling holds every open trace in memory until its decision. Each value is a named key in the file with a comment, so the reviewer changes a number rather than reads a processor manual.

The agent validates without Docker. It reads `COLLECTOR_CONTRIB_IMAGE` in `.env` to learn the contrib version in use, downloads that version's `otelcol-contrib` binary, exports the environment variables the config references, and runs `otelcol-contrib validate` with the same ordered `--config` flags each compose file passes, once for the base pair and once for the full observability stack. It runs `make yamllint`, which applies the repository's `.yamllint` rules, and reads `test/telemetry/test_traces.py`, which polls Jaeger until each service has at least one trace, and notes in the pull request that the `critical` policy and the error policy keep the checkout path those tests drive, while a service that only ever emits fast, successful spans may need its own policy or a higher probability. Following `CONTRIBUTING.md`, it adds a `.chloggen` fragment from `TEMPLATE.yaml` with `change_type: enhancement` and `component: collector`, and runs `make chlog-validate`.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add tail-based sampling to this repository's OpenTelemetry Collector configuration. Read every collector config file, how the compose files layer them, and the environment the collector gets first. Keep the span metrics connector fed with unsampled spans by splitting traces into a pipeline that feeds span metrics and a forward connector, and a sampled pipeline that runs tail_sampling and batch before the trace exporters; repeat exporter lists in every layered file that touches those pipelines and update their comments. Use policies for error status, latency, a resource attribute the services already set for criticality, and a probabilistic remainder, with every threshold a commented key. Size decision_wait and num_traces against the collector's memory limit. Validate with the contrib collector binary at the version the repository pins, run the repository's YAML lint and add its changelog fragment. Do not run the demo. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch that changes `src/otel-collector/otelcol-config.yml`, `src/otel-collector/otelcol-config-observability.yml`, the comment block in `src/otel-collector/otelcol-config-extras.yml`, and adds one fragment under `.chloggen/`. The description shows the two `otelcol-contrib validate` runs with the exact `--config` order from each compose file, the `yamllint` and `chlog-validate` results, and a table of the policies with their keys and the values chosen, with the memory reasoning for `num_traces` written out.

The run's record keeps the binary download at the pinned version and every validation with its output, so a reviewer can see the configuration was accepted by the same Collector build the compose files run, not by a different version's schema.

Before merging, a reviewer sets the latency threshold and the probability for their traffic, decides whether `cart` and other `high` services should join the always-keep policy, and starts the stack once with `compose.observability.yaml` to confirm Jaeger shows the sampled traces and Prometheus still shows unchanged request rates from `span_metrics`, the one check that needs the running demo.

## Run it on every Collector repository

Start this once from a terminal with `POST /o/{org}/sessions`, this body and the repository in `resources`. The run keeps going in its sandbox after you close the window, and a follow-up through `POST /o/{org}/sessions/{id}/messages` can tune a threshold, add a policy for a new attribute or apply the same split to a Kubernetes values file, in the same sandbox with the binary already downloaded.

A platform team with a Collector configuration per environment starts one run per repository from a script with the same body, each with its own `request_id`, a UUID, so a retried start returns the run it already began; the key's `webhook_url` receives `run.completed` for each with a `session_url` from which the handler reads the pull request link.

## Why run it on Rush?

The common mistake with tail sampling is putting it in front of everything, so the metrics derived from spans quietly become metrics derived from a sample. The agent reads where the span metrics connector sits, splits the pipeline so it keeps seeing every span, and validates the layered result with the Collector build the repository pins, which is the part a quick edit on a laptop skips.

The sandbox downloads one binary and runs it against the config, leaves nothing behind, and keeps every validation in the run's record. Several Collector repositories can be done at once, each with its own pull request and its own validation output.

## Frequently asked questions

### Will my RED metrics change?

No. The `span_metrics` connector stays on the unsampled pipeline and sees every span; only what continues to the trace exporters is sampled. The pull request's reviewer check is to compare request rates in Prometheus before and after, which should match.

### How are errors and slow traces kept?

A `status_code` policy keeps any trace with an `ERROR` span, a `latency` policy keeps traces over the threshold you set, and a `string_attribute` policy keeps every trace that touches a service marked `critical` in its resource attributes, which the demo already sets per service. The probabilistic policy applies to the rest.

### Why does the config repeat exporter lists in several files?

The Collector merges layered configuration files but replaces arrays rather than appending, which the repository's own comments warn about. A layer that changes a pipeline's exporters must list all of them, so the agent updates every layer that names the traces pipelines and the comments that document the names.

### Does the agent run the demo?

No. The prompt says not to, the sandbox has no Docker, and `otelcol-contrib validate` checks the configuration without starting any service. Seeing sampled traces in Jaeger is the one step left for the reviewer with the stack running.

### What about the memory limit?

Tail sampling holds every trace in memory until `decision_wait` passes, so `num_traces` is sized against the `400M` limit and `GOMEMLIMIT` the compose file gives the Collector, and `memory_limiter` stays first in the pipeline. The reasoning is in the pull request so a different deployment can resize it.

## More OpenTelemetry jobs

- [Drop Health Check Spans from OpenTelemetry Traces](https://getrush.ai/run/drop-health-check-spans/opentelemetry/)
- [Investigate a Microservices Outage from OpenTelemetry Data](https://getrush.ai/run/investigate-outage-with-grafana/opentelemetry/)
- [Investigate a Latency Spike from Traces](https://getrush.ai/run/investigate-trace-latency/opentelemetry/)

## More jobs like this: Hand off a long task

- [Group Generated Tasks into TaskGroups in an Airflow DAG](https://getrush.ai/run/add-task-groups/airflow/)
- [Add Telemetry to a VS Code Extension](https://getrush.ai/run/add-telemetry/vscode-extension/)
- [Add Tenant Isolation to a Flux Repository](https://getrush.ai/run/add-tenant-isolation/flux/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
