What the agent does
The agent clones the repository into its own cloud sandbox and reads the Collector configuration in full before adding a processor. In a system like open-telemetry/opentelemetry-demo, the base file src/otel-collector/otelcol-config.yml has one traces pipeline: receiver otlp, processors resource_detection, memory_limiter, transform/sanitize_spans, gen_ai_normalizer, transform/redact_sensitive_data and redaction, and exporters debug and span_metrics, which is a connector that turns every span into request, error and duration metrics for the metrics pipeline. Three more files layer on top through repeated --config flags in compose.yaml and compose.observability.yaml: otelcol-config-observability.yml sets the traces exporters to debug, otlp_grpc/jaeger and span_metrics, and both it and otelcol-config-extras.yml warn in their comments that the Collector replaces arrays when merging, so a pipeline's exporter list must be repeated in every layer that touches it. There is no sampling anywhere.
That connector decides the design. A tail_sampling processor placed in the existing pipeline would also sample what reaches span_metrics, and the rate and error metrics would be computed from a biased subset. So the agent splits the pipeline with the forward connector: traces keeps its receiver and processors and exports to span_metrics and forward, and a new traces/sampled pipeline receives from forward, runs tail_sampling and then batch, and exports to debug. In the observability layer, traces/sampled exports to debug and otlp_grpc/jaeger while traces is repeated unchanged, and the comment blocks in both layered files that list the upstream exporter names are updated so a fork following them does not reattach Jaeger to the unsampled pipeline.
The policies come from what the demo already marks. compose.yaml sets service.criticality in OTEL_RESOURCE_ATTRIBUTES per service, critical for checkout and high for cart among others, so one string_attribute policy keeps every trace that touches a critical service. A status_code policy keeps traces with an ERROR span, a latency policy keeps traces over a threshold the reviewer sets, and a probabilistic policy keeps a share of the rest, with decision_wait and num_traces chosen against the 400M memory limit and GOMEMLIMIT=160MiB the compose file gives the Collector, since tail sampling holds every open trace in memory until its decision. Each value is a named key in the file with a comment, so the reviewer changes a number rather than reads a processor manual.
The agent validates without Docker. It reads COLLECTOR_CONTRIB_IMAGE in .env to learn the contrib version in use, downloads that version's otelcol-contrib binary, exports the environment variables the config references, and runs otelcol-contrib validate with the same ordered --config flags each compose file passes, once for the base pair and once for the full observability stack. It runs make yamllint, which applies the repository's .yamllint rules, and reads test/telemetry/test_traces.py, which polls Jaeger until each service has at least one trace, and notes in the pull request that the critical policy and the error policy keep the checkout path those tests drive, while a service that only ever emits fast, successful spans may need its own policy or a higher probability. Following CONTRIBUTING.md, it adds a .chloggen fragment from TEMPLATE.yaml with change_type: enhancement and component: collector, and runs make chlog-validate.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Add tail-based sampling to this repository's OpenTelemetry Collector configuration. Read every collector config file, how the compose files layer them, and the environment the collector gets first. Keep the span metrics connector fed with unsampled spans by splitting traces into a pipeline that feeds span metrics and a forward connector, and a sampled pipeline that runs tail_sampling and batch before the trace exporters; repeat exporter lists in every layered file that touches those pipelines and update their comments. Use policies for error status, latency, a resource attribute the services already set for criticality, and a probabilistic remainder, with every threshold a commented key. Size decision_wait and num_traces against the collector's memory limit. Validate with the contrib collector binary at the version the repository pins, run the repository's YAML lint and add its changelog fragment. Do not run the demo. Open a pull request.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
What comes back is a pull request on its own branch that changes src/otel-collector/otelcol-config.yml, src/otel-collector/otelcol-config-observability.yml, the comment block in src/otel-collector/otelcol-config-extras.yml, and adds one fragment under .chloggen/. The description shows the two otelcol-contrib validate runs with the exact --config order from each compose file, the yamllint and chlog-validate results, and a table of the policies with their keys and the values chosen, with the memory reasoning for num_traces written out.
The run's record keeps the binary download at the pinned version and every validation with its output, so a reviewer can see the configuration was accepted by the same Collector build the compose files run, not by a different version's schema.
Before merging, a reviewer sets the latency threshold and the probability for their traffic, decides whether cart and other high services should join the always-keep policy, and starts the stack once with compose.observability.yaml to confirm Jaeger shows the sampled traces and Prometheus still shows unchanged request rates from span_metrics, the one check that needs the running demo.
Run it on every Collector repository
Start this once from a terminal with POST /o/{org}/sessions, this body and the repository in resources. The run keeps going in its sandbox after you close the window, and a follow-up through POST /o/{org}/sessions/{id}/messages can tune a threshold, add a policy for a new attribute or apply the same split to a Kubernetes values file, in the same sandbox with the binary already downloaded.
A platform team with a Collector configuration per environment starts one run per repository from a script with the same body, each with its own request_id, a UUID, so a retried start returns the run it already began; the key's webhook_url receives run.completed for each with a session_url from which the handler reads the pull request link.
The use case this job follows, with its automation and trigger: Hand off a long task.
Why run it on Rush?
The common mistake with tail sampling is putting it in front of everything, so the metrics derived from spans quietly become metrics derived from a sample. The agent reads where the span metrics connector sits, splits the pipeline so it keeps seeing every span, and validates the layered result with the Collector build the repository pins, which is the part a quick edit on a laptop skips.
The sandbox downloads one binary and runs it against the config, leaves nothing behind, and keeps every validation in the run's record. Several Collector repositories can be done at once, each with its own pull request and its own validation output.
Frequently asked questions
Will my RED metrics change?
No. The span_metrics connector stays on the unsampled pipeline and sees every span; only what continues to the trace exporters is sampled. The pull request's reviewer check is to compare request rates in Prometheus before and after, which should match.
How are errors and slow traces kept?
A status_code policy keeps any trace with an ERROR span, a latency policy keeps traces over the threshold you set, and a string_attribute policy keeps every trace that touches a service marked critical in its resource attributes, which the demo already sets per service. The probabilistic policy applies to the rest.
Why does the config repeat exporter lists in several files?
The Collector merges layered configuration files but replaces arrays rather than appending, which the repository's own comments warn about. A layer that changes a pipeline's exporters must list all of them, so the agent updates every layer that names the traces pipelines and the comments that document the names.
Does the agent run the demo?
No. The prompt says not to, the sandbox has no Docker, and otelcol-contrib validate checks the configuration without starting any service. Seeing sampled traces in Jaeger is the one step left for the reviewer with the stack running.
What about the memory limit?
Tail sampling holds every trace in memory until decision_wait passes, so num_traces is sized against the 400M limit and GOMEMLIMIT the compose file gives the Collector, and memory_limiter stays first in the pipeline. The reasoning is in the pull request so a different deployment can resize it.
More OpenTelemetry jobs
More jobs like this: Hand off a long task
- Group Generated Tasks into TaskGroups in an Airflow DAG (Apache Airflow)
- Add Telemetry to a VS Code Extension (VS Code extension)
- Add Tenant Isolation to a Flux Repository (Flux)