All jobs

Add Prometheus Metrics to a Tokio Server

Yes. An agent adds the metrics and metrics-exporter-prometheus crates to a Tokio server, counts connections and commands, serves /metrics on a flag, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads the server's structure before it counts anything. In a Tokio project like tokio-rs/mini-redis, Cargo.toml depends on tokio 1 with features = ["full"], tokio-stream, bytes, clap 4 with derive, tracing and tracing-subscriber with env-filter, plus an optional otel feature that pulls opentelemetry, tracing-opentelemetry, opentelemetry-aws and opentelemetry-otlp. Two binaries are declared: mini-redis-cli from src/bin/cli.rs and mini-redis-server from src/bin/server.rs, whose Cli struct has one --port flag and whose main binds 127.0.0.1:{port} and calls server::run(listener, signal::ctrl_c()). src/server.rs holds a Listener with a Semaphore of MAX_CONNECTIONS (250) permits, an accept loop with exponential backoff up to 64 seconds, and a Handler::run loop that reads a frame, parses it with Command::from_frame and calls cmd.apply. src/cmd/mod.rs has the Command enum (Get, Publish, Set, Subscribe, Unsubscribe, Ping, Unknown) with a get_name() method, and src/db.rs keeps entries with expirations purged by a background task. The traces exist; there is no counter, gauge or /metrics anywhere, and .github/workflows/ci.yml runs cargo build, cargo test, both again with --features otel, and cargo fmt --all --check on Ubuntu, macOS and Windows.

The agent adds the metrics facade crate and metrics-exporter-prometheus to [dependencies], versions taken from the registry at the time of the run and pinned in Cargo.lock. A new src/metrics.rs exposes install(addr: SocketAddr), which builds PrometheusBuilder::new().with_http_listener(addr).install() so the exporter serves GET /metrics on its own Tokio task, and src/bin/server.rs gains --metrics-port: Option<u16> on Cli; with no flag no listener is opened and the macros write to the no-op recorder, so the default behaviour of mini-redis-server does not change. The instrumentation goes where the state already changes hands: counter!("mini_redis_connections_accepted_total").increment(1) after self.accept() in Listener::run, gauge!("mini_redis_connections_active") incremented there and decremented in the spawned task right where drop(permit) returns the semaphore slot, counter!("mini_redis_connection_errors_total") in the branch that logs connection error, and in Handler::run a counter!("mini_redis_commands_total", "command" => cmd.get_name()) plus a histogram!("mini_redis_command_duration_seconds", "command" => …) timed with Instant::now() around cmd.apply. Because get_name() is a &'static str for every variant including unknown, the label set is bounded.

A key count needs one new method: Db::len() on src/db.rs, which locks the state and returns entries.len(), called from a small Tokio interval task that sets gauge!("mini_redis_db_keys") every few seconds rather than on every command, so the mutex is not touched in the hot path. The agent keeps MAX_CONNECTIONS as a constant and exports it once as mini_redis_connections_limit so a dashboard can draw active against limit.

Tests avoid opening a port. tests/server.rs already starts server::run on an ephemeral listener and talks to it with raw TcpStream writes, so the agent adds a test that installs a recorder with PrometheusBuilder::new().build_recorder() through a std::sync::Once, sends a SET and a GET, and asserts that handle.render() contains mini_redis_commands_total{command="set"} 1 and the get line, and that the active-connections gauge returns to 0 after the client drops. The agent runs cargo build, cargo build --features otel, cargo test, cargo test --features otel and cargo fmt --all --check as the workflow does, then starts mini-redis-server --metrics-port 9000, runs the CLI against it and shows a curl localhost:9000/metrics excerpt in the description. README.md gets a short section on the flag and the metric names; examples/ and the otel feature are untouched.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Prometheus metrics to this Tokio server. Read Cargo.toml, the server binary, the server module, the command module and the CI workflow first. Use the metrics facade crate with metrics-exporter-prometheus, behind an optional --metrics-port flag on the server binary; with no flag, open no listener and change nothing about default behaviour. Instrument accepted connections, active connections as a gauge paired with the semaphore permit, connection errors, commands by name using the existing command name method, command duration as a histogram, and a periodic key-count gauge from a new len method on the database, read on an interval rather than per command. Add a test that installs a recorder in-process, drives a set and a get through the existing test harness and asserts on the rendered text. Run the same cargo build, test and fmt commands the CI workflow runs, with and without the otel feature, start the server with the flag and include a /metrics excerpt. Open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request that changes Cargo.toml and Cargo.lock (two crates), src/bin/server.rs (the flag and the install call), src/server.rs (the counters, gauge and histogram at the accept loop and the handler loop), src/db.rs (len), src/lib.rs (the new module) and README.md, and adds src/metrics.rs and a test in tests/server.rs.

The description lists every metric name with its labels and the line of code that drives it, quotes the cargo test summaries with and without otel, the cargo fmt result and a /metrics excerpt from a running server. The commands and their full output are in the transcript at GET /o/{org}/sessions/{id}/messages.

Before merging, a reviewer checks the histogram buckets against the latencies they expect, decides whether the exporter should bind 127.0.0.1 or 0.0.0.0, and whether the key-count interval is right for their key volume. A follow-up message to the same session can add a tokio runtime metrics gauge or a Grafana dashboard JSON for the names above.

Run it on every Rust service with tracing but no metrics

A team with several Tokio services starts one run per repository with one POST /o/{org}/sessions each, the repository in resources and this prompt as input. Each run reads its own accept loop and request types, so an HTTP service built on hyper gets counters by route and status rather than by Redis command.

Send a request_id, a UUID v5 of the repository's full name, so a retried call returns the run it already started. The key's webhook_url reports when each run the key starts finishes, with the run's final answer in result.

The use case this job follows, with its automation and trigger: Hand off a long task.

Why run it on Rush?

Instrumenting an async server means finding the few places where a connection or a command changes state and touching them without holding a lock or blocking the loop. The agent reads the semaphore, the spawn and the handler loop, puts each macro next to the state it measures, and proves the numbers with a test that renders the registry in-process, all in a sandbox where it can build with and without the otel feature as the CI does.

The pull request carries the metric names, the code behind each one and the rendered output, so review is reading a diff rather than wiring an exporter locally and guessing which label sets are bounded. Several services can be done at once, each in its own sandbox with its own pull request.

Frequently asked questions

Does this change the default server?

No. Without --metrics-port no exporter is installed and no socket is opened; the metrics macros write to a no-op recorder. The DEFAULT_PORT of 6379 and the --port flag are untouched.

Why the metrics crate rather than OpenTelemetry metrics?

The repository's otel feature is optional and covers traces only, and the CI builds with and without it. The metrics facade keeps the counters in the default build with no feature gate and a plain Prometheus endpoint. If you already run an OTel collector, say so and the agent uses the OpenTelemetry SDK's meter behind the otel feature instead.

Will command labels explode in cardinality?

No. The label value is the existing get_name() of the Command enum, one of seven fixed strings including unknown, so there is one series per command type. Keys and client addresses are never used as labels.

Can the test run without a port?

Yes. The test builds a recorder with build_recorder() and reads handle.render() in-process; only the existing test harness opens an ephemeral listener for the server itself. Installing the recorder through a Once keeps the test file correct when cargo test runs its tests in parallel.

More Tokio jobs

The same job on other stacks

Keep reading