# Summarize PostHog Experiment Results

Yes. An agent reads an experiment's variants and goal metrics through PostHog and returns results.md with the winner, effect size, significance and caveats.

[Run this on Rush](https://cloud.getrush.ai/run/experiment-results/posthog/).

## What the agent does

The run uses the PostHog connection, not a git repository. After the org admin connects PostHog (OAuth to `https://mcp.posthog.com/mcp`), the request names that connection. The agent lists experiments in the project, picks the one that is running (or the one the prompt names), and reads its variants, exposure or enrollment counts and the goal metric definitions PostHog stores for it.

For each variant it pulls the conversion (or other goal) numbers PostHog already computed: exposures, conversions, conversion rate and the difference from control. It records the significance or credible-interval summary PostHog reports for that experiment rather than inventing a p-value of its own. If PostHog marks the experiment as needing more sample, or if a variant's exposure is far below the others, that becomes a sample-size caveat in the report.

When the project has several experiments, the agent lists their keys and statuses before choosing. A draft or archived experiment is skipped unless the prompt names it. Feature-flag payloads and multivariate variants are reported with the same exposure and conversion fields PostHog returns for each variant key, so a three-variant test gets three rows rather than a collapsed treatment bucket.

It then writes a decision memo. The recommendation paragraph states whether to ship the treatment, keep running, or stop without shipping, grounded only in the numbers and flags PostHog returned. Caveats cover peeking (if the experiment is still running), multiple goals when more than one is attached, and any feature-flag rollout percentage that is not 100% of the exposed population. The agent does not stop the experiment, change flag rollouts, or create insights.

Output is a single file, `results.md`, left in the sandbox and published as a session artifact. No events are written back into PostHog.

## Start it with one API call

`POST https://api.prix.dev/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Use the PostHog connection to read the running experiment in this project: its variants, exposure counts and goal metrics. Write results.md with each variant's conversion, the difference from control, the significance PostHog reports, sample-size caveats and a one-paragraph recommendation. Read only; do not stop or ship the experiment.",
  "connections": [
    "posthog"
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.prix.dev/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is `results.md` as an artifact. It names the experiment and its feature flag, lists each variant with exposures, conversions, rate and lift versus control, quotes the significance PostHog reports, lists sample-size and peeking caveats, and ends with one recommendation paragraph.

A product manager can paste the memo into a launch doc; an engineer can open the same experiment in PostHog and match the numbers to the transcript. Before shipping, a reviewer confirms the goal metric is the one the team agreed on and that a still-running experiment is not treated as final.

## Run it on every experiment nearing its planned end date

When your experiment tracker marks an experiment as due for a decision, your server sends this request with the experiment name in the prompt. The agent reads the live PostHog stats that day and returns a memo without anyone exporting CSVs.

A webhook on the API key delivers `run.completed` with the session URL; store the artifact next to the experiment ticket so the decision and the numbers stay linked.

## Why run it on Rush?

PostHog's experiment screen is built for exploration; a decision memo is what a launch review needs. The agent pulls the variant stats through the connection and writes that memo once, with the queries kept in the transcript.

It does this in its own sandbox without stopping the experiment or changing flags. Several experiments can be summarized in parallel, each as its own session and artifact.

Product managers often export a CSV, paste numbers into a doc and lose the link to how PostHog defined significance. This run keeps the connection calls and the memo together on one session, so a later question about a caveat can be answered from the same transcript.

## Frequently asked questions

### Will it stop or ship the experiment?

No. The prompt says read only and forbids stopping or shipping. The recommendation is prose in `results.md`; a person applies it in PostHog.

### How do I point it at one experiment?

Name the experiment (or feature flag key) in the prompt. Without a name it picks the running experiment it finds; if several are running it lists them and stops until you say which one.

### Does it compute its own statistics?

No. It reports the significance and rates PostHog already calculated. Sample-size caveats describe what PostHog shows (for example not enough exposures), not a new power analysis.

### Can it read a finished experiment?

Yes. Say so in the prompt. The memo then treats the result as final and drops the peeking caveat that applies to a still-running test.

### What if PostHog shows no significant result?

The memo says so and the recommendation is usually to keep running or to stop without shipping, depending on sample size and the planned end date. It does not invent significance the API did not report.