All jobs

Add Slack Alerts on Failed Runs to a Dagster Project

Yes. An agent adds a run failure sensor that posts the job, run and error to Slack, routes by the alert_team tag, tests it with mocked requests and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and reads how the Dagster code is organised before adding anything. In a workspace like dagster-io/hooli-data-eng-pipelines, dg.toml lists six projects, dagster_cloud.yaml maps each to a Dagster+ code location built from its src/ directory, and .github/workflows/deploy-dagster-cloud.yml deploys a branch deployment per pull request. The main project, hooli-data-eng, loads its definitions with load_defs(hooli_data_eng.defs) in definitions.py, so every subpackage under defs/ with its own definitions.py is picked up: pandas, dbt, files, databricks, kubernetes and the rest. defs/pandas/schedules.py already defines predict_job with tags={"alert_team": "ml"}, an asset_sensor and a freshness-check schedule, and defs/files/sensors.py shows the house style for a sensor that reads the environment through utils.get_env(), which returns PROD, BRANCH or LOCAL from DAGSTER_CLOUD_DEPLOYMENT_NAME and DAGSTER_IS_DEV_CLI. pyproject.toml depends on requests and responses but not on dagster-slack, and nothing in the repository mentions Slack.

It adds a new defs/alerts/ package in the same shape as its neighbours: sensors.py and definitions.py. The sensor is a @dg.run_failure_sensor with monitor_all_code_locations=True, so one definition in this location covers failures in basics, batch_enrichment, hooli_data_ingest and the other locations too, and default_status=dg.DefaultSensorStatus.RUNNING so it is on after deploy without a click in the UI. Its body reads the run's job name, run id and the failure event's message from the sensor context, reads the alert_team tag the repository already puts on predict_job, and posts a short message through requests to a Slack incoming webhook. The webhook URL comes from an environment variable, SLACK_ALERTS_WEBHOOK_URL, read the way resources.py reads SNOWFLAKE_ACCOUNT, and a per-team mapping lets the ml tag reach its own channel while untagged jobs reach the default. When get_env() returns LOCAL, the sensor logs the message instead of posting it, so dagster dev on a laptop never pages a channel.

The message carries what an on-call engineer needs to act: job name, run id, the deployment name from DAGSTER_CLOUD_DEPLOYMENT_NAME, the first line of the error, and a link to the run built from the deployment URL, with the team tag as a prefix so Slack search finds a team's failures. The agent does not add dagster-slack: the repository already has requests, an incoming webhook needs no bot token or scopes, and one fewer package in a pyproject.toml that already lists more than twenty keeps the uv lock change to nothing. It documents the alternative, Dagster+ alert policies configured outside the code, in the pull request and says why a sensor in the repository was chosen: it is versioned, tested and visible in the same place as the jobs it watches.

Then it proves the sensor works without a Slack workspace. A new hooli_data_eng_tests/test_alerts.py builds a context with dg.build_run_status_sensor_context and an ephemeral DagsterInstance, uses the responses library the project already depends on to intercept the webhook call, and asserts on the posted JSON: job name, run id and the routed channel for a run tagged alert_team: ml and for one with no tag. The existing test_repo_loads.py pattern gets one more assertion that the definitions load with the sensor present. In the sandbox the agent runs make dependencies, pytest hooli-data-eng/hooli_data_eng_tests, ruff check as the deploy workflow does, and dg check defs from the dev extras to confirm the location still loads, and pastes the output into the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Slack alerts on failed runs to this Dagster project. Read dg.toml, dagster_cloud.yaml, pyproject.toml, definitions.py, every defs subpackage's definitions.py, the existing sensors and schedules, utils, resources.py and the tests first, and note any alert-related tag the jobs already carry. Add a defs/alerts package with a run_failure_sensor that monitors all code locations, is running by default, reads the job name, run id, deployment name and failure message from the context, routes by the alert_team tag to a per-team channel with a default, posts through requests to a Slack incoming webhook read from the SLACK_ALERTS_WEBHOOK_URL environment variable, and only logs when the environment is LOCAL. Do not add dagster-slack. Add tests that build a run status sensor context, mock the webhook with the responses library and assert on the payload for a tagged and an untagged run, and extend the repo-loads test. Add a README note on setting the variable in Dagster+. Run the dependency install, pytest, ruff check and dg check defs in your sandbox, and open a pull request with the output and a note on Dagster+ alert policies as the alternative.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request adding hooli-data-eng/src/hooli_data_eng/defs/alerts/sensors.py and definitions.py, a test_alerts.py with the mocked webhook assertions, one more line in test_repo_loads.py, and a README paragraph on where SLACK_ALERTS_WEBHOOK_URL is set for each Dagster+ location. No dependency changes, so uv.lock is untouched. The description shows the exact JSON the sensor posts for a tagged and an untagged failure.

The run's record keeps the pytest, ruff check and dg check defs output from the sandbox, so a reviewer sees the definitions load with the new sensor and the payload assertions pass before the branch deployment the deploy workflow will create for the pull request.

Before merging, a reviewer creates the incoming webhook in Slack, sets the variable on the production location in Dagster+, picks the channel names for the team mapping, and decides whether a failure in a branch deployment should post to the same channel or be silenced by treating BRANCH like LOCAL.

Run it on every Linear issue labelled data-platform

A data team that tracks platform work in Linear starts one run per issue from an automation with a linear trigger on the data-platform label, in the Project that holds the Dagster repository. The trigger adds the issue after this prompt, so an issue that asks for a second channel, alerts on run_status_sensor success for a critical job, or a message that includes the partition key is what the agent reads.

Each run checks out the workspace in its own sandbox, runs the tests and dg check defs, and opens its own pull request linking the issue, which the deploy workflow turns into a branch deployment a reviewer can watch. An automation run has its own max_runs_per_hour cap.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

A failure sensor is twenty lines of Python and an afternoon of questions: which locations it covers, how the environment is detected so laptops stay quiet, where the tag the jobs already carry can route a message, and how to test a Slack call without Slack. The agent answers each from the repository's own conventions, get_env(), the alert_team tag, the responses dependency, and leaves the proof in the pull request as test output rather than a screenshot of a channel.

The same request runs across every Dagster workspace an organisation deploys, each in its own sandbox with its own definitions.py read first, so a dozen teams get the same alert shape with their own routing and their own tests, and the record of every command is kept alongside the pull request.

Frequently asked questions

Why not use Dagster+ alert policies?

They work, and the pull request says so. A sensor in the repository is versioned, reviewed and tested with the jobs it watches, runs the same way under dagster dev, and can read run tags for routing. Alert policies live outside the code and are configured per deployment; a team can use both.

Does it need dagster-slack or a bot token?

No. An incoming webhook accepts a JSON post with no token, and the project already depends on requests. dagster-slack is the right choice when the message needs a bot identity, threads or channel lookup; for a failure notice to a fixed channel it adds a dependency for nothing.

Will it alert on every code location?

Yes. monitor_all_code_locations=True makes one sensor in the data-eng-pipeline location watch failures in every location in the deployment, so the other five projects need no code change. A team that wants per-location routing can add a location name to the message, which the context provides.

What stops it posting from my laptop?

The same get_env() helper the rest of the project uses. When DAGSTER_IS_DEV_CLI is set the sensor logs the message and returns, so dagster dev never reaches Slack. Branch deployments count as BRANCH and post by default; a reviewer can change that to log-only in one line.

How is it tested without a Slack workspace?

With dg.build_run_status_sensor_context and an ephemeral instance to construct a failure, and the responses library to intercept the webhook post and capture its body. The tests assert on the job name, run id and channel in the JSON, so a change to the message format shows up as a failing test.

More Dagster jobs

The same job on other stacks

More jobs like this: Turn a Linear label into a PR

Keep reading