# Add Slack Alerts on Task Failure to Airflow DAGs

Yes. An agent adds one Slack failure callback to every DAG in your Airflow project through a shared default_args module, tests it with a DagBag load, and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-slack-failure-alerts/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent reads what each DAG does on failure today. In an Astro project like astronomer/airflow-example-dags, the `Dockerfile` is `FROM quay.io/astronomer/astro-runtime:6.0.0`, `requirements.txt` is empty, `packages.txt` adds `postgresql`, and `airflow_settings.yaml` holds empty placeholders for connections, pools and variables. Each of the five live DAGs under `dags/` declares its own `default_args` dictionary. `databricks_sample.py` sets `'email': ['EMAIL']` and `'email_on_failure': True`, a placeholder address that can never deliver. `s3_upload_dag.py`, `s3_upload_copy_dag.py`, `s3_upload_copy_delete.py` and `kube_pod.py` set `'email_on_failure': False` with `'retries': 1` and a five-minute `retry_delay`. `s3_upload_copy_dag.py` goes further: it defines an empty `on_failure_function(**kwargs)` whose docstring says to insert notification logic here, and passes it as `on_failure_callback` on each `PythonOperator` in its loop. The sixth file, `example_external_task_sensor.py`, is commented out entirely and is left alone.

It then adds the callback once, not five times. A new `include/callbacks.py` defines `notify_slack_failure(context)` that reads `dag_id`, `task_id`, `execution_date`, `try_number` and the log URL from the task instance in `context`, and posts a short message through `SlackWebhookHook` from `apache-airflow-providers-slack` against a connection id `slack_alerts`. That provider is not part of every runtime image, so the agent checks the image's Airflow install with `pip show` and, when it is missing, adds one pinned line to the empty `requirements.txt`, the file the Astro build already installs from. A `include/defaults.py` exposes a `DEFAULT_ARGS` dictionary with `on_failure_callback` set to that function, `retries` and `retry_delay` carried over from the DAGs that already set them, and `email_on_failure` off. Each DAG then builds its `default_args` from `DEFAULT_ARGS` with its own keys on top, so `databricks_sample.py` keeps `databricks_conn_id` and drops the dead email address, and the placeholder `on_failure_function` in `s3_upload_copy_dag.py` goes away. The callback sits on the DAG, so a task that fails after its last retry fires it once.

The connection is documented, not invented. The agent adds a `slack_alerts` entry to `airflow_settings.yaml` with the `conn_type` for a Slack webhook and the password field left for the real webhook URL, the way the file's placeholders are meant to be filled, and writes in the README how to set the same connection as an environment variable in production. It checks `.astro-registry.yaml`, which lists the five DAG paths for the Astronomer Registry, and confirms no path changes. Nothing in `.drone.yml`, which builds and pushes the image, needs to move.

The proof is a load and a fired callback. The agent installs the Airflow version the runtime image carries in a virtual environment, runs a `DagBag` over `dags/` and shows no import errors and the callback present on every task of every DAG; then it runs the callback directly with a fabricated context against a local HTTP server standing in for Slack and shows the payload that arrives, with the DAG id, task id and log URL in it. If the project already has tests, as one with a `tests/dags/` directory would, the assertion that every task has an `on_failure_callback` joins them.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Add Slack alerts on task failure to every DAG in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml, .astro-registry.yaml and every file under dags first, and note which DAGs set email_on_failure, retries or an on_failure_callback today. Add include/callbacks.py with a function that posts the DAG id, task id, execution date, try number and log URL to Slack through the Slack provider's webhook hook on a connection named slack_alerts, adding a pinned apache-airflow-providers-slack line to requirements.txt only if the runtime image does not already include it, and include/defaults.py with a DEFAULT_ARGS dictionary that sets on_failure_callback to it. Make each live DAG build its default_args from DEFAULT_ARGS, remove placeholder email addresses and empty callbacks, and leave the commented-out DAG alone. Add the slack_alerts placeholder to airflow_settings.yaml and a README note on setting it in production. Install the runtime's Airflow version, load the DagBag to show no import errors and the callback on every task, fire the callback once against a local server to show the payload, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request on its own branch with two new files under `include/`, one small change in each of the five live DAGs where `default_args` now starts from `DEFAULT_ARGS`, a `slack_alerts` placeholder in `airflow_settings.yaml` and a README section. `requirements.txt` gains a pinned `apache-airflow-providers-slack` line only if the runtime image lacks it; the `Dockerfile`, `packages.txt` and `.astro-registry.yaml` are untouched, and the commented-out DAG is unchanged.

The description quotes the DagBag summary with zero import errors, a table of DAG and task counts showing the callback on each, and the JSON payload the callback sent to the stand-in server, so a reviewer sees the exact Slack message shape without running anything. The transcript at `GET /o/{org}/sessions/{id}/messages` keeps every command and its output.

Before merging, a reviewer creates the `slack_alerts` connection in their deployment with the real webhook URL, decides whether `retries` should stay at one for the S3 DAGs, and whether `databricks_sample.py` should keep email as a second channel. A follow-up message to the same session can add an `sla_miss_callback` to the same module or route alerts to a channel per DAG owner.

## Run it on every Linear issue labelled data-platform

A data team that tracks pipeline work in Linear starts one run per issue from an automation with a `linear` trigger on the `data-platform` label, in the Project that holds the Airflow repository. The automation adds the issue after this prompt, so an issue that names the channel, or asks for alerts on SLA misses as well, is what the agent reads.

Each run loads the DAGs in its own sandbox and opens its own pull request that links the issue, and an automation run has its own `max_runs_per_hour` cap. The DagBag output in the description is what a reviewer reads first.

## Why run it on Rush?

Failure alerts are the kind of change that is five copies of the same ten lines, each slightly different, plus a connection nobody documents. The agent reads every DAG's `default_args`, puts the callback in one module, removes the dead email address and the empty placeholder function it finds, and proves the result by loading the DAGs with the same Airflow version the runtime image carries, in a sandbox with its own virtual environment.

The pull request carries the DagBag summary and the payload the callback produced, so the review is a read of two small modules and five one-line changes. Several Airflow repositories can be handled at once, each in its own sandbox with its own pull request.

## Frequently asked questions

### Why a shared default_args module instead of editing each DAG's callback?

Because the five DAGs already disagree: one has a placeholder email, one has an empty callback function, three have neither. One `DEFAULT_ARGS` dictionary in `include/defaults.py` gives every DAG the same callback, and a DAG that needs something different adds its own keys on top.

### Does it need a new dependency?

Only if the image lacks it. The agent checks whether `apache-airflow-providers-slack` is installed in the runtime the `Dockerfile` names; if not, it adds one pinned line to `requirements.txt`, which the Astro build already installs from. The DagBag load in the pull request shows the import succeeding either way.

### How is the Slack webhook configured?

As an Airflow connection named `slack_alerts`. The agent adds a placeholder for it in `airflow_settings.yaml`, which the Astro CLI loads locally, and the README explains setting the same connection as an environment variable in the deployment. The webhook URL itself never goes into the repository.

### Will a task that retries alert on every attempt?

No. `on_failure_callback` fires when a task fails for good, after its retries are used up; the DAGs that set `retries: 1` keep that value. The message includes `try_number` so the channel shows how many attempts ran.

## More Airflow jobs

- [Group Generated Tasks into TaskGroups in an Airflow DAG](https://getrush.ai/run/add-task-groups/airflow/)
- [Migrate Airflow DAGs From PythonOperator to the TaskFlow API](https://getrush.ai/run/migrate-to-taskflow-api/airflow/)
- [Upgrade an Astro Project's DAGs to Airflow 3](https://getrush.ai/run/upgrade-to-airflow-3/airflow/)

## The same job on other stacks

- [Add Slack Alerts on Failed Runs to a Dagster Project](https://getrush.ai/run/add-slack-failure-alerts/dagster/)
- [Post Slack Alerts When GitHub Actions Workflows Fail](https://getrush.ai/run/add-slack-failure-alerts/github-actions/)

## More jobs like this: Turn a Linear label into a PR

- [Add Slash Commands to a discord.py Bot](https://getrush.ai/run/add-slash-command/discord-py/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
