All jobs

Add Slack Alerts on Task Failure to Airflow DAGs

Yes. An agent adds one Slack failure callback to every DAG in your Airflow project through a shared default_args module, tests it with a DagBag load, and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent reads what each DAG does on failure today. In an Astro project like astronomer/airflow-example-dags, the Dockerfile is FROM quay.io/astronomer/astro-runtime:6.0.0, requirements.txt is empty, packages.txt adds postgresql, and airflow_settings.yaml holds empty placeholders for connections, pools and variables. Each of the five live DAGs under dags/ declares its own default_args dictionary. databricks_sample.py sets 'email': ['EMAIL'] and 'email_on_failure': True, a placeholder address that can never deliver. s3_upload_dag.py, s3_upload_copy_dag.py, s3_upload_copy_delete.py and kube_pod.py set 'email_on_failure': False with 'retries': 1 and a five-minute retry_delay. s3_upload_copy_dag.py goes further: it defines an empty on_failure_function(**kwargs) whose docstring says to insert notification logic here, and passes it as on_failure_callback on each PythonOperator in its loop. The sixth file, example_external_task_sensor.py, is commented out entirely and is left alone.

It then adds the callback once, not five times. A new include/callbacks.py defines notify_slack_failure(context) that reads dag_id, task_id, execution_date, try_number and the log URL from the task instance in context, and posts a short message through SlackWebhookHook from apache-airflow-providers-slack against a connection id slack_alerts. That provider is not part of every runtime image, so the agent checks the image's Airflow install with pip show and, when it is missing, adds one pinned line to the empty requirements.txt, the file the Astro build already installs from. A include/defaults.py exposes a DEFAULT_ARGS dictionary with on_failure_callback set to that function, retries and retry_delay carried over from the DAGs that already set them, and email_on_failure off. Each DAG then builds its default_args from DEFAULT_ARGS with its own keys on top, so databricks_sample.py keeps databricks_conn_id and drops the dead email address, and the placeholder on_failure_function in s3_upload_copy_dag.py goes away. The callback sits on the DAG, so a task that fails after its last retry fires it once.

The connection is documented, not invented. The agent adds a slack_alerts entry to airflow_settings.yaml with the conn_type for a Slack webhook and the password field left for the real webhook URL, the way the file's placeholders are meant to be filled, and writes in the README how to set the same connection as an environment variable in production. It checks .astro-registry.yaml, which lists the five DAG paths for the Astronomer Registry, and confirms no path changes. Nothing in .drone.yml, which builds and pushes the image, needs to move.

The proof is a load and a fired callback. The agent installs the Airflow version the runtime image carries in a virtual environment, runs a DagBag over dags/ and shows no import errors and the callback present on every task of every DAG; then it runs the callback directly with a fabricated context against a local HTTP server standing in for Slack and shows the payload that arrives, with the DAG id, task id and log URL in it. If the project already has tests, as one with a tests/dags/ directory would, the assertion that every task has an on_failure_callback joins them.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Add Slack alerts on task failure to every DAG in this Airflow project. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml, .astro-registry.yaml and every file under dags first, and note which DAGs set email_on_failure, retries or an on_failure_callback today. Add include/callbacks.py with a function that posts the DAG id, task id, execution date, try number and log URL to Slack through the Slack provider's webhook hook on a connection named slack_alerts, adding a pinned apache-airflow-providers-slack line to requirements.txt only if the runtime image does not already include it, and include/defaults.py with a DEFAULT_ARGS dictionary that sets on_failure_callback to it. Make each live DAG build its default_args from DEFAULT_ARGS, remove placeholder email addresses and empty callbacks, and leave the commented-out DAG alone. Add the slack_alerts placeholder to airflow_settings.yaml and a README note on setting it in production. Install the runtime's Airflow version, load the DagBag to show no import errors and the callback on every task, fire the callback once against a local server to show the payload, and open a pull request.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request on its own branch with two new files under include/, one small change in each of the five live DAGs where default_args now starts from DEFAULT_ARGS, a slack_alerts placeholder in airflow_settings.yaml and a README section. requirements.txt gains a pinned apache-airflow-providers-slack line only if the runtime image lacks it; the Dockerfile, packages.txt and .astro-registry.yaml are untouched, and the commented-out DAG is unchanged.

The description quotes the DagBag summary with zero import errors, a table of DAG and task counts showing the callback on each, and the JSON payload the callback sent to the stand-in server, so a reviewer sees the exact Slack message shape without running anything. The transcript at GET /o/{org}/sessions/{id}/messages keeps every command and its output.

Before merging, a reviewer creates the slack_alerts connection in their deployment with the real webhook URL, decides whether retries should stay at one for the S3 DAGs, and whether databricks_sample.py should keep email as a second channel. A follow-up message to the same session can add an sla_miss_callback to the same module or route alerts to a channel per DAG owner.

Run it on every Linear issue labelled data-platform

A data team that tracks pipeline work in Linear starts one run per issue from an automation with a linear trigger on the data-platform label, in the Project that holds the Airflow repository. The automation adds the issue after this prompt, so an issue that names the channel, or asks for alerts on SLA misses as well, is what the agent reads.

Each run loads the DAGs in its own sandbox and opens its own pull request that links the issue, and an automation run has its own max_runs_per_hour cap. The DagBag output in the description is what a reviewer reads first.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

Failure alerts are the kind of change that is five copies of the same ten lines, each slightly different, plus a connection nobody documents. The agent reads every DAG's default_args, puts the callback in one module, removes the dead email address and the empty placeholder function it finds, and proves the result by loading the DAGs with the same Airflow version the runtime image carries, in a sandbox with its own virtual environment.

The pull request carries the DagBag summary and the payload the callback produced, so the review is a read of two small modules and five one-line changes. Several Airflow repositories can be handled at once, each in its own sandbox with its own pull request.

Frequently asked questions

Why a shared default_args module instead of editing each DAG's callback?

Because the five DAGs already disagree: one has a placeholder email, one has an empty callback function, three have neither. One DEFAULT_ARGS dictionary in include/defaults.py gives every DAG the same callback, and a DAG that needs something different adds its own keys on top.

Does it need a new dependency?

Only if the image lacks it. The agent checks whether apache-airflow-providers-slack is installed in the runtime the Dockerfile names; if not, it adds one pinned line to requirements.txt, which the Astro build already installs from. The DagBag load in the pull request shows the import succeeding either way.

How is the Slack webhook configured?

As an Airflow connection named slack_alerts. The agent adds a placeholder for it in airflow_settings.yaml, which the Astro CLI loads locally, and the README explains setting the same connection as an environment variable in the deployment. The webhook URL itself never goes into the repository.

Will a task that retries alert on every attempt?

No. on_failure_callback fires when a task fails for good, after its retries are used up; the DAGs that set retries: 1 keep that value. The message includes try_number so the channel shows how many attempts ran.

More Airflow jobs

The same job on other stacks

More jobs like this: Turn a Linear label into a PR

Keep reading