# Move Airflow Connections to a Secrets Backend

Yes. An agent points Airflow at AWS Secrets Manager, names the secret each DAG's connection id expects, proves the lookup with a mocked AWS and opens a PR.

[Run this on Rush](https://cloud.getrush.ai/run/add-secrets-backend/airflow/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

The agent clones the repository into its own cloud sandbox and finds every connection the DAGs expect. In an Astro project like astronomer/airflow-example-dags, the `Dockerfile` is one line, `FROM quay.io/astronomer/astro-runtime:6.0.0`, `requirements.txt` is empty and `packages.txt` adds `postgresql`. Under `dags/`, `s3_upload_dag.py`, `s3_upload_copy_dag.py` and `s3_upload_copy_delete.py` all build an `S3Hook(aws_conn_id=S3_CONN_ID)` with `S3_CONN_ID='astro-s3-workshop'`, `databricks_sample.py` adds `databricks_conn_id: 'adb_workspace'` in its default arguments and a `PostgresOperator` with `postgres_conn_id='prod_postgres'`, and `kube_pod.py` reads the namespace from Airflow's own configuration. `airflow_settings.yaml` is the blank Astro CLI template with an empty `conn_id` entry, which means every one of those three connection ids is created by hand in each environment today.

It writes the backend configuration where Astro reads it. A new `.env.example` carries `AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend` and `AIRFLOW__SECRETS__BACKEND_KWARGS` as the JSON `{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "region_name": "..."}`, with a comment that the same two variables go on the Deployment in Astro. A short section in a new `README.md` lists the secrets to create, one per connection id the DAGs use: `airflow/connections/astro-s3-workshop`, `airflow/connections/adb_workspace` and `airflow/connections/prod_postgres`, each stored as a plain-text connection URI, with no leading slash. The agent checks whether the Runtime 6.0.0 image ships the Amazon provider the backend class lives in; if it did not, `apache-airflow-providers-amazon` would go into `requirements.txt`, and the pull request says which it found.

Two things it finds on the way go into the same pull request. `.gitignore` lists `.env`, which is right, and `airflow_setttings.yaml` with three t's, which does not match the real `airflow_settings.yaml`, so a developer who fills the template with a password commits it; the agent fixes the spelling. And `astro-s3-workshop` contains dashes, so the fallback of an `AIRFLOW_CONN_` environment variable is a name a shell cannot export, one more reason the backend is the right home for it. The DAG files themselves are not changed: the whole point is that `aws_conn_id='astro-s3-workshop'` keeps working while the value moves.

The sandbox has no Docker, so the agent proves the configuration with Airflow itself rather than the Astro image. It installs Airflow 2.4 with its constraints file into a virtual environment, plus the Amazon, Databricks, Kubernetes and Postgres providers the DAGs import, points `AIRFLOW_HOME` at a scratch directory, sets the two variables from `.env.example`, and runs `airflow dags list-import-errors` to show every DAG still parses. Then it installs `moto`, creates the secret `airflow/connections/astro-s3-workshop` in a mocked Secrets Manager, and shows `BaseHook.get_connection('astro-s3-workshop')` returning a connection whose host and login came from the secret, with the metadata database holding no connections at all. That transcript is pasted into the pull request.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Configure an AWS Secrets Manager secrets backend for this Astro Airflow project. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml, .gitignore, .astro/config.yaml and every file under dags/ first, and list every connection id and Variable the DAGs use. Add a .env.example with AIRFLOW__SECRETS__BACKEND and AIRFLOW__SECRETS__BACKEND_KWARGS using the airflow/connections and airflow/variables prefixes, and a README section that names the secret to create for each connection id as a connection URI. Check whether the Runtime image in the Dockerfile ships the Amazon provider and add it to requirements.txt only if it does not. Fix the misspelt settings file entry in .gitignore. Do not change the DAG files. In your sandbox install Airflow 2.4 with its constraints and the providers the DAGs import, run airflow dags list-import-errors with the backend configured, then use moto to create one connection secret and show BaseHook.get_connection reading it with an empty metadata database. Open a pull request that lists the files changed, the secrets to create, and the command output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

What comes back is a pull request adding `.env.example` and `README.md`, editing `.gitignore`, and touching `requirements.txt` only if the image lacked the provider. The description lists the three connection ids found in `dags/`, the secret name for each, the exact two environment variables to set on the Deployment, and the reason the settings file was never ignored.

The run's record holds the `airflow dags list-import-errors` output and the mocked round trip from a Secrets Manager secret to a `Connection` object, so a reviewer can see the backend class loads and resolves a name with the configured prefix before any AWS account is involved.

Before merging, a reviewer creates the secrets in the real account, attaches a role that can read them to the Deployment, sets the two variables there, and decides one thing the pull request leaves to the team: whether `prod_postgres` should also move, or stay a per-environment connection in the Airflow UI.

## Run it on every Linear issue labelled security

A data team that tracks hardening work in Linear starts one run per issue from an automation with a `linear` trigger on the `security` label, in the Project that holds the Airflow repository. The trigger adds the issue after this prompt and fills in no placeholder, so an issue that asks for Parameter Store instead of Secrets Manager, or for Variables only, is what the agent reads and acts on.

Each run clones the project into its own sandbox, installs Airflow there, and opens its own pull request linking the issue. An automation run has its own `max_runs_per_hour` cap, and a second labelled issue gets a second sandbox.

## Why run it on Rush?

A secrets backend is mostly configuration, and configuration is where typos hide: a prefix with a leading slash, a connection id that does not match the DAG, a class path with one wrong segment. The agent reads every DAG for the ids it needs, then exercises the backend against a mocked Secrets Manager in a real Airflow process, so the pull request arrives with the lookup shown working rather than described.

The sandbox installs Airflow with the constraints file for the version the image runs, a step most laptops never bother with, and keeps the full command record. The same request runs against every Astro project an organisation has, each with its own `dags/` read first, and each pull request lists exactly the secrets that project needs.

## Frequently asked questions

### Do the DAG files change?

No. `aws_conn_id='astro-s3-workshop'` and the other ids stay as they are. Airflow checks the secrets backend first, then environment variables, then the metadata database, so moving the value changes nothing in the code that names it.

### How are the secrets named?

With the prefix from the backend kwargs followed by a slash and the id the DAG uses: `airflow/connections/astro-s3-workshop`, and `airflow/variables/<key>` for a Variable. The value is a plain-text connection URI with no surrounding quotes or braces, and the name has no leading slash.

### Does local development still work?

Yes. The same two variables go into the project's `.env`, which the Astro CLI passes to the local containers and `.gitignore` already excludes, plus AWS credentials that can read the secrets. Without them, Airflow falls through to `airflow_settings.yaml` and the UI as before.

### Why mock AWS in the sandbox instead of using the real account?

The run has no AWS credentials and should not need them to prove the configuration. `moto` serves the Secrets Manager API locally, so the agent can show the backend class resolving a secret name into a `Connection` with the exact prefix and id the pull request documents.

## More Airflow jobs

- [Add Slack Alerts on Task Failure to Airflow DAGs](https://getrush.ai/run/add-slack-failure-alerts/airflow/)
- [Group Generated Tasks into TaskGroups in an Airflow DAG](https://getrush.ai/run/add-task-groups/airflow/)
- [Migrate Airflow DAGs From PythonOperator to the TaskFlow API](https://getrush.ai/run/migrate-to-taskflow-api/airflow/)

## More jobs like this: Turn a Linear label into a PR

- [Store Auth Tokens in Secure Storage in a React Native App](https://getrush.ai/run/add-secure-storage/react-native/)
- [Add Pod and Container Security Contexts to a Helm Chart](https://getrush.ai/run/add-security-context/helm/)
- [Add Security Headers and a CSP to an Eleventy Site](https://getrush.ai/run/add-security-headers/eleventy/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
