All jobs

Move Airflow Connections to a Secrets Backend

Yes. An agent points Airflow at AWS Secrets Manager, names the secret each DAG's connection id expects, proves the lookup with a mocked AWS and opens a PR.

Run this on RushOpens the console with this prompt ready to send, in a Project that has your repository.

What the agent does

The agent clones the repository into its own cloud sandbox and finds every connection the DAGs expect. In an Astro project like astronomer/airflow-example-dags, the Dockerfile is one line, FROM quay.io/astronomer/astro-runtime:6.0.0, requirements.txt is empty and packages.txt adds postgresql. Under dags/, s3_upload_dag.py, s3_upload_copy_dag.py and s3_upload_copy_delete.py all build an S3Hook(aws_conn_id=S3_CONN_ID) with S3_CONN_ID='astro-s3-workshop', databricks_sample.py adds databricks_conn_id: 'adb_workspace' in its default arguments and a PostgresOperator with postgres_conn_id='prod_postgres', and kube_pod.py reads the namespace from Airflow's own configuration. airflow_settings.yaml is the blank Astro CLI template with an empty conn_id entry, which means every one of those three connection ids is created by hand in each environment today.

It writes the backend configuration where Astro reads it. A new .env.example carries AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend and AIRFLOW__SECRETS__BACKEND_KWARGS as the JSON {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "region_name": "..."}, with a comment that the same two variables go on the Deployment in Astro. A short section in a new README.md lists the secrets to create, one per connection id the DAGs use: airflow/connections/astro-s3-workshop, airflow/connections/adb_workspace and airflow/connections/prod_postgres, each stored as a plain-text connection URI, with no leading slash. The agent checks whether the Runtime 6.0.0 image ships the Amazon provider the backend class lives in; if it did not, apache-airflow-providers-amazon would go into requirements.txt, and the pull request says which it found.

Two things it finds on the way go into the same pull request. .gitignore lists .env, which is right, and airflow_setttings.yaml with three t's, which does not match the real airflow_settings.yaml, so a developer who fills the template with a password commits it; the agent fixes the spelling. And astro-s3-workshop contains dashes, so the fallback of an AIRFLOW_CONN_ environment variable is a name a shell cannot export, one more reason the backend is the right home for it. The DAG files themselves are not changed: the whole point is that aws_conn_id='astro-s3-workshop' keeps working while the value moves.

The sandbox has no Docker, so the agent proves the configuration with Airflow itself rather than the Astro image. It installs Airflow 2.4 with its constraints file into a virtual environment, plus the Amazon, Databricks, Kubernetes and Postgres providers the DAGs import, points AIRFLOW_HOME at a scratch directory, sets the two variables from .env.example, and runs airflow dags list-import-errors to show every DAG still parses. Then it installs moto, creates the secret airflow/connections/astro-s3-workshop in a mocked Secrets Manager, and shows BaseHook.get_connection('astro-s3-workshop') returning a connection whose host and login came from the secret, with the metadata database holding no connections at all. That transcript is pasted into the pull request.

Start it with one API call

POST https://api.phoenix.so/o/{org}/sessions with this body:

{
  "agent": "claude",
  "input": "Configure an AWS Secrets Manager secrets backend for this Astro Airflow project. Read the Dockerfile, requirements.txt, packages.txt, airflow_settings.yaml, .gitignore, .astro/config.yaml and every file under dags/ first, and list every connection id and Variable the DAGs use. Add a .env.example with AIRFLOW__SECRETS__BACKEND and AIRFLOW__SECRETS__BACKEND_KWARGS using the airflow/connections and airflow/variables prefixes, and a README section that names the secret to create for each connection id as a connection URI. Check whether the Runtime image in the Dockerfile ships the Amazon provider and add it to requirements.txt only if it does not. Fix the misspelt settings file entry in .gitignore. Do not change the DAG files. In your sandbox install Airflow 2.4 with its constraints and the providers the DAGs import, run airflow dags list-import-errors with the backend configured, then use moto to create one connection secret and show BaseHook.get_connection reading it with an empty metadata database. Open a pull request that lists the files changed, the secrets to create, and the command output.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
  1. Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
  2. Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
  3. Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

What comes back

What comes back is a pull request adding .env.example and README.md, editing .gitignore, and touching requirements.txt only if the image lacked the provider. The description lists the three connection ids found in dags/, the secret name for each, the exact two environment variables to set on the Deployment, and the reason the settings file was never ignored.

The run's record holds the airflow dags list-import-errors output and the mocked round trip from a Secrets Manager secret to a Connection object, so a reviewer can see the backend class loads and resolves a name with the configured prefix before any AWS account is involved.

Before merging, a reviewer creates the secrets in the real account, attaches a role that can read them to the Deployment, sets the two variables there, and decides one thing the pull request leaves to the team: whether prod_postgres should also move, or stay a per-environment connection in the Airflow UI.

Run it on every Linear issue labelled security

A data team that tracks hardening work in Linear starts one run per issue from an automation with a linear trigger on the security label, in the Project that holds the Airflow repository. The trigger adds the issue after this prompt and fills in no placeholder, so an issue that asks for Parameter Store instead of Secrets Manager, or for Variables only, is what the agent reads and acts on.

Each run clones the project into its own sandbox, installs Airflow there, and opens its own pull request linking the issue. An automation run has its own max_runs_per_hour cap, and a second labelled issue gets a second sandbox.

The use case this job follows, with its automation and trigger: Turn a Linear label into a PR.

Why run it on Rush?

A secrets backend is mostly configuration, and configuration is where typos hide: a prefix with a leading slash, a connection id that does not match the DAG, a class path with one wrong segment. The agent reads every DAG for the ids it needs, then exercises the backend against a mocked Secrets Manager in a real Airflow process, so the pull request arrives with the lookup shown working rather than described.

The sandbox installs Airflow with the constraints file for the version the image runs, a step most laptops never bother with, and keeps the full command record. The same request runs against every Astro project an organisation has, each with its own dags/ read first, and each pull request lists exactly the secrets that project needs.

Frequently asked questions

Do the DAG files change?

No. aws_conn_id='astro-s3-workshop' and the other ids stay as they are. Airflow checks the secrets backend first, then environment variables, then the metadata database, so moving the value changes nothing in the code that names it.

How are the secrets named?

With the prefix from the backend kwargs followed by a slash and the id the DAG uses: airflow/connections/astro-s3-workshop, and airflow/variables/<key> for a Variable. The value is a plain-text connection URI with no surrounding quotes or braces, and the name has no leading slash.

Does local development still work?

Yes. The same two variables go into the project's .env, which the Astro CLI passes to the local containers and .gitignore already excludes, plus AWS credentials that can read the secrets. Without them, Airflow falls through to airflow_settings.yaml and the UI as before.

Why mock AWS in the sandbox instead of using the real account?

The run has no AWS credentials and should not need them to prove the configuration. moto serves the Secrets Manager API locally, so the agent can show the backend class resolving a secret name into a Connection with the exact prefix and id the pull request documents.

More Airflow jobs

More jobs like this: Turn a Linear label into a PR

Keep reading