← Writing

Migrating from enterprise schedulers to Apache Airflow: patterns that hold up

2026-08-22

Enterprise schedulers like Control-M and Autosys tend to accumulate a decade of implicit knowledge: calendars nobody documented, dependencies that exist only in an operator's head, and jobs whose owner left years ago. Migrating such an estate to Apache Airflow is less a technical port than an archaeology project. Having been through this kind of migration, my main conclusion is that the standards you set before migrating the first job matter more than any DAG code you write afterwards.

The shape of the problem

Three things typically make a legacy scheduler estate hard to reason about:

  1. Implicit dependencies. Jobs coupled through files landing in directories, not through declared relationships. The scheduler shows you when things run, not why.
  2. Calendar sprawl. Business-day calendars, holiday overrides, and month-end variants encoded in scheduler configuration rather than in version-controlled code.
  3. Missing ownership metadata. When a job fails at 2 a.m., finding who is responsible should be a lookup, not tribal knowledge.

An inventory pass that captures dependencies, calendars, and owners for every job — before any migration work — is the single highest-leverage step. Jobs you cannot find an owner for are decommissioning candidates, not migration candidates.

Standardize the DAG template before the first migration

Every migrated DAG should conform to a template that makes retries, alerting, and SLAs declarations rather than habits:

from datetime import datetime, timedelta
 
from airflow import DAG
from airflow.operators.python import PythonOperator
 
DEFAULT_ARGS = {
    "owner": "data-platform-team",      # a team, never a person
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
    "sla": timedelta(hours=2),
    "on_failure_callback": page_owning_team,
}
 
with DAG(
    dag_id="daily_positions_extract",
    start_date=datetime(2025, 1, 1),
    schedule="0 6 * * 1-5",
    catchup=False,                       # backfills are deliberate, not accidental
    default_args=DEFAULT_ARGS,
    tags=["extracts", "wave-2"],
) as dag:
    extract = PythonOperator(
        task_id="extract",
        python_callable=run_extract,
    )

The details worth enforcing:

Coexistence is a strangler fig

Structurally, this migration is the strangler fig pattern: the new system grows around the old one, taking over jobs wave by wave while the legacy scheduler keeps running, until the remaining estate is dead wood you switch off. At every point in between, the combined estate is fully functional — coexistence is not an awkward transition to rush through; it is the mechanism of safety.

The nuance is that Fowler's original framing assumes a request router — a facade that sends each incoming request to either the old or the new system. Batch schedulers have no requests to route. The facade's equivalent here is the dependency contract between jobs: files landing, tables updating, completion events firing. Whoever honors the contract can produce or consume it, which means a job's consumers never need to know whether the job has migrated yet.

Concretely, that took two bridges:

The payoff of bridging in both directions: migration waves can be chosen by business domain rather than dictated by dependency topology. Without the bridges, you can only migrate leaves of the dependency graph inward; with them, any wave boundary is viable, because every cross-system edge flows through the contract.

Reconcile before every cutover

No job should be cut over on faith. The pattern that works — a companion to the strangler fig, sometimes called a parallel run — is to run each wave alongside the legacy scheduler and compare outputs row-for-row until the delta is zero for a full business cycle — which must include a month-end, because month-end is where calendar and dependency bugs hide.

Reconciliation sounds expensive, and it is cheaper than the alternative: a silent divergence discovered downstream by a consumer, after the legacy job has been switched off.

What I'd emphasize in hindsight

Share