Skip per-DAG FAB permission sync when access_control is unset (3.2.2)

[dor-bernstein]

Problem

After upgrading Airflow 3.1.7 → 3.2.2, scheduling became severely slow (mainly visible as Celery tasks not getting queued). Metrics showed the classic signature of the scheduler being blocked, not busy: scheduler.critical_section_duration up, scheduler CPU down.

Live investigation on the metadata DB (Postgres) found dag-table row-lock contention:

• The dag-processor holds a single long transaction (observed ~3m40s, idle in transaction / active on SELECT dag_code…) that has taken SELECT … FOR UPDATE locks on the dag/dag_run rows (DagModelOperation.find_orm_dags).
• Both scheduler replicas block for ~2.5 min on UPDATE dag SET exceeds_max_non_backfill=… WHERE dag.dag_id = …, waiting on transactionid locks held by that dag-processor transaction.

Two things make this new in 3.2:

1. The scheduler now writes exceeds_max_non_backfill to dag rows inside its critical section (added in Separate "next dag run" from "max active runs" apache/airflow#60006, shipped 3.2.0). In 3.1 it computed this in memory and never took a dag-row write lock there, so it never collided with the dag-processor's locks.
2. _serialize_dag_capturing_errors() calls _sync_dag_perms() for every DAG on every parse (FAB auth manager), and sync_perm_for_dag() creates/refreshes a per-DAG DAG:<dag_id> resource (ab_view_menu / ab_permission_view) inside that same transaction — we caught the blocker running exactly these SELECT ab_view_menu … statements.

This is amplified enormously by our workload: thousands of dynamically-generated per-tenant DAGs, so one parse transaction locks a huge number of dag rows for minutes.

Change

Gate the per-DAG permission sync on the DAG actually defining access_control:

if "FabAuthManager" in conf.get("core", "auth_manager") and dag.access_control:
    _sync_dag_perms(dag, session=session)

When a DAG has no explicit access_control, its per-DAG FAB resources are unused — DAG-level access is granted via the global DAG resource on the role. Skipping the sync in that case removes a major component of the parse transaction's lock-hold work with no change to authorization.

Why this is safe for us

• None of our DAGs set access_control (verified).
• Okta SSO is unaffected — it runs through FAB OAuth + our OktaTeamAuthorizer group→role mapping (authentication + role assignment). This change only touches per-DAG resource creation (authorization granularity we don't use).
• Our Okta groups map to the standard global Admin/Viewer roles, which grant DAG access via the global DAG resource, not per-DAG grants.
• Existing per-DAG resources already in the DB are left untouched (just no longer refreshed).

Scope / what this deliberately does NOT do

This does not chunk the update_dags / update_dag_parsing_results_in_db transaction. That transaction's scope is per parsed file, so the deeper fix for the lock-hold duration is to reduce DAGs-per-file in our dynamic DAG factory (nitro side) and/or a properly test-validated core change to batch-commit bulk_write_to_db. Both are follow-ups; this PR is the low-risk, high-confidence first step that removes the FAB perm-sync component we directly observed blocking the scheduler.

Testing

• Run airflow-core dag-processing / permission tests in CI.
• Verify on a repro tenant that dag-processor transactions no longer hold dag locks for minutes and that UPDATE dag SET exceeds_max_non_backfill no longer blocks.

References

• Separate "next dag run" from "max active runs" apache/airflow#60006 — introduced exceeds_max_non_backfill (3.2.0), the new scheduler dag-row write.
• DagFileProcessorManager silently hangs on DB lock contention during startup — no log, no timeout, no recovery apache/airflow#68101 / Fix DagFileProcessorManager silent hang on DB lock contention apache/airflow#68118 — same theme (dag-processor holding dag locks), different function (deactivate_*); first shipped in 3.3.0, does not address this path.
• Fix dagrun starvation apache/airflow#64109 (open) — related dagrun starvation with many dagruns + low max_active_runs.
• Allowing the Airflow Scheduler to use a custom appless security manager apache/airflow#23232 (open) — request for a pluggable appless security manager (the "right" upstream home for opting out of per-DAG perm sync).

🤖 Generated with Claude Code