Skip to content

feat(ui): add aggregate Dag schedule overview view (#68532)#68547

Open
anmolxlight wants to merge 4 commits into
apache:mainfrom
anmolxlight:feature/dag-schedule-overview-68532
Open

feat(ui): add aggregate Dag schedule overview view (#68532)#68547
anmolxlight wants to merge 4 commits into
apache:mainfrom
anmolxlight:feature/dag-schedule-overview-68532

Conversation

@anmolxlight

@anmolxlight anmolxlight commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #68532

What

Adds a new /dags/schedule page that visualizes the typical (mean / median) start and end time of each Dag over a 24-hour timeline, aggregated from recent successful Dag runs. This complements the existing per-Dag Gantt view, which only shows a single run at a time. The new view operates at a different level: it aggregates across all Dags to answer "across my whole deployment, when during the day does each Dag usually run?"

Why

Operators running many Dags currently have no at-a-glance way to see how scheduled load is distributed across the day. The new view makes the whole schedule visible at once, which helps to:

  • spot overlapping heavy windows where many Dags pile up at the same hour,
  • find quiet windows for maintenance / deploys / resource-intensive jobs,
  • notice drift when a Dag's typical run time shifts over recent runs,
  • plan capacity and SLAs across the fleet.

How

Backend

  • New /ui/dag_schedule_overview endpoint (in api_fastapi/core_api/routes/ui/dag_schedule_overview.py) gated by requires_access_dag(method="GET", access_entity=DagAccessEntity.RUN).
  • New DagScheduleOverviewService (api_fastapi/core_api/services/ui/dag_schedule_overview.py) that:
    • fetches all DagModel rows (so dags with no runs still appear as zero-statistic rows),
    • pulls up to 200 most-recent successful DagRuns per dag, optionally scoped by a run_after window,
    • computes mean/median of start_date and end_date mapped to seconds-of-day in UTC, plus mean/median duration,
    • supports dag_id_pattern and dag_display_name_pattern filters.
  • New datamodels DagScheduleOverviewEntry + DagScheduleOverviewCollectionResponse (api_fastapi/core_api/datamodels/ui/dag_schedule_overview.py).
  • 11 backend tests in tests/unit/api_fastapi/core_api/routes/ui/test_dag_schedule_overview.py: 200/401/403, all-three-dag fixture, morning/evening/no-runs statistics, FAILED-run exclusion, pattern filters, run_after window edges.
  • OpenAPI _private_ui.yaml updated; pnpm codegen regenerated.

Frontend

  • New page src/pages/DagsList/ScheduleOverview/ mounted at /dags/schedule with:
    • one row per Dag (linkable to the Dag detail page),
    • a 24h tick rail at 0/3/6/9/12/15/18/21/24 hours,
    • a horizontal bar showing the mean start→end window,
    • a hover tooltip with mean and median start / end, mean and median duration, and the recent-runs date range,
    • a client-side dag-id filter input.
  • New sidebar nav button Schedule Overview (FiBarChart2) on the main nav, alongside Dags.
  • New i18n keys under dags.scheduleOverview.* and common.nav.scheduleOverview (English locale).

Verification

# Backend
uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_dag_schedule_overview.py --with-db-init
# 11 passed
uv run --project airflow-core ruff check + ruff format --check + mypy
# All clean
# Frontend
pnpm lint      # eslint --quiet && tsc -p tsconfig.app.json — passes
pnpm prettier --check public/i18n/locales/en/dags.json public/i18n/locales/en/common.json
# Passes

The two pre-existing vitest failures in useTagFilter.test.tsx and Logs.test.tsx reproduce on main (verified with git stash), so they are unrelated to this PR.

@boring-cyborg boring-cyborg Bot added area:API Airflow's REST/HTTP API area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default labels Jun 14, 2026
@anmolxlight anmolxlight force-pushed the feature/dag-schedule-overview-68532 branch from 64767cb to a33d4bd Compare June 14, 2026 22:49
anmolxlight added a commit to anmolxlight/airflow that referenced this pull request Jun 15, 2026
@anmolxlight
fix(pr-68547): regenerate OpenAPI spec and fix positional @provide_se…
9bfa0f7 
…ssion

Address failures of CI image checks / Static checks on PR apache#68547:

1. Regenerate airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml
   and the airflow-core/src/airflow/ui/openapi-gen/ TS client files. The prek
   generate-openapi-spec hook wanted to reorder /ui/teams and /ui/dag_schedule_overview,
   expand the multi-line docstring for get_dag_schedule_overview, switch minimum: 0
   to minimum: 0.0 (Pydantic v2 float coercion), and drop start_*_seconds / end_*_seconds
   from the DagScheduleOverviewEntry required list. This change matches what prek
   would have done in CI.

2. Move the @provide_session 'session' argument to keyword-only in the
   autouse setup fixture of test_dag_schedule_overview.py. New @provide_session
   functions must declare session after a bare '*' per scripts/ci/prek/check_provide_session_kwargs.py.

Local verification:
- prek run --files <changed files>: passed
- prek run mypy-airflow-core --files <changed files>: passed
- prek run check-no-new-provide-session-positional --files <test file>: passed
- uv run --project airflow-core pytest test_dag_schedule_overview.py: 11/11 passed
@choo121600

choo121600 commented Jun 15, 2026

Copy link
Copy Markdown
Member

Could you please attach a short recording of this feature following the PR template? It would be helpful for the review.

@anmolxlight
feat(ui): add aggregate Dag schedule overview view (apache#68532)
083efaa 
Adds a 24h Gantt-style view that answers the question `across my whole
deployment, when during the day does each Dag usually run?` — a
complement to the per-Dag Gantt view, which only shows a single run.

Backend
  * New `/ui/dag_schedule_overview` endpoint that aggregates the
    most-recent successful Dag runs per Dag and returns mean/median
    start/end times-of-day in UTC, plus duration statistics.
  * Optional `run_after_gte` / `run_after_lte` window and
    `dag_id_pattern` / `dag_display_name_pattern` filters.
  * 11 backend tests covering 200/401/403, all-three-dag fixture,
    morning/evening/no-runs statistics, FAILED-run exclusion, and the
    filter / window edge cases.
  * OpenAPI YAML updated, `pnpm codegen` regenerated.

Frontend
  * New `/dags/schedule` page (one row per Dag, mean start→end bar
    across a 24h tick rail, hover tooltip with mean+median start, end,
    duration, and recent-runs range).
  * New `Schedule Overview` sidebar nav link.
  * i18n keys for English locale.
@anmolxlight
fix(pr-68547): regenerate OpenAPI spec and fix positional @provide_se…
80e1c1e 
…ssion

Address failures of CI image checks / Static checks on PR apache#68547:

1. Regenerate airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml
   and the airflow-core/src/airflow/ui/openapi-gen/ TS client files. The prek
   generate-openapi-spec hook wanted to reorder /ui/teams and /ui/dag_schedule_overview,
   expand the multi-line docstring for get_dag_schedule_overview, switch minimum: 0
   to minimum: 0.0 (Pydantic v2 float coercion), and drop start_*_seconds / end_*_seconds
   from the DagScheduleOverviewEntry required list. This change matches what prek
   would have done in CI.

2. Move the @provide_session 'session' argument to keyword-only in the
   autouse setup fixture of test_dag_schedule_overview.py. New @provide_session
   functions must declare session after a bare '*' per scripts/ci/prek/check_provide_session_kwargs.py.

Local verification:
- prek run --files <changed files>: passed
- prek run mypy-airflow-core --files <changed files>: passed
- prek run check-no-new-provide-session-positional --files <test file>: passed
- uv run --project airflow-core pytest test_dag_schedule_overview.py: 11/11 passed
@anmolxlight
fix(pr-68547): widen formatTooltipBody entry type to allow undefined
add116b 
The OpenAPI spec regen in the previous commit changed the DagScheduleOverviewEntry
schema: start_*_seconds, end_*_seconds, duration_*_seconds, and their median
counterparts were moved out of the 'required' list in Pydantic v2. The auto-generated
TS type now marks these fields as 'number | null | undefined'.

The hand-written formatTooltipBody in scheduleOverviewUtils.ts only accepted
'number | null', causing:

    error TS2345 in ScheduleOverviewRow.tsx:44
    Argument of type 'DagScheduleOverviewEntry' is not assignable to parameter
    of type '{ readonly end_mean_seconds: number | null; ... }'
    Types of property 'end_mean_seconds' are incompatible.
    Type 'number | null | undefined' is not assignable to type 'number | null'.

Widen the formatTooltipBody parameter to allow 'undefined' for the now-optional
duration/start/end fields. The downstream helpers (formatSecondsAsClock,
formatDurationSeconds) already accept undefined and render it as em-dash.

Local verification:
- prek run ts-compile-lint-ui --files <changed file>: Passed
- prek run --files <changed file>: Passed
@anmolxlight
fix(pr-68547): use optional (?) on formatTooltipBody duration/start/e…
b3353f0 
…nd params

The previous commit added '| undefined' to the duration/start/end types in
formatTooltipBody's parameter, but TypeScript's structural typing is stricter
about optional vs required properties. Since DagScheduleOverviewEntry marks
end_mean_seconds / end_median_seconds / start_mean_seconds / start_median_seconds
as '?: number | null' (truly optional, undefined comes from optionality, not
the value type), the parameter must mirror that with '?:' on the same fields.

'?: number | null' and ': number | null | undefined' are NOT structurally
assignable in either direction under TypeScript's strictness here.

Use '?: number | null' to match the auto-generated type shape. The
downstream helpers (formatSecondsAsClock, formatDurationSeconds) still
handle the undefined case and render em-dash.

Local verification:
- pnpm tsc --noEmit -p tsconfig.app.json: clean
- prek run --files <changed file>: Passed
- prek run ts-compile-lint-ui --files <changed file>: Passed
@anmolxlight

anmolxlight commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author
VID_20260615_232836_381.mp4

Screen recording of the new Schedule Overview page:

Shows the /dags/schedule page with the 24h UTC timeline bars, hover tooltips, and the filter working.

@anmolxlight anmolxlight force-pushed the feature/dag-schedule-overview-68532 branch from a60f1d5 to b3353f0 Compare June 15, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@shubhamraj-git shubhamraj-git Awaiting requested review from shubhamraj-git shubhamraj-git is a code owner

@guan404ming guan404ming Awaiting requested review from guan404ming guan404ming is a code owner

@vatsrahul1001 vatsrahul1001 Awaiting requested review from vatsrahul1001 vatsrahul1001 is a code owner

@choo121600 choo121600 Awaiting requested review from choo121600 choo121600 is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@rawwar rawwar Awaiting requested review from rawwar rawwar is a code owner

@jason810496 jason810496 Awaiting requested review from jason810496 jason810496 is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area:API Airflow's REST/HTTP API area:translations area:UI Related to UI/UX. For Frontend Developers. translation:default

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add an aggregate Dag schedule view: typical daily run times across all Dags

2 participants

@anmolxlight @choo121600