Skip to content

Milestone: evaluated flight-delay prediction model #2

Closed

Description

@mitulpatel123

Objective

Replace the rules-only delay score with a reproducible trained-model milestone while retaining baseline-rules-v1 as the benchmark and fallback.

Data source

Use the official U.S. Bureau of Transportation Statistics Airline On-Time Performance data:

Record the exact months, download date, selected fields, and transformation logic. Do not commit a large raw dataset to Git.

Acceptance criteria

  • Define the prediction time and target: arrival delay of at least 15 minutes
  • Create a documented data contract and reproducible preprocessing command
  • Prevent target leakage by excluding actual arrival/departure outcomes unavailable at prediction time
  • Use a chronological train/validation/test split
  • Compare a naive majority baseline, logistic regression, and one tree-based model
  • Report ROC-AUC, PR-AUC, precision, recall, F1, Brier score, and calibration
  • Include error slices by carrier, origin airport, route volume, and time period
  • Choose the operating threshold from a documented operational cost assumption
  • Save versioned model metadata without committing unsafe or opaque serialized artifacts
  • Publish a model card with data limitations, intended use, non-goals, and failure modes
  • Add unit tests for preprocessing and an integration test for model-backed prediction
  • Keep baseline-rules-v1 available as a transparent fallback

Recruiter evidence

The README must show a compact comparison table, one calibration visualization, one error-slice finding, the exact reproduction command, and an honest explanation of what improved and what did not.

Out of scope

  • claiming real-time airline production readiness
  • claiming causality
  • using post-event fields that leak the outcome
  • adding an LLM before the predictive model and evaluation are credible

Activity

  1. mitulpatel123 commented on Aug 8, 2026

    @mitulpatel123
    OwnerAuthor

    The first milestone slice is now in draft PR #3: a fixed T-24h prediction contract, explicit leakage exclusions, chronological evaluation split, calibration/error-slice requirements, and artifact-safety rules. Both CI runs pass.

  2. mitulpatel123 commented on Aug 8, 2026

    @mitulpatel123
    OwnerAuthor

    Completed in PR #4. The milestone now includes official BTS data lineage, a 90,000-row chronological evaluation, logistic and fixed-budget tree baselines, validation-only threshold selection, calibration, error slices, a portable JSON model, a model card, API integration, and passing CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions