Skip to content

Add portfolio-grade data evidence - #3

Merged
DataTideHH merged 15 commits into
mainfrom
portfolio/data-evidence
Jul 27, 2026
Merged

Add portfolio-grade data evidence#3
DataTideHH merged 15 commits into
mainfrom
portfolio/data-evidence

Conversation

@DataTideHH

Copy link
Copy Markdown
Owner

Summary

  • add named SQL data-quality checks and execute them through the application
  • add dataset summary, data-quality and ingestion-run endpoints
  • expose the versioned OpenAPI 3.1 document through the Flask application
  • document the architecture, relational model, data dictionary and field-level provenance
  • expand ingestion-run evidence with API access and failure-path tests
  • add OpenAPI parsing and route tests
  • extend CI to install development requirements, validate the contract, build fixture data and require a passing persisted quality report
  • rewrite the README around recruiter-facing Data/BI, process and API evidence

New endpoints

  • GET /api/v1/summary
  • GET /api/v1/data-quality
  • GET /api/v1/ingestion-runs
  • GET /openapi/openapi.yaml

Validation target

  • Python 3.12 compilation succeeds
  • all unit, persistence, route, CLI, reporting and OpenAPI tests pass
  • deterministic fixture ingestion succeeds without live network access
  • the persisted SQL quality report returns passed
  • Ubuntu and Windows CI jobs remain green

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1ab083477

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

-- name: incomplete_ingestion_runs
SELECT COUNT(*) AS violations
FROM ingestion_runs
WHERE status = 'running';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ignore currently active ingestion runs

When /api/v1/data-quality is requested while a normal refresh is still fetching or validating source data, this condition counts that legitimate active run as an incomplete_ingestion_runs violation and changes the overall report from passed to warning. The check is described as detecting runs left running indefinitely, so it should apply an age threshold rather than flag every row whose current state is running.

Useful? React with 👍 / 👎.

Comment thread country_api/database.py
Comment on lines +352 to +354
WHERE status = 'success'
ORDER BY run_id DESC
LIMIT 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Order successful ingestions by completion time

When two refresh commands overlap and the earlier-started run finishes after the later-started run, ordering by run_id returns the earlier completion timestamp even though the other run committed the currently persisted data afterward. This makes lastSuccessfulIngestion inaccurate in both the summary and data-quality provenance; select the latest completed_at instead.

Useful? React with 👍 / 👎.

@DataTideHH
DataTideHH merged commit 058e52e into main Jul 27, 2026
2 checks passed
@DataTideHH
DataTideHH deleted the portfolio/data-evidence branch July 27, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant