Static technical auditing for machine-learning notebooks.
Live demo · Current sample audit · Audit a notebook
ModelHawk is a portfolio-grade prototype for reviewing Jupyter notebooks before they are submitted, defended or shown to technical reviewers.
It does not execute notebook code. Instead, it parses .ipynb content, extracts workflow evidence and applies deterministic checks for issues such as:
- test-set reuse during model or strategy selection
- threshold tuning against held-out test labels
- preprocessing fitted before cross-validation
- resampling order and leakage risk
- weak or inappropriate evaluation metrics
- missing reproducibility controls
- unsupported model-selection scores
- insufficient limitations, business context or technical explanation
The result is a structured audit with risk scores, findings, remediation priorities and viva/interview questions. An optional examiner layer can turn the structured audit into clearer narrative guidance, but the deterministic audit remains the source of evidence.
Production URL: https://modelhawk.onrender.com
Useful routes:
/— product overview/demo— sample audit aligned with the current hardened rules/upload— notebook upload workflow/api/health— deployment health and revision endpoint
The public instance currently runs on Render's free web-service tier. It can spin down after inactivity, so the first request after an idle period may briefly show Render's loading screen while the service wakes. The application itself is containerised and the same Docker image is exercised in CI before merge. Render is configured to deploy from main only after the GitHub checks pass.
The public deployment does not require a hosted AI-provider key. If no external provider is configured, the examiner layer uses the deterministic fallback path.
The bundled credit-card fraud notebook is intentionally useful as a regression fixture because it contains both strong project documentation and methodological problems that a reviewer should catch.
The current hardened audit reports:
| Dimension | Score |
|---|---|
| Leakage risk | 75 / 100 |
| Metric quality | 76 / 100 |
| Reproducibility | 90 / 100 |
| Portfolio readiness | 79 / 100 |
Current high-severity findings include:
- Possible test-set reuse for model or strategy selection
- Threshold exploration appears to use the test set
- Preprocessing appears to be fitted before cross-validation
The audit also flags a medium-severity concern around a custom weighted model-selection score.
These values are taken from the current deterministic /api/audit output for the bundled notebook rather than from an older static report.
The obsolete static sample PDF has been retired. Current PDF output is generated on demand from the current audit result through /api/report/pdf.
.ipynb upload
|
v
Notebook parser
|
v
Workflow and evidence extraction
|
v
Deterministic ML audit rules
|
+--> risk scores
+--> findings
+--> good-practice signals
+--> remediation priorities
+--> viva questions
|
v
Optional examiner layer
|
v
Web report / LaTeX PDF route
The design deliberately keeps evidence extraction and deterministic checks separate from explanatory AI output.
The current implementation includes:
- notebook/request size limits
- input and notebook-shape validation
- bounded AI-provider request sizes
- AI request concurrency limits
- provider timeouts
- API keys sent in headers rather than URLs
- deterministic fallback behaviour when providers are unavailable
- bounded PDF-generation concurrency
pdflatextimeout handlingpdflatex -no-shell-escape- process spawning without a shell
- temporary-file cleanup
- production security headers
- non-root Docker runtime
- deployment health checks
ModelHawk is still a static-review prototype. It should be treated as technical guidance, not as proof that a model or notebook is correct.
The audit regression suite covers cases including:
- SMOTE imported before a split without falsely calling it leakage
- actual resampling before a split
predict_probawithout falsely calling it threshold tuning- explicit threshold selection
- one model family versus multiple distinct model families
- confidence calibration from weak evidence
- scaler fitting before a split
- test-set model ranking
- threshold grids scored on
y_test - validation threshold tuning without mislabelling it as test tuning
- preprocessing outside cross-validation
- custom weighted composite scores
Run locally with:
npm testEvery pull request runs two gates.
npm ci
npm test
npm run typecheck
npm run lint
npm run build
npm audit --omit=dev --audit-level=high
CI then:
- builds the real Docker image
- starts the production container
- waits for
/api/health - verifies the LaTeX runtime inside the image
- submits the bundled fraud notebook to
/api/audit - asserts the deterministic regression scores and findings
- posts that audit result to
/api/report/pdf - validates that the response is a complete PDF document
- tears the container down
This catches both application regressions and runtime packaging failures. The PDF path is therefore exercised through the same HTTP boundary used by the deployed service rather than being reduced to a pdflatex --version check.
Third-party GitHub Actions are pinned to immutable commit SHAs rather than floating tags.
Requirements:
- Node.js 24.x (
.nvmrc) - npm
- Docker, if you want to reproduce the production container
pdflatexonly if running the PDF route outside Docker
Install and start:
npm ci
npm run devThen open:
http://localhost:3000
Optional provider configuration is documented in .env.example.
Build and run the production image:
docker build -t modelhawk .
docker run --rm -p 3000:3000 modelhawkHealth check:
curl http://localhost:3000/api/healthAll three screenshots below were captured from the current public deployment at a 1440 × 1000 desktop viewport.
ModelHawk currently performs static notebook analysis. It does not execute cells, reproduce training runs or independently validate reported metrics.
A static finding can be strong evidence of a workflow risk, but absence of a finding is not proof of correctness. The intended use is to surface review questions early and make notebook methodology easier to defend.
Future work can add bounded execution-based checks, stress tests and stronger evaluation provenance without weakening the deterministic review layer.
app/
api/audit/ deterministic notebook audit
api/audit/ai/ optional examiner layer
api/health/ deployment health endpoint
api/report/pdf/ LaTeX report generation
demo/ current sample audit
upload/ notebook upload workflow
lib/
audit-heuristics.mjs
tests/
audit-heuristics.test.mjs
public/
examples/
Dockerfile
render.yaml
.github/workflows/ci.yml
.github/dependabot.yml
Alberto Reyes Castro
MSc Artificial Intelligence Technology · London, UK