Proofpack

Proofpack is a narrow browser prototype that compares a one-page packing list with up to three delivery photos. It returns evidence-backed confirmations, visible identity or quantity mismatches, and conservative Unverified results when the submitted views do not prove a conclusion.

The app does not use arbitrary retail-product recognition. It reads printed SKUs and unique UNIT-XX labels attached to controlled household objects.

Core safety rule

Every physical object keeps one unique UNIT-XX label in every photograph. The same label in multiple photos is one object. If labels are obscured or the capture set cannot distinguish repeated objects, Proofpack refuses to verify the count. A missing camera view never becomes a missing-item claim.

Run locally

Requirements: Node.js 20+ and an OpenAI API key with access to an image-capable Responses API model.

npm install
cp .env.example .env.local
# Set OPENAI_API_KEY in .env.local. Never commit it.
npm run dev

Open http://localhost:3000.

The default analysis model is gpt-5.6-luna with low reasoning and a 4,000-token output cap. Override it with OPENAI_MODEL if required.

Reproducible test set

• sample-data/packing-list.pdf - one-page, four-row source document.
• sample-data/printable-unit-labels.pdf - six initial labels and one correction label.
• sample-data/expected-findings.json - expected outcomes recorded before testing.
• sample-data/README.md - physical contents, photo convention and corrected-delivery procedure.
• sample-data/synthetic/initial/ - three AI-generated development photos for the mismatch/ambiguity case.
• sample-data/synthetic/corrected/ - two AI-generated development photos for the corrected case.
• sample-data/results/ - raw measured Responses API output for the three synthetic test cases.

The synthetic photos are useful for repeatable development and model smoke tests, but they are not evidence of actual physical contents. For the hiring submission, recreate the same controlled setup with real household objects and place the phone photographs in sample-data/initial/ and sample-data/corrected/.

Quality checks

npm test
npm run lint
npm run build

The server validates model output before returning it to the browser. It downgrades unsafe conclusions to Unverified when:

• a confirmation lacks exact SKU, quantity or image evidence;
• a quantity mismatch lacks enough distinct UNIT-XX IDs;
• an identity mismatch lacks a visibly different SKU;
• a result references the wrong document row;
• a conclusion claims an item is missing, absent or not delivered from photographs alone.

Architecture

1. The browser keeps the original previews but converts large photos to bounded JPEG uploads before building multipart form data.
2. A Next.js server route validates the prepared file type, count and size.
3. The OpenAI Responses API reads the PDF and photos in a single multimodal request and returns strict JSON with document rows, conclusions and percentage bounding boxes.
4. Deterministic server rules validate identity, count evidence, row references and missing-item language.
5. The browser displays the row result and overlays the returned bounding box on the supporting image.
6. Token usage and wall-clock time are converted to an estimated variable cost and shown with the result.

Scope and limits

• English only.
• One-page text PDF, maximum five product rows and 1 MB.
• One to three source photographs, maximum 10 MB each; browser-prepared uploads are capped at 900 KB per photo for the hosted request limit.
• Printed SKU and UNIT labels required.
• No warehouse integration, supplier complaint automation, accounts or payments.
• Bounding boxes come from the vision model and can be approximate; they are evidence pointers, not metrology.
• No automatic API retries in the prototype. A failed operation is shown to the user and costs must be measured separately if manually retried.
• The public demo applies a best-effort in-memory limit of 12 valid analysis runs per IP per hour. Serverless cold starts can reset this counter, so the OpenAI project budget remains the hard spending backstop.

Reused components and original work

Reused: Next.js, React, OpenAI JavaScript SDK, ReportLab, ESLint and TypeScript.

Original for this assignment: the capture convention, multimodal prompt, strict response schema, safety validator, evidence UI, cost instrumentation, controlled test set, expected outcomes and tests.

See docs/DELIVERY_NOTES.md for measured results and docs/VIDEO_SCRIPT.md for the walkthrough.

The deployed app also serves the generated 1:25 walkthrough at /proofpack-walkthrough.mp4.