Arbiter

A self-hosted hackathon platform whose judging you can defend.

Teams sign up, form teams and submit projects. Organizers invite judges, weight a rubric and assign reviews. Judges score in isolation. The platform evens out harsh and generous judges with a published, tested statistical model, lets the public vote with abuse controls, and publishes results as a frozen, hashed snapshot with Ed25519 signed records that anyone can verify offline.

Built for the Dogfood 72-Hour Hackathon (Hackathon Raptors, 2026). MIT licensed.

docker compose up
# open http://localhost:8080

That is the whole setup: one container, SQLite in a named volume, seeded with the shared Dogfood fixtures on first boot. No cloud account, hosted database, CDN or external API is used at runtime, so it works with the network off once the image is built.

Official checker	acceptance-report.txt: 7/7 PASS, verified T1 T2 (run.py has no T3/T4 checks, see below)
Our T3/T4 probes	acceptance-report-extended.txt: 18/18 PASS
Tests	pytest: 76 tests (lifecycle, authorization matrix, judging math, voting, audit, migrations, both checkers)
Judging evidence	docs/NORMALIZATION-PROOF.md, reproducible with one command
Demo video	demo.mp4 (3:21: create, submit, judge, publish; also on the v1.0 release)
Docs	ARCHITECTURE.md, DATA-MODEL.md, JUDGING.md, THREAT-MODEL.md, docs/openapi.json

Logins (demo mode)

The seed step prints these when the container boots. Every seeded account (all fixture judges and team members) also works at /login with the password dogfood.

role	header for scripts	or sign in as
admin	Cookie: session=demo-admin	admin@arbiter.local
organizer	Cookie: session=demo-organizer	organizer@arbiter.local
judge A (fixture jdg_24)	Cookie: session=demo-judge-a	diego.herrera@example.org
judge B (fixture jdg_26)	Cookie: session=demo-judge-b	jonas.vogel@example.org
participant (captain of tm_01)	Cookie: session=demo-participant	priya1@example.org

Demo mode is switched on in docker-compose.yml (ARBITER_DEMO_SESSIONS=1). Turn it off for a real event: fixed tokens and a shared password are for the checker and the demo only.

What it does, by tier

T1 core

• Accounts (PBKDF2-SHA256 passwords, 240k iterations, per user salt), sessions stored as SHA-256 of the token, per event roles (organizer, judge, participant) plus a site admin.
• Create an event with tracks, prizes, a submission window, team size and judging settings.
• Teams: create, invite link (rotatable), join, leave, captain, one team per person per event, size limit enforced on the server (one team per person is also a database constraint).
• Projects: draft, edit (any team member), submit with required fields, withdraw back to draft.
• The deadline is enforced on the server for every write that matters (edit, submit, team changes) with 409 deadline_passed, regardless of what the UI shows.
• Public gallery with search, track filter and sort; project pages with team and comments.

T2 judging

• Judge invites by single use link bound to the invited email, or grant the role directly. Judges declare expertise tracks and conflicts of interest.
• Assignment as a min cost flow: every project gets its target reviews, conflicts are impossible, expertise is preferred, load is spread evenly, reruns only top up. Manual assignment too.
• Weighted rubric (criteria, weights, scale per criterion). Keys lock once scores exist; weights stay editable and the change is audited.
• Judge isolation is enforced in one policy function (authz.can_read_judge_scores): a judge asking for another judge's scores gets 403 from the API, not a hidden button.
• Live organizer progress view (per judge and per project, refreshes every 4 s).
• Normalization: five methods side by side, default is a ridge regularised judge offset model. Integrity checks flag the flat judge, outlier judges, single review judges, under reviewed projects and the duplicate submission. See JUDGING.md.
• CSV exports: results, scores, projects, assignments, audit, votes, comments.

T3 public

• Community voting with a per voter budget, one vote per project, not for your own team, and judges of the event do not vote.
• Each voter sees the ballot in their own shuffled order, stable across reloads.
• Tallies are hidden from everyone (organizers included) until voting closes. Closing early is audited.
• Cheating: rate limits on login, signup, votes and comments; flags for accounts created after voting opened, email aliases (a.b+x@gmail.com = ab@gmail.com), and many voters sharing one network (shown as context only, since venues share Wi-Fi). Flagged votes are shown, not silently dropped, and ranking uses trusted votes.
• Comments with duplicate suppression and organizer moderation.
• CSRF: cookie authenticated POSTs must be same origin JSON.

T4 stretch

• REST API under /api/v1 for everything the UI does, with bearer tokens for scripts. OpenAPI at /openapi.json and a readable reference at /api (no CDN needed).
• Webhooks signed with HMAC-SHA256 (X-Arbiter-Signature), retried, with a delivery log.
• Publishing freezes a results snapshot (SHA-256 shown publicly) and issues Ed25519 signed records for every judge and participant. Certificate pages at /records/{id}; verify offline with python tools/verify_record.py <record.json or url>. The public key is at /.well-known/arbiter-key.json.
• Embeddable gallery: an iframe (/embed/{event}) or a one line script (/static/widget.js).
• Bulk import (the fixture format or an Arbiter bundle) and full export.
• Hash chained audit log of every write, with tamper detection shown in the UI.
• Pairwise judging mode (Bradley-Terry with active pair selection), usable alongside the rubric.

The acceptance reports

docker compose up -d
python3 tools/acceptance/run.py .dogfood.toml --fixtures data/fixtures.json > acceptance-report.txt
python3 tools/extended_checks.py .dogfood.toml > acceptance-report-extended.txt

tools/acceptance/run.py is an unmodified copy of the official checker. It contains checks for T1 and T2 only, so for any team claiming T3 or T4 it prints note: claimed but not verified: T3 T4. That note is about the checker, not a failed check. We claim T1 to T4 in .dogfood.toml because we built them, and tools/extended_checks.py is our own evidence for T3 and T4: plain HTTP probes in the same style (ballot shuffling, hidden tallies, own team votes, CSRF, rate limits, API tokens, webhooks, signing key, forged record rejection, embed, import and export). It says clearly in its output that it is unofficial. The extended probes make a few harmless writes, so run them against a demo instance.

Development

python -m venv .venv && . .venv/bin/activate
pip install -r requirements-dev.txt
PYTHONPATH=src ARBITER_DEMO_SESSIONS=1 python -m arbiter      # http://localhost:8080
pytest                                                         # 76 tests
ruff check .
python tools/normalization_proof.py                            # regenerates docs/NORMALIZATION-PROOF.md
python tools/export_openapi.py                                 # regenerates docs/openapi.json

Schema changes go through Alembic (src/arbiter/migrations); a test fails if the models and the migrations drift apart. Reset the container's data with docker compose down -v.

Configuration is by environment variable (src/arbiter/config.py): ARBITER_DATA_DIR, ARBITER_DATABASE_URL (defaults to SQLite in the data dir), ARBITER_SEED, ARBITER_FIXTURES, ARBITER_DEMO_SESSIONS, ARBITER_DEMO_PASSWORD, ARBITER_RATE_LIMITS, ARBITER_WEBHOOKS, ARBITER_PUBLIC_URL, ARBITER_SESSION_DAYS, ARBITER_PORT.

Known limitations

Written down rather than discovered by a judge:

• One process. The rate limiter and the webhook dispatcher live in memory, which is exact for a single process and wrong for several. Scaling out needs a shared store (Redis or the database).
• SQLite by default. Fine for an event of a few thousand people. The ORM and migrations are portable, but PostgreSQL is untested.
• No email. Arbiter runs offline, so invites and password resets are links the organizer copies and sends. There is no self service password reset.
• Abuse detection is heuristic. It flags, it does not prove. A determined attacker with many real email addresses and networks can still stuff a public vote; the flags make it visible.
• Normalization has one offset per judge, not per judge and track, and it cannot rescue a project that only one careless judge saw. The Results tab flags those so organizers can add reviews.
• The audit chain is tamper evident, not tamper proof. Someone with write access to the database can rewrite the whole chain; the published head hash in exports and results is what catches that.
• Webhooks are retried three times in a background thread; deliveries still pending when the process stops are not resumed (they stay visible as pending in the delivery log). There is no private address blocklist for webhook URLs yet, so only organizers can register them.
• Inline styles are still allowed by the CSP (scripts are not: every inline script carries a per response nonce). Injected CSS cannot run code, but it could restyle a page.
• The docs page is our own, generated from the OpenAPI document, because Swagger UI loads from a CDN. Point any OpenAPI viewer at /openapi.json if you prefer it.

Repository map

src/arbiter/          application package
  judging/            pure functions: scoring, normalization, Bradley-Terry, assignment, integrity
  services/           business logic, one module per area; all policy calls go through authz.py
  routes/             api.py (REST) and pages.py (server rendered HTML)
  migrations/         Alembic
tests/                pytest suite
tools/                acceptance checker, extended checks, normalization proof, record verifier
data/fixtures.json    the shared Dogfood fixtures (seeded on first boot)
docs/                 normalization proof, OpenAPI document