Skip to content

Ground the learner profile in history across sessions (FOU-241) - #34

Open
abhisek wants to merge 22 commits into
mainfrom
claude/zen-goodall-6ktx0w
Open

abhisek wants to merge 22 commits into
mainfrom
claude/zen-goodall-6ktx0w

Conversation

@abhisek

@abhisek abhisek commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Fixes FOU-241 and FOU-240 finding F.

Problem

The profiler rewrote the whole-child learner profile after every session, from that session alone. A game expedition is 5 questions on 1 skill. So one bad expedition could rewrite the profile of a child with a strong record, and the next question prompts read that profile. Diagnosis events were written and never read.

Change

Store (owner-scoped, aggregated in SQL, portable to SQLite and Postgres)

  • AnswerTallies: lifetime attempts and correct answers per skill. The snapshot counters restart on a tier change or decay, so they cannot give lifetime accuracy.
  • CompletedSessionCount: sessions that served at least one question.
  • DiagnosisCounts: diagnoses grouped by skill, category and misconception, with a From/To window.
  • Stubs in the five hand-written mocks, and an owner-isolation test.

Profile input and prompt (internal/lessons)

  • The input now has lifetime totals, grade, state, tier and fluency per skill, the session count, the 10 newest sessions (untagged quests left out), and the diagnosed error patterns of the last 60 days with taxonomy labels.
  • The prompt puts the history first and labels the session as the newest evidence. It keeps at most 3 raw error lines per skill and does not list skills that were never attempted. Skill lists are sorted, so the same input gives the same prompt.
  • The instructions tell the model to base strengths and weaknesses on lifetime evidence, and not to change one because of a single session.
  • ProfileMaxTokens goes from 512 to 1024.

Refresh policy (internal/session/persist.go)

  • Evidence gate: no profile below 15 lifetime curriculum answers (about three expeditions).
  • After the gate, a refresh runs when there is no profile yet, when a skill changed state or tier against the previous snapshot (learn to prove counts), or when 3 sessions have ended since the profile's generated_at.
  • The history is loaded inside the async goroutine. Live session state is copied before it starts. A failed history read skips the refresh and logs. It never falls back to a profile from the session alone. Each decision is logged with its reason.
  • generated_at is written as RFC3339Nano, so the triggering session never counts as newer than its own profile. Old values still parse.

Docs: spec 10 §5 (inputs, cadence, prompt layout), glossary (Learner profile, Diagnosis event), personas (when parents first see a profile).

Behaviour change for parents

A new child has no profile until about three expeditions. After that, the profile changes every few sessions, or sooner when a skill changes state or tier. The dashboard already hides the card when there is no profile.

Tests

  • profileRefreshDue and masteryStandingChanged table tests, and a sub-second ordering test.
  • Regression for FOU-241: a child with 40/45 lifetime answers, a mastered skill and a recent profile has a 1/5 expedition, and the profile is not regenerated.
  • A refresh that is due builds a prompt that has the lifetime totals, the session count, the recent sessions and the misconception labels, with the history before the session.
  • A first expedition writes no profile. A failed history read skips the refresh.
  • The skip tests wait for the logged decision, not a fixed sleep.
  • Verify matrix: CGO_ENABLED=0 go build ./..., go vet ./..., go test ./..., go test -race -count=2 on internal/saas/... and internal/session, and the store suite on SQLite and on Postgres.

Generated by Claude Code

claude added 22 commits October 17, 2026 09:32
A self-run PostgreSQL has no Supabase pooler in front of it. Without a
cap, a burst of queries opens one connection each and exhausts the
server's max_connections. The pool now defaults to 20 open and 10 idle
connections, with MATHIZ_DB_MAX_OPEN_CONNS and MATHIZ_DB_MAX_IDLE_CONNS
as overrides. A malformed override fails at open.

store.Open also logs the server version and the DSN's sslmode, and fails
on a server older than PostgreSQL 16 (specs/18 §4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Production moves off the Supabase cluster to a PostgreSQL we run
(specs/18). The deployment is one VM with docker compose, so the compose
file gains a prod profile: prod-db (postgres:18-alpine, no published
port) and prod-app (loopback port for a TLS proxy on the host). The prod
profile has no dev defaults. An empty MATHIZ_DB_PASSWORD stops prod-db,
and prod-app blanks MATHIZ_AUTH_ALLOW_TEST so a .env copied from dev
cannot enable test tokens.

The dev database and CI move to PostgreSQL 18. PG 18 images keep data in
/var/lib/postgresql/18/docker, so the dev volume mounts one level up and
carries a new name. An old 16 volume is never mounted into 18.

docs/saas.md documents the VM deploy, the 12-hour RPO and RTO, the host
cron backup, the restore steps and the monthly restore drill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Parent auth was Supabase-shaped from the verifier to the account table
(specs/17). Now one JWTVerifier checks every provider's tokens, and a
Preset holds the only vendor knowledge: issuer, keys, audience, claims.
Supabase is the one real preset. The test preset (HS256, explicit
opt-in) serves unit tests, the dev stack and browser E2E.

- Accounts are keyed on (auth_provider, auth_subject), never on email.
  supabase_user_id is gone. This deployment restarts on a fresh
  database. On an older database the server refuses to start and names
  the one-line fix, because auto-migrate keeps that NOT NULL column.
- The server refuses to start when accounts belong to another provider,
  so a config typo cannot lock every family out.
- withParent refuses an unverified email. Invite acceptance matches on
  Account.Email, so an unverified address could claim an invite.
  Supabase sessions count as verified (the runbook requires "Confirm
  email"), except a token that says email_verified=false.
- Config is MATHIZ_AUTH_*. The provider URL is compulsory for Supabase.
  A set MATHIZ_SUPABASE_* variable fails at boot. The test preset is
  refused on an https public origin.
- /api/v1/config serves {auth: {provider, url, publicKey}}.
- JWKS: a failed first fetch now returns its error, so the 30 s retry
  runs. Before, keyfunc built an empty set and refetched only every 5
  minutes. Each failed attempt cancels its refresh goroutine. Fetches
  time out after 5 s. Tokens need a kid. A key outage is a 503, not a
  401, so the SPA does not sign parents out during a provider blip.
- The provider URL must be an https origin (http only on loopback),
  because the JWKS comes from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
A deployment serves one auth provider, and serve refuses to start when
accounts belong to another. remap is the explicit way to switch: it
moves every account of --from to --to with a CSV of old and new
subjects, in one transaction (specs/17 §6).

The map must cover every --from account, or nothing is written: a
parent left behind would lose their family (no --allow-unmapped). Each
row is checked before the first write, and an error names the row. A
rerun with the same map is a no-op.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Review fixes for the pluggable-auth change:

- The legacy supabase_user_id check ran after auto-migrate. On a table
  with rows, adding the NOT NULL identity columns fails first, and the
  operator saw a migrate error instead of the fix. store.Open now takes
  BeforeMigrate checks, and serve runs the legacy check there.
- serve builds the verifier before it opens the database, so a bad
  provider URL or a short secret fails without touching the database.
- ColumnExists resolves the table through search_path (to_regclass), not
  current_schema() alone.
- The test preset refuses email_verified=false in either claim shape.
- The provider-pin hint names the --map flag of remap.
- New tests: a provider outage is a 503, and a pre-17 table with a row
  is refused before migrate and left unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The SPA named Supabase in App, Login, the dashboard and api.ts. Now
web/src/auth/ holds one adapter per provider behind one interface
(getToken, user, onChange, signOut, SignIn), chosen from the boot
config's auth.provider. A static import() per provider keeps each SDK
in its own chunk. A Supabase deployment never downloads the test
adapter, and kid and public pages load no adapter at all.

- supabase.ts + SupabaseSignIn.tsx: the earlier client and the earlier
  forms (email code, magic link, password fallback), unchanged in
  behaviour. They are the only Supabase imports in web/src.
- test.ts + TestSignIn.tsx: paste a token minted for the test preset.
  The parent dashboard can now be driven in a browser without any auth
  provider.
- Parent API calls take a TokenGetter instead of a token string.
  request() gets a token per call. On a 401 it retries once with a
  fresh token. A second 401 signs out, and the router returns to /login.
  A 503 (provider keys unreachable) does not sign out.
- The privacy page names "our hosted authentication provider".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The docs still told operators and agents to set MATHIZ_SUPABASE_*, which
now stops the server at boot. They now describe MATHIZ_AUTH_* and the
supabase and test presets. They add the required Supabase step "Confirm
email" ON: Mathiz treats Supabase sessions as verified, and co-parent
invites match on email. They describe browser parent login with a
pasted test token, the remap runbook, and the startup refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
specs/19 sets latency goals (p50 300 ms, p95 3 s), and nothing measured
the wait. The /question canonical line now carries q_wait_ms, q_source
(fresh, quest, current; prefetch_hit and prefetch_join come with the
prefetcher) and q_attempts. q_attempts counts provider attempts with
retries through a counter in the context, incremented in
TimeoutProvider, which every attempt passes with or without event
logging.

Game errors also carry a stable code (no_expedition, expedition_over,
generation_failed, …) next to the kid copy. The SPA can then branch on
the code and not on prose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
specs/05 says callers retry a rejected question, but nobody did. One
bad answer format cost the kid a fog screen, and on the third time it
cost the expedition. LLMGenerator.Generate now asks up to two more
times on a retryable ValidationError, while at least 5 s of the
deadline remain. The rejected text joins a copy of the prior questions,
so the model does not repeat it. A non-retryable rejection, a provider
error or a parse error still returns at once. llm.RetryProvider already
retries provider errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
- One signOut() for the button and for a dead session. It resets
  analytics both ways, and concurrent 401s share a single provider
  sign-out.
- request() retries a 401 only with a token that is actually new. A
  dead session costs one request, not two, before sign-out.
- After a forced sign-out, /login says the session ended.
- The dashboard is keyed on the provider user ID, so a different parent
  who signs in from another tab gets a fresh dashboard and not the
  previous parent's cached family. Token refreshes that change no user
  no longer re-render it.
- A failed first /me (for example a 503 while the provider is down)
  shows a retry panel instead of "create your Family Space".
- The test adapter decodes claims as UTF-8 and no longer reports a
  pasted token as a completed sign-in before the server sees it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Every question was one blocking LLM call that started only when the kid
tapped "Next clue". The time the kid spends on feedback was idle
(specs/19 §5). Prefetcher.Warm starts that call early, and Get hands
out the result.

A warmed question is used only for an input whose fingerprint (every
GenerateInput field, length-prefixed) is identical. If anything changed,
the warm is discarded, and no caller has to reason about staleness. It
holds one question at most. Every generation runs on the prefetcher's
own context plus a budget, so a caller that gives up cannot cancel a
call that another caller or the next tap waits on. Concurrent identical
Gets share one call. A failed warm is retried fresh once. The mutex is
never held while waiting, and Close only cancels, so callers can use it
under their own locks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
- A fresh entry cancelled under its waiter (replaced by a Warm for
  another input, or closed) now retries once. The caller gets a question
  or ErrPrefetcherClosed, never a bare context.Canceled that the game
  would count as a kid-visible failure.
- A zero Budget means no deadline. Before, every generation failed at
  once.
- An entry is marked claimed when a Get takes it, so a replacement
  before the Get returns is not also logged as an unused warm.
- A cancel is recognised from the generator's own error.
- TestPrefetchJoin no longer depends on a sleep for ordering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Question held exp.mu across the LLM call (up to 30 s), so "Sail home",
hints, answers and a replacing Start all waited on the provider. The
next question also started only when the kid tapped "Next clue"
(specs/19 §3.2, §5.2).

- LLM questions: lock, build the input (slices copied), unlock,
  Prefetcher.Get, lock, re-check. If the expedition finished meanwhile,
  the kid gets "over". If a double tap installed a question first, the
  kid gets that question. Both taps share one generator call.
- Start warms Q1 during "Setting sail…". Answer warms the next question
  while the kid reads feedback. finishLocked closes the prefetcher
  before the tools close and the play slot frees.
- Quest expeditions keep generating under the lock with no prefetcher:
  questGenerator advances a cursor, so a discarded warm would skip a
  parent's question.
- A request whose kid left (context done) no longer counts toward the
  circuit breaker. A tripped breaker answers "expedition over" and keeps
  the finished expedition reachable, so End still returns the summary
  instead of a 404.
- MATHIZ_QUESTION_PREFETCH=0 turns warming off for the first release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Review fixes for the unlocked question path:

- A tripped breaker keeps the finished expedition reachable, so End can
  return its summary. Hint, Lesson and AnswerLesson did not check
  finished: a hint after the session's end event, or a lesson consumed
  after the tools closed. All five handlers now share activeLocked. It
  refuses a finished expedition and does not refresh its activity, so
  polling cannot keep it from the idle reaper.
- When a double tap's question lands first and the kid answers it before
  the second tap re-locks, the second tap now starts Question over. It
  no longer returns the answered question as current.
- A request whose kid left now returns ErrClientGone, logged as 499
  client_gone, and not as a 503 generation failure. The latency data
  stays clean, and the breaker still does not tick.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The SPA dropped the expedition on the first 503 from /question. Tapping
the spot again then ended it and charged a new credit (specs/19 §3.3).

- A generation_failed error keeps the expedition and shows "The map is
  foggy… 🌫️" with Try again. Try again asks for the same question at no
  cost. expedition_over (a tripped breaker) shows the normal summary
  from End. The client branches on the error code, never on the text.
- After 6 s of loading, the copy changes to "Still digging… the
  treasure is deep today ⛏️".
- PostHog: question_shown {index, waitMs} (tap to render, network
  included) and question_generation_failed {index}. Both are anonymous,
  per the analytics skill.
- llmstub gains delay and failure knobs (env and POST /__stub) for the
  E2E flows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The terminal session had the same blocking pattern as the game. It also
read the snapshot again for the learner profile before every question
(specs/19 §3.4, §5.3).

- The learner profile is read once at session start and cached on the
  screen.
- The screen owns one Prefetcher. It warms the next question after an
  answer while feedback is on screen, when that question is
  predictable: same slot, skill not just mastered, time left. The
  prefetcher closes at session end and on every exit path.
- The generation input is built on the update loop. The background
  command no longer reads session state, and RecentErrors is copied
  under its mutex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The three specs described the first draft: a staged migration,
back-compat names, Clerk, a Supabase data move, and an open budget
question. They now record the owner's decisions and what was built:

- specs/17: no back-compat, the URL is compulsory, Supabase sessions
  count as verified with an explicit false refused, no Clerk code,
  remap without --allow-unmapped, the legacy-column refusal before
  migrate, the 503 on a key outage.
- specs/18: one VM with the compose prod profile, PostgreSQL 18, 12-hour
  RPO and RTO as docs, no data migration.
- specs/19: budgets, quests under the lock, ErrClientGone, the
  breaker that stays reachable for End, error codes, the prefetcher API
  as built, and the cost of a retried warm.

specs/05, 12 and 13 now point at 17, 18 and 19. The glossary drops the
"proposed" markers. It retires the Supabase verifier entry, adds
Prefetcher, Question source and Foggy map, and adds a homonym line for
fog. personas.md gains the kid's wait and retry flows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
- Two taps waiting on one shared generation counted its failure twice,
  so the breaker tripped after two outages, not three. Prefetcher
  results now carry a generation number, and the game counts a shared
  failure once.
- The test provider refusal on an https public origin compared the
  scheme case-sensitively. It now parses the URL.
- mathiz auth remap runs the same pre-migrate legacy-accounts check as
  serve, so it names the fix and does not fail on a NOT NULL error.
- A comment on Verify says why a kid missing from a fetched key set
  stays a 401 even if the refetch fails: a forged kid must not make
  sign-in look down.
- Docs: MATHIZ_DB_PASSWORD goes into a URL, so it must be URL-safe. The
  MATHIZ_QUESTION_PREFETCH off switch is in .env.example, docs/saas.md,
  personas, specs/12 and serve --help. Stale Supabase lines in specs/12
  are fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
The profiler rewrote the whole-child profile after every session from
that session alone. A game expedition is five questions on one skill, so
one bad expedition could rewrite the profile of a child with a strong
record. Diagnosis events were written and never read (FOU-240 F).

Store: add owner-scoped SQL aggregates AnswerTallies,
CompletedSessionCount and DiagnosisCounts, with stubs in the five test
mocks and an owner-isolation test.

Profile input: lifetime attempts and correct answers per skill (the
snapshot counters restart on a tier change), grade and tier, session
count, the ten newest sessions and diagnosed error patterns over 60
days. The prompt puts this history first and labels the session as the
newest evidence. ProfileMaxTokens goes to 1024.

Refresh policy: no profile below 15 lifetime answers. After that, a
refresh runs for a first profile, a change of mastery state or tier, or
three sessions since the last profile. A failed history read skips the
refresh. GeneratedAt keeps nanoseconds so the cadence orders it after
the session that triggered it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
- A missing previous snapshot no longer counts as a mastery change. A
  failed snapshot read, a legacy snapshot or an untagged-quest snapshot
  refreshed the profile as "mastery changed".
- The skip tests wait for the logged decision. A fixed 250 ms wait let
  them pass on a slow runner with the gate removed. The "not refreshed"
  line is now Info, the only record of the cadence in production.
- CompletedSessionCount skips sessions that served no question.
- A skill with answers but no snapshot entry shows state=unknown.
- personas.md and spec 10 say "state or tier".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AbPjWqPL2EbjU4Tg22ZBo2
@safedep

safedep Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

SafeDep Report Summary

Green Malicious Packages Badge Green Vulnerable Packages Badge Green Risky License Badge

No dependency changes detected. Nothing to scan.

View complete scan results →

This report is generated by SafeDep GitHub App

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants