Skip to content

feat(course): publish quiz assignments verbatim and verify by read-back - #166

Merged
workshop-maybe merged 11 commits into
mainfrom
feat/course-import-assignment-quiz
Sep 6, 2026
Merged

feat(course): publish quiz assignments verbatim and verify by read-back#166
workshop-maybe merged 11 commits into
mainfrom
feat/course-import-assignment-quiz

Conversation

@workshop-maybe

Copy link
Copy Markdown
Contributor

Summary

Teachers can now publish a quiz assignment from the CLI. andamio course import-assignment <course-id> <module-code> <file.json> takes the quiz envelope the FCB Fan Campus app grades client-side, validates it with the same rules that app enforces, sends only the assignment key of the module update, and then reads the module back to prove the stored value matches. Until now the only path was a hand-built curl with the JWT copied out of ~/.andamio/config.json, the same gap #62 closed for module creation.

The export/import round trip also stops destroying quizzes. course export used to run the Markdown converter over the envelope, match nothing, and write an empty assignment.md, which a later course import published as the assignment. A non-doc assignment is now written verbatim to assignment.quiz.json, course import <dir> sends that file back verbatim, and export followed by import of a quiz module is a server-side no-op.

Closes #165. Related: #59, #61, #62.

What a reviewer cannot see in the diff

  • Only the assignment is sent, and every metadata field is carried. db-api's processAssignmentUpdate overwrites title, description, image_url and video_url from the input unconditionally, so an omitted field is nulled, not preserved. The command fetches the module first and carries those fields; --title / --description override. A module with no assignment yet requires --title, and course import with a quiz file on such a module refuses for the same reason rather than storing an empty title.
  • Read-back is structural, and it knows when it cannot see. The gateway re-serializes from jsonb, so equality is reflect.DeepEqual over decoded values, never bytes. The teacher module list is a merged read that answers 206 with meta.warning when one backend is down, and it carries content only from db-api. A module missing from a degraded list is therefore a distinct outcome (errDegradedRead) from "not found": before the write it refuses to send, after the write it reports kind: verify naming the warning, and a module found with content is trusted regardless of the warning. course import refuses (and never creates) on the same signal.
  • New error kind verify, exit 1. A write the gateway accepted but whose read-back did not confirm the stored value. Success would hide that the stored value is wrong; server would hide that the module was modified. Additive; documented in help exit-codes, README, the context doc and CLAUDE.md.
  • Validation mirrors fcb-fan-engagement-app by hand. internal/quiz reuses the app's issue codes and control flow; testdata/quiz/ pins 26 fixtures labeled app or cli-additional, and SOURCE.md records the app commits they were copied from. Drift in the app is re-mirrored by hand, not detected by CI.
  • Both assignment files present is a hard error. assignment.md and assignment.quiz.json together fail at parse time; export removes the stale counterpart on a --force re-export so it does not manufacture that state.

Open decision for the author

andamio-app-v2's quiz validator enforces three rules the FCB app does not: empty-prompt (trimmed), empty-option-label, empty-option-value. The issue names the FCB app as the reference, so the CLI does not enforce them and the docs now say "FCB Fan Campus app" rather than "the Andamio app". If quizzes published here must also render in app.andamio.io, those three rules should be added with source: app fixtures. Recorded with the other review residuals in docs/residual-review-findings/feat-course-import-assignment-quiz.md.

Validation

go test ./... is green (cmd/andamio ~88s). New coverage: 8 quiz-package tests over 26 golden fixtures; export writes assignment.quiz.json and removes the stale counterpart; the export→import round trip yields a payload deep-equal to the stored quiz; import-assignment unit tests cover verbatim publish, metadata carry, --title override, title-required, every rejection before any request, dry-run, read-back mismatch (content and metadata), degraded pre-fetch and read-back in the gateway's real chain-only shape, Andamioscan-only warning still publishing, module-not-in-list as not_found, 503 read-back keeping its kind; binary-level tests pin the single-document kind: verify JSON, the dry-run envelope, stdout/stderr split and the expired-JWT fail-fast. Goldens refreshed for the new command, its flags and the new envelopes.

Not run: the live preprod check against an ON_CHAIN module. The stored session targets mainnet and is expired. The "assignments are editable in any module status" statement rests on the gateway and db-api source (the aggregate update soft-skips only SLTs on non-DRAFT modules); CHANGELOG and docs/COURSE-LIFECYCLE.md say so.

Plan: docs/plans/2026-09-05-001-feat-course-import-assignment-quiz-plan.md.

New concepts

Write-then-read-back verification against an opaque store. When a service stores a value it does not interpret (here content_json as jsonb behind a gateway that re-serializes it), a 200 on the write proves only that the request was accepted, not that what was stored is what you meant. Reading the value back through the same read path the consumers use and comparing it turns an assumption into a check. Two things make it honest here: the comparison is structural (decode both sides, reflect.DeepEqual) because byte identity is not a property the store offers, and every failure after the POST says the write was applied, so a caller never mistakes "unconfirmed" for "nothing happened". The read-back must also know when it is blind: a degraded merged read that omits the module is reported as unconfirmed, not as a mismatch and not as success.

// After the POST: re-fetch through the same list endpoint the pre-fetch used.
stored, err := fetchExistingModule(ctx, c, courseID, moduleCode)
// errDegradedRead -> VerifyError (applied, unconfirmed); other errors keep their kind but say "accepted".
if !reflect.DeepEqual(stored.Assignment["content_json"], sent) { return &apierr.VerifyError{...} }

Do not use it when the write returns the canonical stored representation itself, or when the read path is eventually consistent with the write path; then a read-back either duplicates the response or produces false mismatches.


Compound Engineering

workshop-maybe and others added 9 commits September 5, 2026 06:48
Implementation plan for publishing quiz-envelope assignments from the CLI,
plus a CONCEPTS.md entry for the Quiz Envelope.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cepted write

A write the gateway accepted but whose read-back differs, or came back
degraded, is neither a success nor a server failure: the module WAS
modified. Kind "verify" names that outcome. It shares exit 1 per the
main.go rule that kinds already distinguishable by name do not get their
own code. Documented in help exit-codes, README, the context doc and the
CLAUDE.md Failure Contract. (#165)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ad of an empty assignment.md

tiptapToMarkdown matched no node of a quiz envelope and wrote an empty
assignment.md, which a later course import published as the assignment.
A non-doc content_json is now written pretty-printed to assignment.quiz.json
and the stale counterpart file is removed on re-export, so export followed
by import cannot destroy a quiz or trip import's both-files error. (#165, #59)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… fixtures

Add internal/quiz, a dependency-free package that mirrors the Andamio app's
quiz-envelope.ts: Recognize (isQuizContentEnvelope) classifies a content_json
value as doc / quiz / other / not-an-object, Validate (validateQuizDefinition)
collects every violated rule with the app's exact issue codes, and Summarize
digests question count, pass threshold and question ids for --output json.

Three CLI-additional checks the app does not enforce get their own codes:
malformed-prompt, malformed-help, malformed-intro. Validate never panics on
malformed elements and never stops at the first issue.

testdata/quiz pins the rule set: 3 valid fixtures, 23 invalid fixtures each
with a .issues sidecar labeled source: app (same code set the app emits;
every validateQuizDefinition test case is mirrored) or source: cli-additional.
SOURCE.md records the upstream commits the mirror was read from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e assignment

A module directory may hold assignment.quiz.json in place of assignment.md.
The file is validated as a v1 quiz envelope through internal/quiz (every
violated rule listed, no bypass), sent as content_json without re-encoding,
and the module's existing assignment title and metadata are preserved by the
existing merge. Both files present is a parse-time error before any request.
--dry-run reports the quiz digest and --output json gains an additive
assignment_quiz object; the schema golden now scans internal/quiz so those
tags are pinned. fetchExistingModule surfaces the gateway's meta.warning so a
degraded 206 read is never mistaken for a module without an assignment. (#165)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d verifies it by read-back

andamio course import-assignment <course-id> <module-code> <file.json>
replaces the hand-built curl a teacher needed to publish a quiz assignment.
The file is validated before any request (every violated rule listed, no
bypass), only the assignment key is sent, and the existing title,
description, image and video URLs are carried because db-api overwrites
every field it is not given. After the POST the module is re-fetched: a
differing stored value or a degraded 206 read-back is kind verify (exit 1),
a failed re-fetch keeps its own kind, and every message says the update was
accepted. --dry-run / --show-payload follow the #61 convention; --output json
emits the ImportAssignmentEnvelope pinned in the schema golden. Joins the
expired-JWT fail-fast table as the eighth hand-rolled PreRunE. (#165)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ify kind

README command list and Quiz assignments section, COURSE-LIFECYCLE step 9,
CLAUDE.md and context-doc command tables, and the CHANGELOG Unreleased
entries. The published-module statement rests on gateway and db-api source;
the changelog says the live preprod check was not run. (#165)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…efusal, not_found mapping

Review findings on #165:
- The merged module list carries content only from db-api, so a db-api
  outage returns chain-only modules that fetchExistingModule treated as
  'not found' and an Andamioscan outage tripped the meta.warning guard on
  a complete module. fetchExistingModule now returns errDegradedRead when
  the module is missing from a degraded list; import-assignment maps it
  to the R7a refusal before the write and to kind verify after, course
  import refuses to send (and never creates), and a found module is
  trusted regardless of the warning. Tests use the gateway's real
  chain-only shape.
- course import with assignment.quiz.json on a module that has no
  assignment refuses with the R5 title-required error instead of storing
  an empty title.
- A module missing from the list is kind not_found (exit 2) in
  import-assignment, not the generic kind.
- CLAUDE.md exit-code table row order; docs name the FCB Fan Campus app
  as the validator the CLI mirrors; new text-summary test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@workshop-maybe

Copy link
Copy Markdown
Contributor Author

Review — findings only, no verdict recorded (self-authored PR)

Conformance to andamio-cli#165: every acceptance criterion is met. CI green; go build, go vet and the full go test ./... pass locally at 366fba8.

Verified outside the diff:

  • db-api processAssignmentUpdate overwrites title / description / image_url / video_url unconditionally, so carrying the existing metadata is required, not optional. Confirmed.
  • db-api's aggregate update skips only SLTs on non-DRAFT modules ("other content remains editable in any status"). The "works on published modules" statement holds at the source level; the preprod ON_CHAIN run is still outstanding as the PR says.
  • testdata/quiz/SOURCE.md cites 3842a31 and 77aa833; those are the latest commits touching the two mirrored files in fcb-fan-engagement-app. Provenance is accurate.

Decisions before merge

  1. Validator reference: FCB app or the union with andamio-app-v2. andamio-app-v2 (src/lib/quiz/quiz-envelope.ts, last changed ca30c2a, 2026-07-07 — the residual doc's 69c7b57 is the repo HEAD read, not the file's commit) ships the same quiz UX plus three rules the CLI lacks: empty-prompt (trimmed), empty-option-label, empty-option-value. Any course on the gateway is viewable in app.andamio.io, so a quiz this command accepts can render there as the invalid-quiz fallback. Recommendation: enforce the union, with source: app-v2 fixtures. A stricter CLI never blocks a quiz that renders in both apps.
  2. Kind after an accepted write. course_import_assignment.go wraps a non-degraded read-back failure with %w, so apierr.Kind unwraps to the underlying error: a connection reset after a 200 reports unreachable (exit 5, documented as "the request never reached the service"); a JWT expiring between the two calls reports auth. A script branching on kind treats the quiz as unpublished. The deliberate choice is stated in the PR body; the counter-argument is that verify exists precisely for "applied but unconfirmed". Suggest returning VerifyError for every post-POST failure and adding Unwrap() so the cause stays inspectable.

Defects worth fixing in this PR

  • course import --create with assignment.quiz.json creates an empty module, then fails. The create POST (course_import.go:261) runs before the title-required check inside updateModuleContent (line 1462). --create --dry-run returns at the would_create_module branch and cannot predict it. import-all --create repeats this per quiz module. The condition (quiz file, no --title source, module absent) is fully known before any request; refuse in readCompiledModule / importModule before the create, or let course import take a title for the quiz.
  • Any meta.warning on a missing module becomes errDegradedRead, which blocks --create for the whole of an Andamioscan-only outage. When only Andamioscan is down the db list is complete and authoritative, so a genuinely absent module can no longer be created — a regression from before this PR. The comment above the guard describes the asymmetry the code does not implement. Distinguish the two warnings (refuse on unknown text) or ask the gateway for a structured degraded-backend field.
  • Read errors are treated as "file absent". readCompiledModule tests err == nil, not os.IsNotExist. An unreadable assignment.quiz.json (EACCES, EISDIR) silently drops the assignment from the payload and the command exits 0 with has_assignment: false. Conversely a zero-byte assignment.md trips the both-files error although the Markdown branch would ignore it. Return any non-ENOENT error; use len(assignBytes) > 0 for the conflict test.
  • Stale-counterpart removal sits inside if data.Assignment != nil. When the remote assignment was deleted, a previous export's assignment.md / assignment.quiz.json survives and the next course import republishes it. Pre-existing for .md, but CHANGELOG now states removal as a guarantee. Move the two removeIfExists calls outside the nil guard.

Minor

  • isNonDocContent in export is a second classifier beside quiz.Recognize, and its default direction is the lossy one: a type-less object goes down the Markdown path and exports as an H1-only file. Export one Kind classifier from internal/quiz and switch on it (Doc → .md, anything else → verbatim JSON).
  • --force export now removes a locally drafted assignment.quiz.json when the remote is still a doc (and vice versa) with no notice beyond "overwriting existing directory". A one-line stderr notice is cheap.
  • docs/residual-review-findings/ is a new directory. CONTRIBUTING ## todos/ says findings that are real but not being fixed now go in todos/NNN-<status>-<priority>-<slug>.md with frontmatter; the parked P1 belongs there so the triage sweep sees it.

Follow-up candidates (not this PR)

  • The unguarded course-modules/list scans in course.go, course_export.go, course_teacher_ops.go have the same 206 blind spot the plan scoped to import only; a shared findTeacherModule helper would close it.
  • existingString duplicates getStr (user.go); the assignment-metadata merge exists in two shapes; the hand-maintained fields table in verifyStoredAssignment will miss any new assignmentInput field silently.

Refuted during review: DeepEqual dropping explicit nulls on read-back (db-api stores content_json as an untyped JSONMap on jsonb and the gateway forwards it opaquely, so nulls survive).

…e, pre-create refusal, warning-aware degraded reads

Addresses the devkit review on #166.

Decisions taken as recommended:
- Validate against the union of both apps' rules. andamio-app-v2 adds
  empty-prompt (trimmed), empty-option-label and empty-option-value; any
  course is viewable in app.andamio.io, so a quiz the CLI accepts must
  render there too. Fixtures labeled source: app-v2; SOURCE.md records
  ca30c2a (the file's commit).
- Every failure after the accepted POST is kind verify, with the cause
  reachable through VerifyError.Unwrap. A reset connection after a 200 is
  not "the request never reached the service".

Defects:
- course import --create with assignment.quiz.json refuses before the
  create POST instead of leaving an empty module behind.
- errDegradedRead only when the warning names the db side. An
  Andamioscan-only outage leaves the db list authoritative, so a missing
  module is not_found and --create keeps working.
- Only ENOENT means "no assignment file"; unreadable files fail loudly. A
  zero-byte assignment.md never conflicts with a quiz file.
- Export removes whichever assignment file it did not write, including
  both when the remote assignment is gone, and says so on stderr.
- Export classifies through quiz.Classify (one classifier); a type-less
  object is preserved verbatim rather than flattened to an H1-only file.

Residuals move from docs/residual-review-findings/ to todos/039 and
todos/040 per CONTRIBUTING.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@workshop-maybe

Copy link
Copy Markdown
Contributor Author

Addressed in 88bb3b8. go build, go vet ./... and the full go test ./... pass.

Decisions — both taken as recommended

  1. Union validator. internal/quiz now enforces andamio-app-v2's empty-prompt (trimmed; also covers a missing or non-string prompt, which replaces the CLI's own malformed-prompt), empty-option-label (one issue per question, with the count) and empty-option-value (empty string only; a whitespace value is not empty, per app-v2's own test). Fixtures carry source: app-v2; testdata/quiz/SOURCE.md records ca30c2a, the file's commit, and explains the three labels. Help, README, CHANGELOG and the lifecycle doc now say "the union of the rules the two apps enforce".
  2. Kind after an accepted write. Every failure after the POST is VerifyError (kind verify), including a failed re-fetch, errModuleNotFound, and errDegradedRead. VerifyError gained Err + Unwrap(), so errors.As(err, &apierr.ServerError) still works on the cause. Test TestImportAssignment_ReadBackFailureIsVerifyWithCause pins it.

Defects

  • --create + assignment.quiz.json + absent module now refuses before the create POST (also under --dry-run); the message points at import-assignment --title. TestImportModule_CreateWithQuizAndNoModuleRefusesBeforeCreate.
  • errDegradedRead fires only when the warning names the db side (warningHidesDBContent: the gateway's strings start DB API unavailable, … or Andamioscan unavailable, …; unknown text is treated as hiding content). An Andamioscan-only outage leaves the db list authoritative, so a missing module is not_found and --create works. TestImportAssignment_AndamioscanOnlyWarningMissingModuleIsNotFound.
  • Read errors: only os.IsNotExist means absent; EACCES / EISDIR return an error. The both-files check uses len(assignBytes) > 0, so a zero-byte assignment.md never conflicts. Two tests.
  • Stale-counterpart removal moved outside the data.Assignment != nil guard: when the remote assignment is gone, both files go. TestWriteCompiledModule_NoAssignmentRemovesStaleFiles.

Minor

  • Export classifies with quiz.Classify (exported from internal/quiz); isNonDocContent is gone. A type-less object is preserved verbatim rather than flattened to an H1-only file (test added).
  • Removing a stale assignment file prints Removed <name>: it no longer matches the module's assignment on stderr in text mode.
  • docs/residual-review-findings/ is deleted. The parked items are todos/039-pending-p2-teacher-module-list-206-blind-spot.md (the shared findTeacherModule follow-up you named) and todos/040-pending-p3-quiz-assignment-residuals.md (concurrent-edit window, no read-back retry, no wire-shape fixture, drift detection, the duplicated metadata merge and verifyStoredAssignment field table, ExportResult structured field, and the still-outstanding ON_CHAIN preprod run).

Left as follow-ups per your note: the unguarded list scans in course.go / course_export.go / course_teacher_ops.go (todo 039) and existingString vs getStr (todo 040). PR stays open for your call.

…8bb3b8)

88bb3b8 landed only the fixtures and todos: its git add named the
residual-findings directory that the same command had just deleted, so
git rejected the pathspec and staged nothing else. This is the rest of
that change — the union validator (empty-prompt, empty-option-label,
empty-option-value), VerifyError.Err/Unwrap and verify for every
post-write failure, the pre-create refusal, warning-aware degraded
reads, read-error handling, quiz.Classify in export with stale-file
cleanup outside the nil guard, and the doc wording.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@workshop-maybe

Copy link
Copy Markdown
Contributor Author

Correction: 88bb3b8 carried only the fixtures and todos (a bad pathspec dropped the rest from the add). The code, tests and docs for everything above are in 19ecae7. CI is green on that head.

@workshop-maybe
workshop-maybe merged commit 2aa61fe into main Sep 6, 2026
4 checks passed
workshop-maybe added a commit that referenced this pull request Sep 6, 2026
The quiz-assignment work (#165, #166) adds a command, a module-directory
file and an error kind — all additive, so this is a minor bump, not the
patch it might look like from "one feature". A 1.0.1 tag advertising a new
command would misread the policy line at the top of this file and leave
the next genuine bugfix release without a natural number.

Promoting the section is also what unblocks `scripts/release.sh`: its
preflight hard-exits on a non-empty [Unreleased] with no versioned
heading, which is exactly this state.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: course import-assignment — publish a quiz (JSON content_json) assignment without raw curl

1 participant