Skip to content

Teach local skills by demonstration and add qualification labs - #3

Merged
steph4n-gh merged 58 commits into
mainfrom
codex/teachable-skill-network
Sep 21, 2026
Merged

steph4n-gh merged 58 commits into
mainfrom
codex/teachable-skill-network

Conversation

@steph4n-gh

@steph4n-gh steph4n-gh commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

System1 can learn a bounded decision from demonstrations, save the skill, and reuse it locally. This PR adds document, courier and Pokémon teaching labs, optional improvements to teaching and inference, and reproducible n8n workflows with explicit review and teacher fallback. The n8n quality gauntlet remains unqualified; merging these optional experiments does not establish production readiness for those workloads.

Product changes

  • The document workspace turns filing choices and corrections into examples, checks different documents and exports a saved skill. Changing a lesson invalidates stale suggestions until the skill is taught again. Courier and Pokémon labs explore related teaching and composition ideas.
  • choice_solver="logistic" is an opt-in Python compiler option. SciPy is needed during teaching; saved choice heads retain NumPy inference and separate calibration. Ridge remains the default. Receipt opt-out works on fresh decisions, uncached execution avoids unused digests, and conformal tail lookup preserves inclusive ties.
  • Importable n8n workflows preserve item identity, validate the decision contract and send uncertain, malformed or failed requests to review. The live-teacher rehearsal uses an identified saved development artifact.
  • Keep the research evidence in Git, outside Python distributions. The source archive falls from about 317 MiB to 8.6 MiB; the wheel remains about 189 KiB. CI checks archive contents and size. Generated evidence is collapsed in GitHub's default diff; curated result summaries remain visible. About 465 MiB of gauntlet files still remain in the repository.

Merge preparation

Final sanity review is complete in f0758dd, following merge preparation in 84a30e6 and the source audit in 4184246. The README, guides, papers and indexes agree on source-only features and unqualified experiments. No version bump, tag or package release is included.

  • Required CI: all six jobs passed on final head f0758dd (final CI): package, Ubuntu Python 3.11–3.14, and macOS Python 3.13.
  • The branch contains the current main revision 1ddf14c. Main requires all six checks on an up-to-date branch, including for administrators; force pushes and deletion are disabled.
  • Clean wheel installation and CLI smoke passed outside the checkout, including both protocol assets and the reflex compatibility import. No SciPy, scikit-learn, ONNX Runtime or MLX was installed in that environment.
  • The built source archive's document example runs using that minimal environment. Browser verification covered a manual filing choice, adding the sample session without overwriting it, teaching, a fresh batch, visible sample-answer evaluation, and correction invalidation.
  • Focused core/lab/integration checks: 126 passed, four ROM-dependent skips. A separate run with the user-provided Pokémon Red ROM passed all seven ROM tests, including those four cases.
  • Documentation checks now validate links to this repository’s main branch as well as relative paths; publishing repeats documentation and archive checks.
  • Strict distribution metadata checks, lockfile validation and documentation checks pass (92 Markdown files, 648 checked links and nine version labels). The new size guard also rejects the original oversized archive.
  • All 147 files in the latest research freeze retain their hashes. Packaging preparation changes no candidates, datasets, evaluation labels, targets or results. ROMs and credentials are excluded.

Qualification remains unfinished

The preregistered scope is all 150 CLINC intents plus unfamiliar requests and all 77 BANKING77 intents. Targets remain at least 80% supported local coverage, at least 99% correctness among accepted supported requests, at most 1% CLINC unfamiliar false acceptance, and complete warm single-request adapter p95 below 5 ms.

System1 evaluation Supported coverage Correct among accepted supported Unfamiliar false acceptance Complete p95
First held-out CLINC 3,757/4,500 (83.49%) 3,721/3,757 (99.04%) 59/1,000 (5.9%): fails 1.82 ms
First held-out banking 2,516/3,080 (81.69%) 2,485/2,516 (98.77%): fails No native cohort 4.39 ms
Latest observed-test CLINC regression 3,956/4,500 (87.91%) 3,903/3,956 (98.66%): fails 46/1,000 (4.6%): fails 2.65 ms
Latest observed-test banking regression 2,524/3,080 (81.95%) 2,492/2,524 (98.73%): fails No native cohort 3.48 ms

The original frozen evaluation and latest observed-test regression retain all 17,160 decisions apiece, accepted errors, per-intent counts, intervals, timings and candidate identities. Conventional baselines are faster but still fail the 80% coverage requirement. Rescoring observed tests is not independent confirmation.

The latest seven-correction development experiment adds one accepted banking case for each method and does not improve the selected CLINC candidates. The new banking candidate has not undergone full-scope evaluation. All 10,110 actual saved-adapter responses match selection with zero runtime errors, teacher calls or new API costs. The retained failed reconstruction attempt, committed repair and exact standalone artifacts remain available. These small development gains do not supersede the failed full evaluations.

The gauntlet evidence index preserves every earlier experiment, matched baseline, failure and cost record. Inference measurements include text encoding, classification, review and response construction, with OS networking denied and caches/receipts disabled; they exclude HTTP/n8n transport. Encoder weights are unchanged.

Demonstrations and limits

The real n8n disconnection rehearsal used the exact saved banking review artifact: two Gemini fallback responses, then a stopped teacher process and ten decisions with zero new provider attempts. Local responses remained identical; uncertain requests and local-service failure went to review. Raw evidence, 2,176 tokens, a $0.000699 price estimate and an execution screenshot are retained. This is an unqualified integration rehearsal, not the promised qualified recording.

Other labs remain bounded examples: document suggestions get 29/30 raw and 27/27 accepted ordinary cases correct, but accept one of six unusual cases incorrectly; courier strict completion improves from 22/40 to 31/40 fresh maps. Pokémon's 144 controlled runs across 18 edited fixtures move wins from 6/18 to 7/18 and strict wins from 2/18 to 6/18; handwritten rules match seven wins and are faster. These are not campaign-play claims.

Independent full-scope confirmation is still missing. The source inventory records the available human supplements and their scope/mapping limitations; reserved samples remain unscored. Passing that confirmation, connecting the exact qualified artifact and recording its teacher disconnection remain separate unfinished research work. They are not claimed as delivered by this merge.

The subsequent expert banking-source audit checks pinned 231- and 385-row expert-selected files. They add zero new normalized requests and no changed labels; all rows occur in the original training data, with some original-test overlap. The independent collection brief preserves the full taxonomies and original cohort sizes. No new teaching or scoring occurred, and no contributors have been engaged.

@steph4n-gh steph4n-gh changed the title Add a live playground for composing and correcting System1 skills Add teaching labs for composing and correcting local System1 skills Sep 21, 2026
@steph4n-gh steph4n-gh changed the title Add teaching labs for composing and correcting local System1 skills Teach, correct and qualify local System1 skills Sep 21, 2026
@steph4n-gh steph4n-gh changed the title Teach, correct and qualify local System1 skills Teach local skills by demonstration and add qualification labs Sep 21, 2026
@steph4n-gh
steph4n-gh merged commit 5c00910 into main Sep 21, 2026
6 checks passed
@steph4n-gh steph4n-gh mentioned this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant