Repository navigation
Teach local skills by demonstration and add qualification labs - #3
Merged
Merged
Conversation
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
System1 can learn a bounded decision from demonstrations, save the skill, and reuse it locally. This PR adds document, courier and Pokémon teaching labs, optional improvements to teaching and inference, and reproducible n8n workflows with explicit review and teacher fallback. The n8n quality gauntlet remains unqualified; merging these optional experiments does not establish production readiness for those workloads.
Product changes
choice_solver="logistic"is an opt-in Python compiler option. SciPy is needed during teaching; saved choice heads retain NumPy inference and separate calibration. Ridge remains the default. Receipt opt-out works on fresh decisions, uncached execution avoids unused digests, and conformal tail lookup preserves inclusive ties.Merge preparation
Final sanity review is complete in
f0758dd, following merge preparation in84a30e6and the source audit in4184246. The README, guides, papers and indexes agree on source-only features and unqualified experiments. No version bump, tag or package release is included.f0758dd(final CI): package, Ubuntu Python 3.11–3.14, and macOS Python 3.13.mainrevision1ddf14c. Main requires all six checks on an up-to-date branch, including for administrators; force pushes and deletion are disabled.reflexcompatibility import. No SciPy, scikit-learn, ONNX Runtime or MLX was installed in that environment.Qualification remains unfinished
The preregistered scope is all 150 CLINC intents plus unfamiliar requests and all 77 BANKING77 intents. Targets remain at least 80% supported local coverage, at least 99% correctness among accepted supported requests, at most 1% CLINC unfamiliar false acceptance, and complete warm single-request adapter p95 below 5 ms.
The original frozen evaluation and latest observed-test regression retain all 17,160 decisions apiece, accepted errors, per-intent counts, intervals, timings and candidate identities. Conventional baselines are faster but still fail the 80% coverage requirement. Rescoring observed tests is not independent confirmation.
The latest seven-correction development experiment adds one accepted banking case for each method and does not improve the selected CLINC candidates. The new banking candidate has not undergone full-scope evaluation. All 10,110 actual saved-adapter responses match selection with zero runtime errors, teacher calls or new API costs. The retained failed reconstruction attempt, committed repair and exact standalone artifacts remain available. These small development gains do not supersede the failed full evaluations.
The gauntlet evidence index preserves every earlier experiment, matched baseline, failure and cost record. Inference measurements include text encoding, classification, review and response construction, with OS networking denied and caches/receipts disabled; they exclude HTTP/n8n transport. Encoder weights are unchanged.
Demonstrations and limits
The real n8n disconnection rehearsal used the exact saved banking review artifact: two Gemini fallback responses, then a stopped teacher process and ten decisions with zero new provider attempts. Local responses remained identical; uncertain requests and local-service failure went to review. Raw evidence, 2,176 tokens, a $0.000699 price estimate and an execution screenshot are retained. This is an unqualified integration rehearsal, not the promised qualified recording.
Other labs remain bounded examples: document suggestions get 29/30 raw and 27/27 accepted ordinary cases correct, but accept one of six unusual cases incorrectly; courier strict completion improves from 22/40 to 31/40 fresh maps. Pokémon's 144 controlled runs across 18 edited fixtures move wins from 6/18 to 7/18 and strict wins from 2/18 to 6/18; handwritten rules match seven wins and are faster. These are not campaign-play claims.
Independent full-scope confirmation is still missing. The source inventory records the available human supplements and their scope/mapping limitations; reserved samples remain unscored. Passing that confirmation, connecting the exact qualified artifact and recording its teacher disconnection remain separate unfinished research work. They are not claimed as delivered by this merge.
The subsequent expert banking-source audit checks pinned 231- and 385-row expert-selected files. They add zero new normalized requests and no changed labels; all rows occur in the original training data, with some original-test overlap. The independent collection brief preserves the full taxonomies and original cohort sizes. No new teaching or scoring occurred, and no contributors have been engaged.