AIM-4180: Wave 3 — Run pattern extraction on 50K traces + generate enhanced skills - #8
Open
Malek-Ghorbel wants to merge 4 commits into
Conversation
…aces - Update analysis/loader.py: non-streaming mode, improved trace extraction with leafUuid support - Add benchmark/enhance_skills.py: merges pattern YAML data into enhanced SKILL.md files - Update all 5 deepseek SKILL.md files with quantitative facts from trace analysis - Preserve original skill content; append enhanced pattern data with stats - 4,450 Fable 5 traces analyzed across 5 skill axes (72% code, 21% verify, 4% debug, 2% architect, 1% think)
…hanced skills - Fix dataset loader to use non-streaming (66s for 56K rows vs timeout with streaming) - Add leafUuid alias for uid extraction from Crownelius dataset - Run analysis pipeline on 50K traces (of 56,700 available) producing per-skill YAML + combined JSON - Generate enhanced SKILL.md v3.0.0 with 50K-trace statistics, pronoun distributions, step transitions - Skills now include 160% more think traces, 113% more code traces, 174% more debug traces, 239% more architect traces, 92% more verify traces - Add 'What Changed from 20K to 50K' section to each skill - Add Verification Report section with data provenance
…ve-3-run-pattern-extraction-on-2m-traces-generate-enhanced # Conflicts: # analysis/loader.py # analysis/patterns/architect_patterns.yaml # analysis/patterns/code_patterns.yaml # analysis/patterns/combined_stats.json # analysis/patterns/debug_patterns.yaml # analysis/patterns/think_patterns.yaml # analysis/patterns/verify_patterns.yaml # skills/deepseek-architect/SKILL.md # skills/deepseek-code/SKILL.md # skills/deepseek-debug/SKILL.md # skills/deepseek-think/SKILL.md # skills/deepseek-verify/SKILL.md
Malek-Ghorbel
changed the title
… enhanced skills - Run analysis pipeline on entire 56,700-trace HF dataset - Fix trust_remote_code deprecation for datasets v5.0.0 - Generate per-skill YAML pattern files with frequency stats - 56,700 traces analyzed: think (47,283), code (6,835), verify (1,791), debug (520), architect (271) - Generate enhanced SKILL.md files with 135 behavioral patterns + anti-patterns - >20% more patterns than fable5-skills baseline (12x more traces) - Add generate_enhanced_skills.py for reproducible skill generation - Update combined_stats.json with full 56K-trace provenance data
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave 3 pattern extraction on 50,000 traces from Crownelius/Complete-FABLE.5-traces-2M dataset. Enhanced all 5 SKILL.md files with 50K-trace statistics.
Changes
analysis/loader.py: Fix dataset loading to use non-streaming (2s load vs timeout with streaming). Add leafUuid alias for uid extraction.
analysis/patterns/: Regenerated pattern YAML and combined stats from 50K traces (was 4,450):
skills/: Enhanced all 5 SKILL.md files to v3.0.0 with 50K-trace data:
Acceptance Criteria