[AutoFyn] autofyn/2026-04-10-9e09cc#71
Closed
kiwi0401 wants to merge 27 commits intoautofyn/2026-04-09-ad650ffrom
Closed
[AutoFyn] autofyn/2026-04-10-9e09cc#71kiwi0401 wants to merge 27 commits intoautofyn/2026-04-09-ad650ffrom
kiwi0401 wants to merge 27 commits intoautofyn/2026-04-09-ad650ffrom
Conversation
- Date spine: flip default from GLOBAL MAX DATE to fact/event table max. Agent now always uses the primary fact table's max date as spine endpoint and references the "← USE THIS" marker from get_date_boundaries. - JOIN type: make LEFT JOIN the explicit default for all JOINs. INNER JOIN now requires both a task-level signal and a compare_join_types tool call to confirm. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a model with current_date already has pre-computed data in the DB, the agent is now told to query that table's max date and use it as the replacement — instead of defaulting to the fact table max from get_date_boundaries. This preserves the original date range for calendar/spine models that were pre-materialized. Also fixes inconsistent GLOBAL MAX DATE reference in the warning block. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dbt is pip-installed to ~/.local/bin but agent subprocesses don't inherit user PATH. Without this, the agent wastes 3-5 turns searching for dbt. This ensures dbt is found immediately. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When agents encounter a locked DuckDB, they sometimes export/reimport to a new file, leaving _locked/_bak/_backup copies. The glob-based selection was picking files in arbitrary order, potentially evaluating the stale copy instead of the live one. Add _find_result_db() helper that filters out backup files and prefers the expected filename or largest file. Fixes netflix001 eval reading the wrong DB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rolling window/MoM/WoW models must output ONE date per entity, not all dates. Agent was building full time-series (airbnb001: 11135 vs 3). - Do not cast ID columns to different types (social_media001: INT→VARCHAR). - Read dbt_packages models before writing SQL to leverage pre-built columns. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merged the optional d2 step into d, making both validate_model_output and audit_model_sources mandatory after every dbt run. audit had 0% adoption as an optional step. Shortened the verbose sub-bullets to save prompt length. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace remaining bare glob patterns in _detect_precomputed_tables, _get_table_row_counts, and connection registration with the _find_result_db helper that filters backup files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Include per-table row counts and a "(largest table)" marker in the TABLE MAX DATES section. This helps the agent identify fact tables by size, complementing the existing date-based markers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agent was filtering out rows with NULL title/name from UNION results, but these rows have valid data in other columns. This caused netflix001 to produce 98 instead of 99 rows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agents were adding WHERE/HAVING filters based on table/column names (e.g., role='ACTOR' because table is 'actor_rating') instead of only filtering when the task description explicitly requires it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of holding a persistent write lock on the DuckDB file, the MCP connector now opens a transient read-only connection per query. This prevents lock conflicts when dbt needs write access between MCP queries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agent was interpreting descriptive phrases like 'based on the movies they appeared in' as justification for INNER JOIN, dropping rows with NULL ratings. Added explicit examples of what IS vs IS NOT exclusion language to prevent this misinterpretation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Agents using ROW_NUMBER() with insufficient ORDER BY columns produce non-deterministic IDs that cause downstream JOIN failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deterministic post-agent pass that rewrites INNER JOIN to LEFT JOIN in all non-ephemeral model SQL files before the final dbt run. Skips rewriting when the task instruction contains explicit exclusion language (e.g., "only", "exclude", "who have"). Scans only models/ directory, skips comments and ephemeral stubs. Filter stripping function implemented but disabled (too many false positives on legitimate staging filters). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After value-verify agent completes, check each eval-critical table for duplicate rows on its YML-defined unique key column. If duplicates exist (fan-out from JOINs), deduplicate using ROW_NUMBER(). Only fires when COUNT(*) > COUNT(DISTINCT key) — tables with correct row counts are never touched. Skips tasks with no unique test in YML. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emphasize that EVERY YML column must appear in SELECT, and add hints for deriving common column patterns (hour_*, day_of_*, *_months, etc.). Missing columns cause eval failure even when row count is correct. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… patterns When CHECK 2 finds missing columns, give the verify agent concrete steps: search source tables, derive from timestamp patterns (hour_X, month_X), handle _fivetran_synced. Block progression to CHECK 3 until columns match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Generate SELECT templates with all required YML columns as aliases, giving the agent a starting point that ensures all columns are included. Templates appear as comments in the REQUIRED COLUMNS section for models that need to be written from scratch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New _add_missing_columns() function adds YML-specified columns missing from eval-critical tables using three strategies: A) Derivation patterns (hour_X, day_of_X, etc.) B) Cross-table join with intelligent source selection C) NULL placeholder for _fivetran_synced/_fivetran_deleted - Also checks common metadata columns (_fivetran_synced) even when not listed in YML, using source tables in main schema - Source table selection prefers id→<table>_id primary key mapping - Move dedup + column adder to always run (even with --skip-agent) - Add RANK() vs DENSE_RANK() guidance to agent prompt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Gold data often preserves negative signs for charges/prices (accounting convention). The previous guidance to use ABS() caused twilio001 to fail on total_spend (-0.158 vs 0.158). Now instructs agent to keep source signs unless task explicitly says otherwise. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The verify agent's CHECK 7 was too aggressive — filtering rows where one column is NULL removes valid data. Restrict to only all-NULL rows and add explicit warning against IS NOT NULL filters. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
COALESCE(col, 0) is correct for COUNT/SUM aggregates from LEFT JOINs (e.g., count_visitors=0 when no events exist) but wrong for non-aggregate columns (names, dates, IDs). Previous guidance was too absolute — it caused pendo001 to produce NULLs where gold expects 0. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ines current_date replacement should only apply to date spines and WHERE clauses. For 'current_age' or 'days_since_X' calculations, the actual current date is correct. This fixes f1003 where driver_current_age was computed with the data max date instead of today. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The verify agent sometimes makes things worse by over-filtering, removing correct COALESCEs, or over-deduplicating. Add explicit rule to only fix issues with high confidence. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous guidance was too absolute — some summary models need ABS() for positive totals (twilio__account_overview) while detail models keep original signs (twilio__number_overview). Make it context-dependent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…m investigation
Gateway MCP subpackage (signalpilot/gateway/gateway/dbt/):
- New dbt_project_map and dbt_project_validate MCP tools for yml-direct
project discovery and dbt parse validation
- Modular subpackage (types, scanner, inventory, work_order, formatters,
validator, cache) — every file under 500 lines
- 43-test suite at tests/test_dbt_project_map.py covers discovery, broken
projects, topological sort, cycle detection, token budgets, and cache
- mcp_server.py exposes both tools as thin asyncio.to_thread wrappers
- store.py: Windows fcntl shim so the gateway imports cleanly on win32
Benchmark runner hardening (run_dbt_local.py + others):
- Prompt externalized to benchmark/prompts/dbt_local_system.md and
dbt_local_user.md (string.Template with ${var} placeholders so dbt Jinja
{{ ref('x') }} passes through unescaped)
- Project-file context dump removed from system prompt (~32k -> ~9k chars).
Agent uses dbt_project_map + Read/Glob on demand instead.
- MCP config injects PYTHONPATH and merges os.environ. Claude Code CLI
strips cwd from MCP stdio configs, so without PYTHONPATH the gateway
subprocess fails to import and shows up as {signalpilot: failed} at init.
- System prompt passed as {type: file, path: ...} to dodge Windows
CreateProcess 32k char argv limit (misleading CLINotFoundError).
- Default max_turns bumped to 200 across all runners; max_budget_usd removed
entirely. Validation loops are legitimate work; turn caps are safety only.
- allowed_tools whitelist removed from every runner. It was shadowing the
Skill tool and stranding MCP tools the prompt referenced.
- Premature max-turns break inside the message loop removed (SDK enforces
it; the runner was stranding the agent mid-write).
Docs (benchmark/docs/):
- non-determinism-investigation.md: full deep-dive on synthea001 cascade
(int__all_visits.sql non-deterministic ROW_NUMBER -> int__final_visit_ids
collisions -> visit_occurrence row loss -> int__cost_procedure drops 1
row -> cost 808 vs 809). Includes 15-version DuckDB sweep showing no
combination reproduces the gold, confirmation that the non-determinism
is inherited from OHDSI/ETL-Synthea upstream (line 113 of
AllVisitTable.sql), and a corpus-wide scan finding 30/68 tasks with
risky ROW_NUMBER patterns (only ~6-7 actually fail because of it).
- continuation-prompt.md: full onboarding doc for the next session
including a 12-check pre-flight checklist that catches every silent
infra failure we hit today (MCP connect, skills load, prompt length,
tool exposure, config regressions).
- progress.md and runs.md updated to reflect the ND investigation
findings and mark the 7 affected tasks with (ND) flags.
Reference material:
- benchmark/ref/Dockerfile.spider-eval: reproducible build env with
duckdb 1.3.1 + dbt-core 1.9.8 + dbt-duckdb 1.9.4 (versions current on
2025-06-26, the gold's last-update date) for the non-determinism
investigation. The ref/ clones themselves (spider/, synthea-omop-etl/)
are gitignored.
.gitignore:
- Adds benchmark/test-env/, _dbt_workdir/, _parse_test/, scratch/, and
the ref/spider and ref/synthea-omop-etl clones.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch:
autofyn/2026-04-10-9e09cc· Run:4d6f1307-33f0-400d-83d2-5a594fd4c415· Generated by AutoFyn