Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 27 additions & 15 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,31 @@ See `docs/ARCHITECTURE.md` for full design principles and non-goals.
QVT α track. v0.2.0 (Foundation Hardening) and v0.3.0
(Platform Skeleton) both fully closed; v0.4.0 (Layer 4 Lifecycle:
T1+T7+T2 first bundle) shipped 2026-05-27.
- **Active theme**: **α-7 closure (5-tier mode reshape) + Track 2c
collaboration parallel + bidi gate security fix** (three concurrent
tracks as of 2026-06-02). α-7 PR #680 draft (graph top-K + threshold +
bucket-(d) phrases). Baseline showed -0.107 abst_f1 regression on
null queries at gemma4 production tier (outside noise); 5-tier
remeasurement in flight (background) to test mode-pattern reshape
across 5 tiers (1b/4b/12b/27b/e4b). Track 2c (Ali Arabic adversarial
18 cases) PR #681 draft — preserved imports + design + schema remap
+ sweep runner + 29-unit-test bidi audit. Bidi runtime gate PR #682
draft — `core/input_normalization.py` + `routes/query.py` wire +
29 tests. α-6 cycle closed 2026-06-01 PM (PR #678).
Sequence: α-7 closure → α-8 typed-filter (evidence-of-absence
preservation rule R1-R5 baked in) → Phase 3b proper on α-8 baseline
per Option C (2026-06-02 decision). Domain pilot (v0.5) still gated.
- **Active theme**: **α-8 ontology typed-filter** (R1-R5 evidence-of-
absence preservation, architectural fix for α-7's wrong-knob). Phase
A (PR #688 `fcf343d`) + Phase B (PR #689 `6d6698e`) landed
2026-06-02:
- `core/ontology.py` +5 horizontal types (event/date/location/
quantity/project) + 6 relations (OCCURRED_AT/HAPPENED_ON/
LOCATED_IN/INVOLVES/MEASURED_AS/WORKED_ON)
- `core/graph_typed_filter.py` new module + `JAMES_DISABLE_TYPED_FILTER`
flag (disable-polarity, default OFF = filter ON)
- Matrix cell `C_rag-ontology` between `C_rag-graph` (now pre-α-8
baseline with flag forced) and `C_rag-full`
- `pipeline_context.build_unified_context` prepends typed entity
summary BEFORE graph context (byte-additive)
- 137 tests pass; ruff clean
- Default behavior post-merge: typed filter ACTIVE in `/query/`

Phase C N=3 baseline at M_M background launched 2026-06-02 (ETA
2.2h). Phase C 5-tier remeasurement + Phase D closure analysis +
`docs/ARCHITECTURE.md §5.7` typed filter section = 다음 세션.
α-7 cycle closed as REJECT 2026-06-02 (PR #680, 10th wrong-fix-
averted). α-6 closed 2026-06-01 PM (PR #678). Track 2c Phase 4
M_M sweep complete 2026-06-02 (0 breach / 11 resisted / 6 partial /
1 manual_review — α-8 acceptance test target poison_01 identified).
Sequence resumes: α-8 Phase C+D → 5-tier remeasurement → Phase 3b
proper on α-8 baseline. Domain pilot (v0.5) still gated.
- **Strategic frame**: We are still building a **mother platform**;
v1.0 vertical domain branching remains the only authorised
specialisation point. The 2026-06-01 strategic discussion narrowed
Expand Down Expand Up @@ -116,7 +127,8 @@ See `docs/ARCHITECTURE.md` for full design principles and non-goals.

| Purpose | File |
|---|---|
| **🟢 NEXT SESSION ENTRY (read this first)** | **`docs/handovers/v0.4-next-session-entry-2026-06-02-post-tracks.md`** |
| **🟢 NEXT SESSION ENTRY (read this first)** | **`docs/handovers/v0.4-next-session-entry-2026-06-02-evening.md`** |
| Previous entry (2026-06-02 post-tracks — now sealed) | `docs/handovers/v0.4-next-session-entry-2026-06-02-post-tracks.md` |
| Previous session entry (α-7 launch — sealed) | `docs/handovers/v0.4-next-session-entry-2026-06-02-alpha7.md` |
| Previous-previous (Phase 3a launch — sealed) | `docs/handovers/v0.4-next-session-entry-2026-06-01-PM.md` |
| Track 2c integration design memo | `docs/design/v0.4-track-2c-arabic-adversarial-integration.md` |
Expand Down
175 changes: 175 additions & 0 deletions docs/handovers/v0.4-next-session-entry-2026-06-02-evening.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# JAMES — Next Session Entry (2026-06-02 evening)

> Cold-start orientation for the next Claude Code session. Sealed
> 2026-06-02 evening after α-8 Phase A+B landed and Phase C M_M
> sanity launched in background.

## 0. TL;DR (30 sec read)

α-7 closed as **REJECT** (10th wrong-fix-averted, PR #680). α-8 architectural
fix landed in **two PRs the same day**:

- **PR #688** `fcf343d` — Phase A: ontology +5 horizontal types + typed
filter module + flag (`JAMES_DISABLE_TYPED_FILTER`)
- **PR #689** `6d6698e` — Phase B: matrix cell `C_rag-ontology` +
pipeline_context typed-prefix overlay

Default behavior post-merge: typed filter **ACTIVE** in `/query/`. Phase C
N=3 baseline at M_M kicked off in background 2026-06-02 (ETA ~2.2h);
output log `reports/research-runs/alpha8-phase-c-sanity.log`, per-cell
JSON in `reports/research-runs/qvt-ablation-cells/`.

Other parallel landings (5 PRs merged this session):
- **PR #684**: 30-error ruff F-class cleanup (main CI debt cleared)
- **PR #685**: Bucket-(d) 11 oracle phrases (gemma3 family refusal coverage)
- **PR #686**: Track 2c Ali Arabic adversarial Phase 1-3 + α-8 R1-R5 update
- **PR #687**: Track 2c Phase 4 M_M sweep results (0 breach + poison_01
acceptance target)
- **PR #682**: Bidi gate runtime input normalization (Track 2c X3 response)

Total: 8 PRs landed + 2 PRs closed (#680 REJECT, #681 superseded by #686).

## 1. First commands when you sit down

```powershell
cd C:\Project\James-RAG-Evol-v010
git status
git log --oneline -10

# Check Phase C background run status (running or completed?)
type "C:\Users\karu-\AppData\Local\Temp\claude\C--Project-James-RAG-Evol-v010\c47eaf4c-729a-45b2-a903-1825b34fc205\tasks\bt5gdm2jw.output"

# Inspect per-cell JSON outputs if completed:
ls reports\research-runs\qvt-ablation-cells\

# If Phase C run was killed / incomplete, restart:
python scripts/qvt_ablation_matrix.py `
--sector-cells C_rag-graph,C_rag-ontology `
--tiers M_M --n-runs 3 --resume
```

## 2. State of α-8 (4-phase plan)

| Phase | Status | Notes |
|---|---|---|
| **A** ontology + typed filter module + flag | ✅ PR #688 merged | core/ontology.py 14.9 KB / core/graph_typed_filter.py 10.7 KB |
| **B** matrix cell + pipeline_context wiring | ✅ PR #689 merged | C_rag-ontology between C_rag-graph (forced flag=1) and C_rag-full |
| **C** N=3 baseline (M_M) | ⏳ background launched 2026-06-02 | log/cells in reports/research-runs/; verify completion before next launch |
| **C-extend** 5-tier remeasurement | ⏸️ pending | M_XS/M_S/M_M/M_L/M_XL × n=3 × 2 cells = 30 cells ≈ 6-10h compute |
| **D** closure analysis + ARCHITECTURE.md | ⏸️ pending | Per-question-type cross-tab + R1-R5 acceptance check (multihop null + Ali poison_01/04) |

## 3. Acceptance tests for α-8

Per design memo §2.4:

1. **Multihop_RAG null query test**: re-run `scripts/research/audit_12b_null_query_refusal_shape.py` against post-α-8 baseline at M_M. Pass if null-query abst_f1 stays ≥ α-6 baseline level (0.609). Fail if regression (= same R1+R2 gap as α-7).

2. **Track 2c poison_01 + poison_04 retest**: re-run `scripts/adversarial_sweep.py` against M_M with α-8 active (default ON). Pass if poison_01 verdict promotes from `manual_review` to `resisted` (LLM emits explicit "leather jacket not in catalog" statement). Fail if poison_01 stays `manual_review` or worse.

Phase 4 sweep results (pre-α-8 baseline for comparison):
- 11 resisted / 6 partial / 0 failed / 1 manual_review (poison_01)
- 0 auto-failure across 18 cases
- File: `reports/research-runs/adversarial-sweep-M_M-20260602T084119.json`

## 4. Quality Delta Card framework (Phase D)

Baseline reference (per design memo §1.4): **C_rag-graph** (post-α-7 baseline = pre-α-8 path with `JAMES_DISABLE_TYPED_FILTER=1` forced in sector cell).

Verdict tree:
- `graded Δ ≥ +0.030` → ⭐⭐ adopt + ARCHITECTURE.md + finding memory promoted
- `+0.010 ≤ graded Δ < +0.030` → tier-gated adopt (which tiers per cross-tab)
- `graded Δ ≤ +0.010 OR abst_f1 < α-6 baseline` → ⭐ operational only (reject typed filter as architectural fix; heuristic top-K declared sufficient on this benchmark, memory promoted as such)

Per design memo §1.3 honest framing:
- Even if Δ exceeds noise, finding is ⭐⭐ partial (typed filter is decades-old technique; MS GraphRAG / LlamaIndex have type hints). JAMES contribution = numbers + routing rule + empirical position.
- ⭐⭐⭐ candidate elevation requires cross-fixture sanity (multihop + Ali Track 2c crossover) — Phase D incorporates this.

## 5. Phase 3b proper status

Per `feedback_graph_layer_top_k_debt`: publishable graded claim requires α-7 closure → α-8 measurement → cross-family sanity. As of session close:

- α-7 closure: ✅ REJECT
- α-8 baseline measurement: ⏳ Phase C in flight
- Cross-family (5 families × n=3 ≈ 45h compute): ⏸️ post-α-8 Phase D
- Position per design memo §1.4 + 2026-06-02 Option C decision: Phase 3b proper runs **on α-8 baseline**, not pre-α-8.

## 6. Track 2c collaboration status (Ali)

| Phase | Status | File |
|---|---|---|
| 1: preserved imports + integration design | ✅ PR #686 | `eval/adversarial/ar_ecommerce-{README-ali,REPORT-provia,provia-results,v1.1-pending}.{md,json,yaml}` + `docs/design/v0.4-track-2c-arabic-adversarial-integration.md` |
| 2: YAML remap (byte-exact U+202E preservation 18/18) | ✅ PR #686 | `eval/adversarial/ar_ecommerce-v1.1-james.yaml` |
| 3: adversarial_sweep.py + DSL parser (18 tests) | ✅ PR #686 | `scripts/adversarial_sweep.py` + `tests/test_adversarial_criteria_parser.py` |
| 4: JAMES M_M sweep | ✅ PR #687 | analysis: `reports/research-runs/track-2c-phase-4-jam-m_m-analysis-20260602.md` |
| 5: cross-stack report (Ali vs JAMES diff) | ⏸️ post-α-8 acceptance | |
| 6: α-8 acceptance integration (poison_01/04 retest) | ⏸️ Phase D | |
| 7: Ali cross-stack DM | ⏸️ post-Phase 6 | |

## 7. Wrong-fix-averted ledger

| Cycle | Count |
|---|---:|
| α-5 | 7 |
| α-6 | 2 (rate-limit + tier override) |
| α-7 | 1 (top-K rejected at 5-tier scale) |
| **Cumulative** | **10** |

Additional category catch this session: 1 "cumulative branch contamination" (PR #681 superseded by #686) — recognised pattern; logged in PR #686 description.

## 8. Files added this session (Where to find what)

| Topic | Path |
|---|---|
| α-8 design memo | `docs/design/v0.4-alpha-8-ontology-typed-filter.md` (§2.4 R1-R5; §3 integration plan; §4 verdict tree) |
| α-7 closure analysis | `reports/research-runs/alpha-7-closure-analysis-20260602.md` |
| Track 2c Phase 4 analysis | `reports/research-runs/track-2c-phase-4-jam-m_m-analysis-20260602.md` |
| Track 2c Phase 4 raw JSON | `reports/research-runs/adversarial-sweep-M_M-20260602T084119.json` |
| Bidi audit | `reports/research-runs/bidi-normalization-audit-20260602.md` |
| α-7 bucket-(d) sub-finding | `reports/research-runs/alpha-7-bucket-d-oracle-phrase-gap.md` |
| α-6 recovery curve | `reports/research-runs/alpha-6-phase-3a-recovery-curve-20260601.md` (§3.1 has 12b correction annotation) |

## 9. Memory updates this session

New entries (4):
- `project_alpha_7_closure_state` — α-7 REJECT context
- `feedback_alpha7_top_k_destroys_positive_contributions` — wrong-knob mechanism
- `feedback_s4_citation_survives_context_reshape` — ⭐⭐⭐ candidate strengthened
- `feedback_12b_pure_capability_misclassified_as_plateau` — 12b plateau correction
- `project_alpha_8_phase_a_b_landed` — α-8 Phase A+B state (this session)

## 10. Open decisions left for next session

1. **Phase C completion**: verify background run completed; if partial, resume with `--resume` flag. If full, proceed to analysis.
2. **5-tier remeasurement scope**: full M_XS+M_S+M_M+M_L+M_XL × n=3 vs subset (e.g., M_M + M_XL only). Decision driver: Phase C M_M result magnitude.
3. **Phase D closure timing**: write closure analysis after 5-tier OR after just M_M+M_XL. Trade-off: completeness vs cycle velocity.
4. **Ali Phase 5 DM share**: wait for Phase 6 acceptance test (poison_01 retest at α-8) OR share Phase 4-5 results immediately for cross-stack discussion.
5. **Phase 3b proper launch position**: per design memo §1.4 Option C confirmed (post-α-8 baseline). Compute timing depends on Phase C completion.

## 11. Don't-forget guardrails

- **No domain features until v1.0** (CLAUDE.md rule #1) — α-8 5 new types are all horizontal (event/date/location/quantity/project); regulation/transaction/recipe deferred.
- **20 KB module gate** — `core/ontology.py` now 14.9 KB; further additions need split-first.
- **Quality Delta Card required** for `core/retrieval|graph|reasoning` PRs (CLAUDE.md rule #2). Phase B PR #689 used `Quality delta: exempt (label: code — A/B oracle infra)` — Phase D PR will land the actual Δ numbers.
- **Honest framing tier** for every finding (memory `feedback_finding_size_honest_framing`). α-8 ceiling = ⭐⭐ partial.

## 12. Korean handover snippet

이번 세션 (2026-06-02): α-7 REJECT 확정 (PR #680, 10번째 wrong-fix-averted)
+ α-8 Phase A+B 두 PR 한 세션 머지 (#688/#689) + Track 2c Phase 4 sweep
(0 breach, poison_01 acceptance target 명시) + 5 PR cleanup cascade
(#683/#684/#685/#682/#686/#687).

α-8 Phase A: `core/ontology.py` 에 horizontal 5 type + 6 relation 추가,
`core/graph_typed_filter.py` 신규 모듈 + flag 추가. R1-R5 evidence-of-
absence preservation rule 모두 코드로 박힘. 137 test pass.

α-8 Phase B: matrix cell `C_rag-ontology` + `pipeline_context` typed-
prefix overlay. Default ACTIVE in production (flag 미설정 시 filter ON).

Phase C N=3 baseline at M_M background 실행 중 — log 위치
`reports/research-runs/alpha8-phase-c-sanity.log`. 다음 세션 cold-start
시 완료 여부 확인 → 결과 분석 또는 재실행.

다음 cycle 순서: Phase C M_M 분석 → 5-tier 확장 측정 → Phase D closure
analysis + ARCHITECTURE.md §5.7 → cross-family Phase 3b proper (45h)
→ v0.5 도메인 (horizontal moat = 기업 내부지식).
Loading