Release 1.6: universal local document imports and hardening - #130
Conversation
Store/Schema: - Transaction rollback safety on Windows (commits_deferred flag) - Skip redundant index.upsert when index shares store vector table - Embedding-space contract validation and direct SQL matrix path - Restore orphaned Store methods (prompt_eligibility_counts, embedding_space_health, context_savings_grouped, add_sync_bytes, get_sync_stats) and harden _logical_digest for sqlite-vec - Dashboard startup self-check for orphaned Store methods - v11 handoff column migration for sessions table Release readiness: - Strip dead TEAM scope surface (Scope.TEAM, MemoryRecord.team_id, SearchFilter.team_id/caller_id) per AGENTS.md §0 - Remove team_id column from memories table (team_members retained) - Remove conflict_aware profile (zero callers/tests/docs) - Add format/group_by params to MemoryService.context_savings() Other: - watch_repo key name, LLM fallback chain cost tracking - context_savings_grouped SELECT columns - Sync robustness, response budgets, docs sync
- MCP tool engraphis_context_savings: add format and group_by params - v2 API route /context-savings: add format and group_by query params - Read-only API /context-savings: add format and group_by query params - MCP HTTP CLI: minor fix
- eval/extractor_quality.py: offline extractor distillation quality harness - eval/handoff_quality.py: structured session-handoff eval - eval/datasets/handoff_quality.jsonl: handoff scenario fixtures - eval/datasets/sample.jsonl: 3 new entries (link_symbol, hierarchical consolidation, extractor/handoff patterns) - scripts/watch_repo.py: file-watcher for code-graph reindexing - scripts/validate_compose_contract.py: schema contract validator
- dashboard_app.py: pass workspace to stats() when allowed_workspaces is configured (prevents ValidationError on workspace-bound instances) - Sync classic_assets and static bundles with latest dashboard changes
- test_cli_entrypoints: clear ENGRAPHIS_WORKSPACES env for subprocess isolation - test_dashboard_security_headers: monkeypatch.delenv ENGRAPHIS_WORKSPACES - test_savings: adjust for context_savings format/group_by signature - test_sync: align with sync robustness hardening - test_update_check: align with 2-part semver acceptance - test_mcp_server, test_smart_mcp_gateway: align with MCP surface wiring - test_store_class_integrity: new Store method integrity coverage - test_dashboard_v2, test_release_infrastructure, test_secret_hygiene: minor alignment with pro-feature changes
The restored method from b7e80c2 predated the hash-validation contract in _public_receipt_row. Without prev_hash and receipt_hash in the SELECT, every receipt was marked invalid_payload and excluded from aggregation, causing the grouped method to return 0 groups despite valid receipts. Verified: all 4 dimensions (workspace/repo/agent/day) now return correct token aggregation. Savings ratio: 0.6 (180 saved / 300 source). Found by round-3 ContextSavingsGroupedRuntime scout.
Save JSON reports to a directory with timestamped filenames. Cherry-picked from codex/fix-pr-104-codeql-v4 (a4928c5).
… preservation, retention SQL, migration safety, port parsing - llm/client.py: _LLMProviderError accepts *args + message kwarg (fixes TypeError on cost-ceiling path) - llm/client.py: parse_provider_chain detects URL ports via :// prefix check (prevents port-as-ceiling stripping) - core/sync.py: quarantine merge preserves prior reasons alongside inherited_quarantine (audit trail) - stores/vectors.py: upsert_memory preserves existing memory_type when caller passes default semantic - routes/memory.py: AVG(stability) SQL approximation for retention stats (avoids OOM on large workspaces) - routes/vault.py: hashlib suffix on smart-import split doc_ids (collision prevention) - routes/vault.py: health_overview and find_stale bounded to 10k limit - service.py: Windows-safe two-step rename with staging file and rollback for v1->v2 migration - stores/__init__.py: graph backfill uses _replace_support_rows + per-namespace rebuild (O(N) not O(N²)) - config.py: embed_dim downstream consumers use 'is not None' pattern - 6 downstream embed_dim call sites fixed (dashboard_app, mcp_server, read_only_api, inspector, v2_api, embedder) - eval/longmemeval_v2.py: _stored_memory_type_counts scoped to workspace - eval/harness.py: --output-dir writing inside try/except for clean CLI errors - KILO_CODE_INTEGRATION.md: tool count corrected to nine; engraphis_forget row added - .env.example: ENGRAPHIS_SYNC_TOKEN_ORIGIN documented - Test file trailing newlines restored
…verage Closes P1 gap flagged by FinalA_Tests review: - port-stripping fix (localhost:8080 vs ceiling) — 5 cases - multi-provider chain parsing with mixed ceilings - empty/whitespace fallback to default chain - _LLMProviderError positional+kwarg construction contract
… harness improvements - .github/workflows/release.yml: CodeQL config-file reference added - .env.example: ENGRAPHIS_SYNC_TOKEN_ORIGIN documented for standalone sync tokens - docs/KILO_CODE_INTEGRATION.md: Smart tool count corrected to nine; engraphis_forget row added to Classic table - engraphis/dashboard_assets/vendor/d3.min.js: unified across classic/static (no new Function) - eval/longmemeval_v2.py: _stored_memory_type_counts scoped to workspace - eval/harness.py: --output-dir writing inside try/except for clean CLI errors - tests/e2e/demo.spec.js, test_dashboard_vendor_assets.py, test_documentation_contracts.py: new coverage - deploy/force-graph license/yarn lock files added - docs/benchmark-evidence offline fixtures added - Multiple doc updates: AGENTS.md, BENCHMARKS.md, CHANGELOG.md, README.md, SECURITY.md, SYNC.md - Demo infrastructure: screen demo HTML, prepare script, record script - Integration updates: hermes plugin, pi MCP client - Skills: engraphis-memory SKILL.md and references updated
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5aaa943bd5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a2110b64e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df587ce27a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Pre-PR Code Review — engraphis#130Structural AssessmentThis is a massive release PR (17,354 additions / 4,830 deletions across 62 files, 50+ commits) covering:
CI Status: ✅ All 23 checks greenCodeQL, test+lint (Python 3.9–3.14), coverage, typecheck, encryption driver gates, Pi extension (Ubuntu + Windows), browser accessibility, docker build/smoke, wheel build, repo hygiene — all passing. Security ObservationsThe PR body documents thorough safety measures:
Hard Gate Blocks (cannot approve yet)
Verdict: REQUEST_CHANGESThe code quality appears high based on CI coverage and iterative review history. Once the contributor gate is satisfied (e.g., another agent authors a refinement commit or the contributor requirement is waived for release PRs), this is approvable. Suggested next steps:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7183db882e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee933a3c57
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…allback paths - Wrap engraphis_get_memory post-inspect body in try/except with _classify_gateway_exception to match all other Smart tools, preventing internal error details from leaking through FastMCP. - Fix malformed SQLite URI on Windows in _keyword_search and memories routes: use Path.resolve().as_uri() + '?mode=ro' instead of bare string interpolation, matching the store's URI construction pattern.
… warnings - Fix README 1.5 release notes anchor link to match CHANGELOG heading format - Add missing CHANGELOG entries for dashboard version display, MCP error redaction fix, Windows SQLite URI fix, graph layers enforcement, and header parse warning - Apply _graph_csv() limit enforcement to /graph endpoint layers parameter, matching all other graph endpoints (64-item, 200-char limits) - Add stderr warnings to _parse_headers for silent ENGRAPHIS_LLM_EXTRA_HEADERS misconfiguration instead of silently dropping invalid headers
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0cb8e658d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…\n- Reject symlinks on import folder and individual candidates\n- Verify resolved folder stays under allowed roots via normcase+startswith\n- Use resolved_folder for relative_to to prevent escape via symlink
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a159c21fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
MCP server and LLM client import pydantic/httpx at module load, which breaks the numpy-only core floor when those extras are absent. Guard both imports so the core remains importable without optional extras, matching the deterministic hashing-embedder floor that CI enforces. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
Closes the last open review concerns before merge: - XLSX worksheets and PPTX slides now follow the workbook/presentation relationship order (r:id via the rels parts) instead of numeric order, with a deterministic fallback when those parts are missing. - RTF \binN binary payloads are skipped without unbalancing the group stack; XHTML encoding honors the XML prolog before <meta charset>. - _walk_tree bounds directory entry sorting before the 10k file cap and marks oversized directories as incomplete scans. - import launchers guard worker.start() and mark the job failed instead of leaving it stuck running when thread start raises. - vault import_folder reads through an fd-safe helper (O_NOFOLLOW, identity and containment revalidation) to close the open/read TOCTOU. - watchdog watcher applies the same exclude policy as the polling backend before enqueueing events. - Store._write_operation joins an active defer_commits boundary instead of opening an unreleased inner savepoint; link reconciliation commits the batch it owns even when the connection reports no ownership. - ENGRAPHIS_LLM_EXTRA_HEADERS parse errors are value-free on stderr. - Tests: workbook/slide order, RTF bin, XHTML prolog, oversized directory, worker-start failure, sync relay origin binding, and external-vector equality for HLC conflict successors. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…gation - Add skip link, main-content focus target, and notice banner to both the v2 ledger and classic dashboards. - Surface a persistent runtime-savings summary (value, meta, rate) and a savings overview section on Today, with receipt-backed fallbacks when the estimate cannot load. - Rename the metric labels to "Live memories" / "All versions, including history" so the counts are unambiguous. - Keyboard support: arrow/Home/End navigation for library options, graph/provenance/manage tabs, and roving tabindex for role=option cards. - View routing via history.pushState/popstate with a ?view= parameter and focus management on view switch. - Dashboard tests and e2e spec updated for the new markup and keyboard behavior. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 909f71f264
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bumps [@playwright/test](https://github.com/microsoft/playwright) from 1.61.1 to 1.62.1. - [Release notes](https://github.com/microsoft/playwright/releases) - [Commits](microsoft/playwright@v1.61.1...v1.62.1) --- updated-dependencies: - dependency-name: "@playwright/test" dependency-version: 1.62.1 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jaixii <algorithmictradingsolutions@gmail.com>
…#134) Bumps [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) from 1.13.5 to 1.14.3. - [Release notes](https://github.com/xberg-io/tree-sitter-language-pack/releases) - [Changelog](https://github.com/xberg-io/tree-sitter-language-pack/blob/main/CHANGELOG.md) - [Commits](xberg-io/tree-sitter-language-pack@v1.13.5...v1.14.3) --- updated-dependencies: - dependency-name: tree-sitter-language-pack dependency-version: 1.14.3 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jaixii <algorithmictradingsolutions@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57599fa3ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ackfill - memory_health() falls back to a portable Python retention calculation when the SQLite build lacks SQLITE_ENABLE_MATH_FUNCTIONS (EXP), so the diagnostic keeps working on SQLCipher and minimal builds. - read-only API returns 413 for streamed/chunked bodies that exceed the limit: the receive hook raises an internal marker the middleware translates directly, since FastAPI's body parser would otherwise turn it into a generic 400 before the ValueError handler runs. - migration backfills jobs.session_id from the live v15 source manifest (not only staged v14 temp tables) so the v16 exact-session triggers never reject lineage for legacy session-scoped import jobs. The staged path shares the same backfill helpers. - Tests: streamed-oversize 413 regression; existing declared-oversize, health, and migration suites pass. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d39adb6be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| targets = self._secure_erase_targets(self.conn, memory_id) | ||
| current_rows = [] | ||
| for target_id in targets: | ||
| marker = self.get_memory_sync_export(target_id) | ||
| current = self._erase_memory_rows(self.conn, target_id, actor=actor) |
There was a problem hiding this comment.
When a synced equal-HLC conflict was published to a separately backed vector index, this new loop erases both the original and its conflict successors from the Store, but MemoryEngine.secure_erase() still calls index.delete([memory_id]) for only the original. The successor's sensitive derived vector therefore remains in the external backend while the response reports vector_index_cleanup="deleted", making breach-remediation status inaccurate; delete every discovered target ID externally or report incomplete cleanup.
AGENTS.md reference: AGENTS.md:L160-L161
Useful? React with 👍 / 👎.
2 participants
Summary
engraphis import documents PATHwith preview, trusted-local confirmation, resumable per-document writes, temporal revisions, idempotent re-import, conflict policies, and complete reportsWhy
Users need to bring existing unsorted documents into Engraphis locally and repeatedly without creating a hidden AI-only copy, invoking an LLM extractor, or making network requests. Existing generic folder ingestion did not provide safe preview, stable source identity, temporal updates, resumability, rich Markdown links, or durable per-file reporting.
Safety and correctness
vlt_sourceSupported inputs
Native parsers cover Markdown/Obsidian, plain text and logs, RST, HTML, JSON/JSONL, CSV/TSV, YAML, TOML, INI-style files, XML, RTF, common source code, DOCX, ODT, EPUB, XLSX, PPTX, ODS, and ODP. PDF, image, audio, and video use optional configured local extractors. Legacy OLE, encrypted/DRM, and unknown binary formats are rejected and reported rather than guessed.
Validation
tests/suite passedruff check .pyright— 0 errors, 0 warningsIntentional limitations
Conservative top-level YAML is parsed rather than arbitrary executable YAML. Dataview/plugins/macros/scripts and transclusion expansion are not executed. Attachments are catalogued but not copied. Scanned PDFs do not gain automatic OCR, ambiguous links remain warnings, and legacy OLE/encrypted/DRM files remain unsupported.