Repository navigation
fix(desktop): workspace scoping + returning-user org recovery + signal drill-down - #1
Closed
Bennettxai wants to merge 235 commits into
Closed
Bennettxai wants to merge 235 commits into
Bennettxai wants to merge 235 commits into
Conversation
Single-app Elixir project absorbing engine, knowledge, memory, and signal subsystems under a unified OptimalEngine.* namespace. Self-contained — no path deps to external repos. 689 tests passing, 29 excluded (RocksDB). Layout: lib/optimal_engine/ — core engine + knowledge, memory, signal subsystems lib/mix/tasks/ — 25 mix optimal.* commands test/ — full test suite (flattened, all under one namespace) docs/ — architecture, concepts, reference, guides, research config/ — env-driven paths, Ollama, hybrid search .github/workflows/ci.yml — OTP 26/27 x Elixir 1.17 matrix
Test fixtures write to tmp/ during runs; shouldn't be tracked.
Matches docs/architecture/ARCHITECTURE.md and PLAN.md Part 6. Module reorganization: lib/optimal_engine/pipeline/ — ingestion stages (intake, classifier, indexer, router, semantic_processor) lib/optimal_engine/retrieval/ — serving path (search, context_assembler, composer, intent_analyzer, l0_cache) lib/optimal_engine/store/ — SQLite + ETS (store, vectors) lib/optimal_engine/graph/ — graph + analysis (graph, analyzer, reflector) lib/optimal_engine/insight/ — learning + synthesis (verify, health, rethink, reweave, remember, simulate, memory_extractor) lib/optimal_engine/embed/ — embedding providers (ollama; whisper + provider stubs pending Phase 5) lib/optimal_engine/wiki/ — Phase 7 target lib/optimal_engine/tenancy/ — Phase 1 target lib/optimal_engine/identity/ — Phase 1 target lib/optimal_engine/audit/ — Phase 1 target lib/optimal_engine/connectors/ — Phase 9 target Kept flat (primitives + unchanged subsystem APIs): optimal_engine.ex application.ex cli.ex context.ex session.ex session_compressor.ex signal.ex topology.ex uri.ex knowledge.ex memory.ex Namespace rename map applied across lib + test + mix tasks. Multi-alias blocks rewritten to explicit per-line `alias ... as: X` where short forms needed preserving. Dead code removed (zero callers after audit): OptimalEngine.MCTS, MonteCarlo, CortexFeed optimal.spec.check (empty stub) optimal.simulate --plan / --mc options (MCTS/MC-dependent features; core impact report retained) Docs: PLAN.md — master execution plan covering Phases 0.5 through 13 docs/architecture/ARCHITECTURE.md — canonical 9-stage pipeline + 3 tiers docs/architecture/ENTERPRISE.md — tenancy, ACLs, connectors, retention, audit, perf targets docs/architecture/WIKI-LAYER.md — Tier-3 Wiki layer spec docs/architecture/UI.md — Tauri + SvelteKit desktop app spec Build verification: mix compile — generated (RocksDB NIF warnings pre-existing; optional backend) mix format — clean mix test — 689/689 passing (29 excluded, all :rocksdb-tagged)
Adds the enterprise-scale substrate per docs/architecture/ENTERPRISE.md and
PLAN.md Phase 1 acceptance criterion.
Storage — 16 versioned migrations (lib/optimal_engine/store/migrations.ex):
001 schema_migrations tracking table
002 tenants
003 principals, groups, roles, role_grants
004 acls
005 chunks + classifications + intents
006 assets
007 clusters + cluster_members
008 wiki_pages + citations
009 connectors + connector_runs
010 retention_policies + legal_holds
011 audiences
012 events (append-only audit log)
013 add tenant_id to existing primary tables (contexts, entities, edges,
decisions, sessions, vectors, observations) with DEFAULT 'default'
014 tenant-first indexes on every primary table
015 seed the default tenant
016 backfill :document-scale chunks from existing contexts
Migrations are idempotent (safe_execute tolerates duplicate-column/-index
errors) and forward-only (schema_migrations tracks applied versions).
Store.open_and_migrate/1 runs them after the baseline DDL so existing
deployments upgrade cleanly on first boot.
Primitives (minimal CRUD — no auth providers yet, those are Phase 9-11):
OptimalEngine.Tenancy.Tenant — get/create/list; default-tenant reserved
OptimalEngine.Identity.Principal — upsert/get; group + role resolution
OptimalEngine.Identity.Group — upsert; add_member/remove_member
OptimalEngine.Identity.Role — upsert; grant to principal OR group
OptimalEngine.Identity.ACL — grant; can?/3 permissive-default check
OptimalEngine.Audit.Event — struct + Event.new/2 builder
OptimalEngine.Audit.Logger — log/1, log/2, query/1; tenant-scoped
Search integration (lib/optimal_engine/retrieval/search.ex):
- `:principal` option added; when set, post-filters hits via ACL.can?/3
- `:tenant_id` option added (defaults to Tenant.default_id/0)
- Audit event emitted on every principal-scoped search ("retrieval.executed")
- Backwards-compatible: no :principal → no filter
Surfaces:
mix optimal.migrate — applies pending migrations
mix optimal.migrate --status — shows version map (✓ applied, · pending)
mix optimal.stats — extended with Phase 1 table row counts
Tests (18 new, 707 total):
test/tenancy/tenant_test.exs — 4 tests
test/identity/acl_test.exs — 8 tests (grant + can?/3 + role transitive)
test/audit/event_test.exs — 3 tests (write + query + principal filter)
test/retrieval/principal_filter_test.exs — 3 tests (principal-scoped search + audit)
Build verification:
mix compile — generated (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 707/707 passing (29 excluded, all :rocksdb-tagged)
mix optimal.migrate --status — 16/16 applied
mix optimal.stats — Phase 1 tables section shows row counts
Acceptance for Phase 1 per PLAN.md met:
✓ migrations apply cleanly from current state
✓ 689 existing tests still pass (+18 new = 707/707)
✓ stats shows row counts for all new tables
✓ search enforces principal-filtered hits when :principal option is set
Accurate status table, three-tier model, 9-stage pipeline, phase-by-phase progress, architecture tree, agent-runtime integration examples. Removes the stale Phase 0 framing. Points at PLAN.md as the single source of truth for forthcoming work.
Stage 2 of the ingestion pipeline, per docs/architecture/ARCHITECTURE.md.
Every input format — text, data, code, office, image, audio, video —
produces the same shape, which is what lets the Decomposer (Phase 3)
break anything down into hierarchical chunks without caring about source.
New contract (lib/optimal_engine/pipeline/parser/):
parsed_doc.ex %ParsedDoc{path, signal_id, text, structure,
assets, modality, metadata, warnings}
structural_element.ex %StructuralElement{kind, text, offset, length, metadata}
kinds: :heading :section :page :slide :timestamp
:code_block :paragraph :table_row
asset.ex %Asset{hash, content_type, size, path, modality,
metadata} with Asset.from_path/2 hasher
backend.ex @behaviour OptimalEngine.Pipeline.Parser.Backend
Dispatcher (lib/optimal_engine/pipeline/parser.ex):
Parser.parse/2 — file → ParsedDoc via the right backend
Parser.parse_text/2 — inline text with a :format hint
Parser.dispatch/1 — path extension → backend module
Parser.backends/0 — the registry
Backends (each is a small module on the Backend behaviour):
markdown.ex heading + paragraph + code-fence structure, line-offset tracked
text.ex blank-line paragraphs, handles .txt .rst .adoc .log + fallback
yaml.ex yaml_elixir; top-level keys surfaced as paragraphs
(TOML deferred — indexed as raw text with warning)
json.ex Jason; top-level keys surfaced as paragraphs
csv.ex NimbleCSV with compile-time comma + tab parsers; row-per-row
html.ex Floki; strips scripts + styles; h1–h6 hierarchy
code.ex 30+ language extensions; def/function/class heuristic
pdf.ex shell to pdftotext; pages via form-feed split; graceful
office.ex native :zip + OOXML; .docx .pptx .xlsx; no shell tools
image.ex shell to tesseract OCR; image kept as Asset; graceful
audio.ex whisper.cpp local HTTP multipart; :timestamp segments; graceful
video.ex ffmpeg → frame (Image) + audio (Audio); graceful
Graceful-degradation invariant: missing shell tools (pdftotext, tesseract,
whisper, ffmpeg) NEVER produce errors. They return a well-formed ParsedDoc
with empty text, warnings explaining the missing dependency, and the
binary preserved as an Asset so Phase 5's multi-modal embedder can still
reach it.
Surfaces:
mix optimal.parse <path> — run Stage 2 + print summary
mix optimal.parse <path> --format raw — full struct dump
mix optimal.parse <path> --format text — extracted text only
Tests (34 new, 775 total):
test/pipeline/parser/dispatcher_test.exs — dispatch + registry
test/pipeline/parser/markdown_test.exs — 5 happy paths
test/pipeline/parser/text_test.exs — paragraphs + empty input
test/pipeline/parser/yaml_test.exs — happy + malformed + TOML
test/pipeline/parser/json_test.exs — happy + malformed + non-object
test/pipeline/parser/csv_test.exs — CSV + TSV
test/pipeline/parser/html_test.exs — strip scripts/styles + h1–h6
test/pipeline/parser/code_test.exs — Elixir + inline language hint
test/pipeline/parser/office_test.exs — docx + pptx + xlsx (on-the-fly ZIP fixtures)
test/pipeline/parser/binary_formats_test.exs — PDF/image/audio/video graceful path
Infra fix: `mix test` alias wipes /tmp/optimal_engine_test_0.db before each
run so schema drift from an older migration set can't leak across runs.
Eliminates the intermittent Role.grant failures seen in Phase 1.
Deps (new): {:nimble_csv, "~> 1.2"}, {:floki, "~> 0.36"}
Build verification:
mix compile — generated (RocksDB warnings pre-existing)
mix format — clean
mix test — 741/741 passing (29 excluded, all :rocksdb-tagged)
mix optimal.parse README.md --format summary — prints structural summary
Acceptance for Phase 2 per PLAN.md met:
✓ every format listed has a backend module
✓ text-based formats fully tested with fixtures
✓ binary formats have graceful-degradation tests
✓ mix optimal.parse returns %ParsedDoc{} with text + structure + modality
Stage 3 of the ingestion pipeline per docs/architecture/ARCHITECTURE.md.
Takes a %ParsedDoc{} from Stage 2 and produces a %ChunkTree{} with chunks
at every scale, parent_id-linked, respecting parser-reported boundaries.
Scales (exhaustive, fixed):
:document — exactly 1 per signal; covers full text; root of the tree
:section — from parser :heading / :page / :slide / :section elements
(document IS the section when none reported)
:paragraph — from parser :paragraph elements; falls back to blank-line
split when absent
:chunk — sliding window within each paragraph; never spans paragraph
boundaries; window_bytes + overlap_bytes configurable
ID convention (deterministic + idempotent):
"{signal_id}:{prefix}-{index}" where prefix ∈ {doc, sec, par, chk}
Re-decomposing the same doc produces identical ids (same hash → same ids).
New modules:
lib/optimal_engine/pipeline/decomposer/chunk.ex
%Chunk{id, tenant_id, signal_id, parent_id, scale, offset_bytes,
length_bytes, text, modality, asset_ref, classification_level}
+ Chunk.build_id/3, Chunk.new/1
lib/optimal_engine/pipeline/decomposer/chunk_tree.ex
%ChunkTree{root_chunk_id, chunks}
+ at_scale/2, children_of/2, parent_of/2, counts/1, reassemble/1
lib/optimal_engine/pipeline/decomposer.ex
decompose/2 — pure ParsedDoc → ChunkTree
decompose_and_store/2 — atomic bulk insert to chunks table
+ window tunables (window_bytes: 2048, overlap_bytes: 256)
Store integration:
OptimalEngine.Store.insert_chunks/1 — bulk INSERT OR REPLACE in one txn
Idempotent on chunk.id (re-running decomposer produces same rows)
Surfaces:
mix optimal.decompose <path> — Parser → Decomposer → print
mix optimal.decompose <path> --format counts — per-scale totals
mix optimal.decompose <path> --window 4096 --overlap 512 — override tunables
mix optimal.decompose <path> --format raw — full ChunkTree dump
Tests (25 new, 766 total):
test/pipeline/decomposer/chunk_tree_test.exs — 5 navigation + reassemble
test/pipeline/decomposer/decomposer_test.exs — 18 acceptance:
· minimum viable (empty + simple text)
· hierarchy invariants (every chunk has valid parent path)
· paragraph scale (element vs fallback)
· chunk scale (short, long, no-cross-boundary)
· reassembly guarantee (unicode + page-structured PDF shape)
· determinism (repeated decompose → identical ids)
· tenant_id + modality propagation
test/pipeline/decomposer/store_integration_test.exs — 2 round-trip:
· decompose_and_store persists every chunk
· re-inserting is idempotent
Acceptance for Phase 3 per PLAN.md met:
✓ 5-page structured doc → 1 document + N sections + M paragraphs + K chunks
with correct parent_id links
✓ root chunk text byte-identical to source (reassembly guarantee)
✓ window + overlap tunable; chunk never spans a paragraph
✓ store.insert_chunks persists the tree atomically; idempotent replay
Build:
mix compile — clean (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 766/766 passing (29 excluded :rocksdb)
Organizational-topology layer per Roberto's Canopy/OptimalOS framing. A
tenant's workspace is the first-class model of the company it represents:
nodes (organizational units), people (principal.kind=:user), agents
(principal.kind=:agent), skills (capabilities), memberships.
Before: `contexts.node` was a freeform string. Routing rules pointed at
strings. There was no registry of "what nodes exist here, who owns them,
what they're about."
After: every node is a row with kind + style + status + parent-child
tree + metadata + time-bounded memberships. Skills and principal_skills
are first-class, shared uniformly between humans and agents.
Schema (migrations 017–022 on top of Phase 1's 001–016):
017 nodes (tenant-scoped, kind: unit|team|project|entity|domain|person,
style: internal|external|mixed, status, parent_id tree)
018 node_members (principal↔node, membership: owner|internal|external|observer,
time-bounded with started_at/ended_at)
019 skills (tenant-scoped capability registry, kind: technical|
communication|strategic|domain|tool)
020 principal_skills (many-to-many, level: novice|intermediate|expert|lead)
021 tenant-first indexes on all four new tables
022 backfill: every distinct contexts.node value becomes a nodes row
so existing routing/retrieval paths keep working
Modules (lib/optimal_engine/workspace/):
node.ex CRUD + children/1 + ancestors/1 (root→self order)
node_member.ex add/remove/members_of/nodes_of with membership filter
skill.ex upsert/list/get; kind + slug-based ID
principal_skill.ex grant/revoke + skills_of/principals_with_skill
+ min_level filter; returns humans + agents uniformly
workspace.ex top-level facade (defdelegate everything)
Surfaces:
mix optimal.workspace [--tenant X] [--nodes-only|--skills-only]
Tests (20 new, 786 total):
test/workspace/node_test.exs — 6 tests (upsert validation,
idempotence, children, ancestors,
list filters)
test/workspace/node_member_test.exs — 5 tests (membership kinds,
filter, nodes_of, remove closes,
idempotent add)
test/workspace/skill_test.exs — 8 tests (skill upsert,
principal_skill round-trip,
min_level filter, mixed
humans+agents return, re-grant,
revoke)
Build:
mix compile — clean (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 786/786 passing (29 excluded :rocksdb)
Docs:
docs/architecture/WORKSPACE.md — full model spec
PLAN.md — Phase 3.5 slotted into the sequence
README.md — status table + test-count bump (707 → 786)
This layer unblocks:
Phase 4 (Classifier) — key off who/what node owns a signal
Phase 7 (Wiki Curator) — audience-aware curation via node+skill resolution
Phase 8 (Deliver) — membership-scoped retrieval boost
Phase 9 (Connectors) — integration_grants keyed on principals + groups
Stage 4 of the ingestion pipeline. Every chunk from Phase 3's %ChunkTree{}
now gets its own Signal Theory classification + intent row, persisted to
the classifications and intents tables Phase 1 created.
Two substages with one acceptance contract each:
4a Classifier.classify_chunk/2 → %Classification{} (per chunk)
Classifier.classify_tree/2 → [%Classification{}] (whole tree)
4b IntentExtractor.extract/2 → %Intent{} (per chunk)
IntentExtractor.extract_tree/2 → [%Intent{}] (whole tree)
New structs:
OptimalEngine.Pipeline.Classifier.Classification
{chunk_id, tenant_id, mode, genre, signal_type, format, structure,
sn_ratio, confidence}
OptimalEngine.Pipeline.IntentExtractor.Intent
{chunk_id, tenant_id, intent, confidence, evidence}
10-value enum: request_info propose_decision record_fact
express_concern commit_action reference narrate
reflect specify measure
IntentExtractor design:
· heuristics-first (regex rule table, 9 explicit rules + :record_fact default)
· Ollama-augmented when heuristic confidence < 0.7 AND Ollama reachable
· gracefully falls back to heuristic when Ollama unreachable / returns garbage
· evidence window (~80-char slice) attached for auditability
Classifier extensions:
· inherits mode + format from chunk.modality (parser already knows)
· sn_ratio via unique-token-density heuristic (0.4..0.9 band)
· confidence scales with number of resolved S=(M,G,T,F,W) dimensions
· reuses existing extract_genre/extract_type helpers (backward compatible)
Store persistence:
OptimalEngine.Store.insert_classifications/1 (bulk, atomic, idempotent on chunk_id)
OptimalEngine.Store.insert_intents/1 (bulk, atomic, idempotent on chunk_id)
Surface:
mix optimal.classify <path> Parser → Decomposer → Classifier → IntentExtractor
mix optimal.classify <path> --format json
Tests (24 new, 810 total):
test/pipeline/classifier/classification_test.exs 6 tests
· Classification struct keyed to chunk_id
· mode inherits from chunk.modality (:code, :data, …)
· confidence scales with resolved-dimension count
· sn_ratio > 0 for non-empty text, = 0 for empty
· classify_tree returns one row per chunk
test/pipeline/intent_extractor/intent_extractor_test.exs 16 tests
· 10 heuristic-per-intent tests (one per canonical value)
· confidence always in 0.0..1.0
· confidence rises with match density
· evidence window is a substring of the source text
· gold-set accuracy test (10 hand-labeled samples) ≥ 80%
test/pipeline/classify_store_test.exs 2 tests
· classifications round-trip keyed by chunk_id, idempotent replay
· intents round-trip with valid enum values only
Regex discipline learned the hard way:
`\b` requires word↔non-word transition. `todo:\b` doesn't match
"todo: " because `:`→space is non-word→non-word. Punctuation-ending
alternatives now use `(?:^|\W)` prefix instead of `\b` suffix. Three
failing rules (:commit_action, :specify, :narrate) fixed this way.
Flaky test fix:
test/retrieval/principal_filter_test.exs — empty-string query to
SearchEngine.search was triggering an Ollama path that timed out when
Ollama responded with empty embeddings. Switched to "test" query.
Build:
mix compile — clean (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 810/810 passing (29 excluded :rocksdb)
Stage 5 of the ingestion pipeline. Every chunk from Phase 3's ChunkTree
now produces a 768-dim aligned vector, written to the new chunk_embeddings
table keyed on chunk_id. Text, image, and audio all land in the SAME
coordinate system — the architectural win that makes cross-modal
retrieval possible (text query → image / audio result).
Provider stack (all local, OSS-first):
OptimalEngine.Embed.Ollama text → nomic-embed-text (existing, extended)
image → nomic-embed-vision (NEW, base64 via /api/embed)
OptimalEngine.Embed.Whisper audio → whisper.cpp local HTTP, multipart upload
→ transcript → text embedder (aligned)
OptimalEngine.Embed.Provider behaviour for future adapters
Pipeline stage:
OptimalEngine.Pipeline.Embedder
embed/2 → ok | error with reason
embed_tree/2 → embeddings plus errors keyed by chunk_id
graceful when providers unreachable
OptimalEngine.Pipeline.Embedder.Embedding struct
{chunk_id, tenant_id, model, modality, dim, vector}
Dispatch matrix:
:text / :code / :data / :mixed → Ollama.embed_text
:image with asset → Ollama.embed_image
:image with OCR text only → Ollama.embed_text (fallback)
:audio with transcript → Ollama.embed_text
:audio with asset, no transcript → Whisper.transcribe → Ollama.embed_text
:video → text fallback (frame-sequence deferred)
Schema:
Migration 023 — chunk_embeddings table
chunk_id TEXT PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE
tenant_id, model, modality, dim (default 768), vector BLOB, created_at
plus tenant-first indexes on modality and model
Store:
Store.insert_embeddings/1 — bulk upsert, idempotent on chunk_id
vectors stored as little-endian float32 BLOBs (768 * 4 = 3072 bytes)
Surface:
mix optimal.embed <path> Parser → Decomposer → Embedder → summary
mix optimal.embed <path> --format json
Tests (15 new, 825 total):
test/pipeline/embedder/embedding_test.exs
test/pipeline/embedder/embedder_test.exs dispatch + graceful degradation
test/pipeline/embedder/store_integration_test.exs round-trip + 3072-byte assertion
test/embed/whisper_test.exs reachability probe + error paths
Critical bug fixed (pre-existing, silent):
Exqlite binds raw binaries as TEXT by default. TEXT ends at the first
null byte. Float32 data contains null bytes constantly. Result: the
ORIGINAL `vectors` table in OptimalEngine.Store.Vectors.store/2 was
saving 0-byte embeddings since Phase 0 — silently, with no test
catching it.
Fix: wrap BLOB params as {:blob, binary} tuple, forcing the correct
type binding. Applied both to the new insert_embeddings path and
retrofitted into Store.Vectors.store. Every embedding now persists
the intended 3072 bytes.
The new Phase 5 store_integration_test.exs asserts
LENGTH(vector) == 3072 — that is what caught it.
Build:
mix compile — clean (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 825/825 passing (29 excluded :rocksdb)
Stage 6 (numbered Stage 8 in ARCHITECTURE.md) — the wide pass across
chunks that groups them into theme-clusters. Completes the ingestion
pipeline's read side: every ingested chunk now has embedding + intent
+ classification + cluster membership, all persisted.
Algorithm: incremental greedy with weighted-similarity threshold.
HDBSCAN is the architecturally ideal choice but has no pure-Elixir hex
library; the greedy threshold approach is deterministic, incremental
(no full rebuild required to absorb new chunks), and produces stable
assignments. HDBSCAN is documented as a Phase 10+ enhancement.
New modules:
OptimalEngine.Pipeline.Clusterer
assign/2 incremental batch assignment; returns clusters + memberships
rebuild/1 full rebuild for a tenant (clears + re-assigns)
OptimalEngine.Pipeline.Clusterer.Cluster
{id, tenant_id, theme, intent_dominant, member_count, centroid, updated_at}
mirrors the clusters table (Phase 1 migration 007)
OptimalEngine.Pipeline.Clusterer.Similarity
sim/2 weighted composite in [0, 1]
cosine/2 with safe 0.0 fallbacks for empty / mismatched / zero vectors
entity_overlap/2 Jaccard over entity name sets
intent_match/2 1.0 for same intent, 0.0 otherwise
node_affinity/2 1.0 same node, 0.5 shared ancestor, 0.0 otherwise
mean_vector/1 element-wise mean for centroid updates
Weighted composite:
sim(a, b) = 0.6 × cosine + 0.2 × entity_overlap + 0.15 × intent_match + 0.05 × node_affinity
Normalized by available-weight: pure-embedding matches still land at
1.0 when cosine = 1.0 (instead of being capped at 0.6 by missing
components). Renormalization only kicks in when a feature is ABSENT,
not when it is present-but-zero.
Store helpers (bulk, atomic, idempotent):
Store.insert_clusters/1 upserts on cluster.id (BLOB centroid via {:blob, _})
Store.insert_cluster_members/1 upserts on (cluster_id, chunk_id)
Surface:
mix optimal.cluster list clusters for the default tenant
mix optimal.cluster --tenant acme list for another tenant
mix optimal.cluster rebuild clear + reassign from chunk_embeddings
mix optimal.cluster rebuild --threshold 0.7
Tests (34 new, 859 total):
test/pipeline/clusterer/similarity_test.exs
19 tests covering cosine edge cases, entity Jaccard, intent match,
node affinity tiers, composite, mean_vector
test/pipeline/clusterer/clusterer_test.exs
12 tests: single-feature seed, similar-joins-together,
dissimilar-seeds-new, threshold override, centroid running-mean,
intent_dominant, pre-existing cluster join, determinism
test/pipeline/clusterer/store_integration_test.exs
3 tests: Store round-trip (theme / intent_dominant / member_count),
rebuild/2 against a real tenant produces the expected 2-cluster
split from 4 synthetic chunks
Design decisions captured in moduledocs:
- Why not HDBSCAN (no hex lib; v1 uses greedy threshold)
- Renormalization of missing features vs present-but-zero
- Incremental vs rebuild cost profile
- Feature construction pulls from 4 Phase 1-5 tables (embeddings,
entities, intents, chunks→contexts→Workspace nodes)
Build:
mix compile — clean (RocksDB NIF warnings pre-existing)
mix format — clean
mix test — 859/859 passing (29 excluded :rocksdb)
Introduces the top tier of the three-tier knowledge architecture:
LLM-curated wiki pages with hot citations, executable directives,
audience-aware versioning, and schema-enforced governance.
Components:
* Wiki.Page — struct + YAML-frontmatter serialization
* Wiki.Directives — whitelisted verb lexer/renderer
(cite / include / expand / search / table /
trace / recent / wikilink)
Output formats: plain, markdown, claude, openai
* Wiki.Integrity — verb whitelist + citation resolution +
claim-density + schema rule enforcement
* Wiki.Curator — deterministic + Ollama-backed curation
(appends `## New signals` with cite directives,
bumps version, records curated_by/last_curated)
* Wiki.Store — wiki_pages + citations tables, version bumps,
latest-by-audience lookups
* .wiki/SCHEMA.md — governance contract
(required_sections, required_frontmatter,
max_bytes)
* mix optimal.wiki — list / view / verify / verify-all CLI
Tests: 32 new wiki tests, full suite at 891.
End-to-end retrieval flow: query + receiver → wiki-first lookup →
hybrid fallback → bandwidth-aware packing → receiver-shaped envelope.
query + receiver
│
▼
IntentAnalyzer — expand query, entities
│
▼
WikiFirst — Tier-3 curated lookup (audience-aware)
│
├──hit──▶ Deliver.render_wiki ──▶ envelope
│
└─miss─▶ Search (hybrid, principal-filtered)
│
▼
BandwidthPlanner.plan (token budget)
│
▼
Deliver.render_chunks ──▶ envelope
New modules:
* Retrieval.Receiver — audience + bandwidth + format profile
* Retrieval.BandwidthPlanner — greedy token-budget packing
* Retrieval.WikiFirst — slug/title matcher with audience fallback
* Retrieval.Deliver — plain/markdown/claude/openai envelopes
(reuses Wiki.Directives for citation
consistency between wiki and chunk paths)
* Retrieval.RAG — ask/2 orchestrator + graceful search
timeout degradation
* Retrieval — top-level facade
Upgrades:
* mix optimal.rag — rewritten around wiki-first ask/2,
adds --audience, --bandwidth, --principal,
--skip-wiki, --trace; `text` stays as
alias for `plain`
Tests: 42 new retrieval tests, full suite at 933.
Contract-driven ingestion layer. Every external system becomes a
one-way pipe whose output is a stream of Signals + an opaque cursor.
Infrastructure:
* Connectors.Behaviour — 7-callback contract every adapter
implements (kind/display_name/
auth_scheme/required_config_keys/
init/sync/transform)
* Connectors.Registry — compile-time adapter roster + lookup
* Connectors.Runner — execute sync cycle with cursor
advancement, rate-limit retry,
exponential backoff, connector_runs
audit row. Fatal errors flip
`enabled = 0`.
* Connectors.Credential — AES-256-GCM envelope encryption at
rest, versioned envelope format,
master key from CONNECTOR_KEY (Phase
11 KMS hook)
* Connectors.RateLimit — ETS token-bucket shared across
adapters + tenants
* Connectors.HTTP — :httpc wrapper with JSON + Retry-After
+ rate-limit integration
* Connectors.Transform — shared signal_id/source_uri/strip_html
helpers for adapter transform/1
* Connectors.Adapters.Base — `use` macro factoring out config +
credential validation boilerplate
* Connectors — top-level facade
14 adapter modules (stubs for Phase 10 API wiring; transform/1
fully wired for realistic payload shapes):
Slack, Gmail, Google Drive, Notion, Jira, Linear, GitHub, Zoom,
Confluence, Microsoft Teams, Dropbox, OneDrive, Salesforce, HubSpot.
CLI: `mix optimal.connector {list|register|run}`
Refinement: Retrieval.WikiFirst audience fallback now triggers on
"no scoring hit in requested audience" rather than "no pages at all
in requested audience" — matches curated-page intent.
Tests: 93 new connector tests (adapter contract × 14, registry,
runner, credential, ratelimit), full suite at 1,026.
Observability + runtime integrity + release-time entry points.
New modules:
* Telemetry — in-memory counter/histogram aggregator. Folds
:telemetry.execute events emitted by intake /
search / retrieval / connector into a snapshot
map (p50/p95/p99 over a 1k-sample window).
Zero external dependencies; Prometheus bridge
lives in Phase 11.
* Health — liveness + readiness (store, migrations,
credential_key, embedder). Returns a structured
report; status/0 collapses to :up | :degraded |
:down for CI / alert routing.
* Backup — online SQLite backup via VACUUM INTO. Safe to
run against a live engine. Refuses to overwrite,
auto-archives pre-restore state, verify/1 runs
PRAGMA integrity_check.
* Release — entry points for `bin/optimal eval …`:
migrate/0, preflight/0, backup/1, healthcheck/0.
Exit codes designed for deployment automation.
Supervision tree: Telemetry now starts first so every subsequent
child's events land in the aggregator.
CLI:
* mix optimal.status — supervisor / store / migrations /
credential / embedder readiness;
--json / --quick / --metrics
* mix optimal.backup — live backup, optional --verify
(runs PRAGMA integrity_check after)
Tests: 14 new ops tests (telemetry, health, backup), full suite at
1,040.
Regulatory controls wired into the engine: right-to-know (DSAR),
right-to-delete (Erasure), legal preservation (LegalHold), retention
enforcement, PII detection + redaction.
New modules:
* Compliance.PII — regex-based detector for email / phone /
SSN / credit-card (Luhn-verified) / IPv4
/ URL. Returns match + offset + kind.
* Compliance.Redact — 4 strategies over PII matches:
placeholder / mask / hash / remove.
Right-to-left splice preserves offsets.
* Compliance.DSAR — GDPR Art. 15: enumerate principal
footprint across principals, contexts,
node_members, principal_skills,
role_grants, principal_groups, events,
plus FTS-style mentions.
* Compliance.Erasure — GDPR Art. 17: cascade delete with
preview/2, audit-event pseudonymization
(stable phash-based "erased:<id>" token
so the trail survives but anonymized),
legal-hold gate unless :force.
* Compliance.LegalHold — place/release/active/held?/count_holds.
`legal_holds` rows persist a freeze that
retention sweeps + erasure must honor.
* Compliance.Retention — TTL-based sweeps driven by
retention_policies. Scopes: node | genre
| tenant | global. Actions: archive |
delete | redact. Every action writes a
retention_action event for SOC 2 audit.
* Compliance — top-level facade
Schema: migration 024 adds `contexts.created_by` + `contexts.archived_at`
with index on (tenant_id, created_by) so Erasure + Retention have a
real link to walk.
CLI: mix optimal.compliance {dsar|erase|hold|retention} with --force,
--preview, --dry-run, --by, --reason.
Tests: 28 new compliance tests (PII, Redact, DSAR, Erasure w/ legal
hold gate + force, LegalHold lifecycle, Retention sweep + dry-run).
Full suite at 1,068.
Wires the engine for external consumption and ships a minimal
desktop client that exercises the new endpoints end-to-end.
HTTP API (Plug + Cowboy, opt-in):
Config:
config :optimal_engine, :api, enabled: true, port: 4200
New endpoints:
GET /api/status — Phase 10 Health.ready report
GET /api/metrics — Phase 10 Telemetry snapshot
POST /api/rag — Phase 8 Retrieval.ask/2 over HTTP
GET /api/wiki — Phase 7 Wiki.list/1 page catalog
GET /api/wiki/:slug — render page in the receiver's format
Hook-up:
* API.Endpoint returns a child-spec list (empty when disabled)
so the supervisor never sees :ignore. Application now composes
base_children() ++ API.Endpoint.children().
* Plug.Parsers added to the router for JSON POST bodies.
* CORS widened to allow POST + content-type.
Desktop shell (desktop/):
* SvelteKit 2 + Svelte 5 + Tauri 2 scaffold
* src/lib/api.ts — typed client over the engine's HTTP API
* routes/: +layout (nav + live status badge), / (Ask), /wiki
(list + render), /status (readiness + degraded).
* tauri.conf.json wires npm run dev behind a native window;
vite.config.ts proxies /api → 127.0.0.1:4200 so browser-only
dev works too.
Tests: 7 new API router tests (status / metrics / RAG / wiki),
full suite at 1,075.
1. connectors/runner.ex — `String.to_existing_atom/1` on DB-sourced
`kind_str`. Previously `to_atom/1` would create a fresh atom for
every junk value in the connectors.kind column and eventually
exhaust the atom table. Unknown kinds now return
`{:error, :unknown_kind}`.
2. connectors/runner.ex — cursor advance + audit completion wrapped
in a SQLite BEGIN IMMEDIATE / COMMIT. Previously a failure in
complete_run_row after a successful advance_cursor would leave the
audit row in 'running' forever and the run signal count silently
missing from the trail.
3. api/router.ex — `/api/metrics` wrapped its `Telemetry.snapshot/0`
call in `try/rescue`, but GenServer `:exit` isn't an exception.
Added `catch :exit, _` so a downed Telemetry GenServer returns the
fallback map instead of 500ing the handler. Same fix in
mix/tasks/optimal.status.ex.
4. compliance/legal_hold.ex — `held?/2` and
`count_holds_for_principal/2` swallowed DB errors as "no hold",
letting retention and erasure proceed as if nothing was held.
Both now return `{:error, :hold_check_failed}` on query error.
Retention treats the error as a skipped row + errors+1; erasure
propagates the error out of `erase/2`. `force: true` bypasses the
check entirely.
5. compliance/retention.ex — `apply_action/4` rescued the whole
block, so a `trace/3` failure flipped a successful delete into
`:error`. Now each action returns `:ok` only on the DB write
result; trace failures log warnings but do not change the return.
6. retrieval/wiki_first.ex — substring scoring let a 2-char query
like `ai` match any slug containing that substring and
short-circuit hybrid retrieval. Added a 4-char minimum for the
contains tier and dropped the weight to 10 so a prefix match on a
different slug always wins.
7. compliance/erasure.ex — pseudonym switched from phash2 (27-bit
int with birthday collisions at scale) to SHA-256 shortened to
16 hex chars.
8. telemetry.ex — handler used `String.to_atom/1` on the event path;
switched to `String.to_existing_atom/1` with ArgumentError rescue.
attach_many rescue narrowed to handle :already_exists.
Test update: legal_hold_test.exs adapts to the new held?/2 return
shape.
Suite: 1074 of 1075 passing; the one failure is a pre-existing
flaky GenServer.stop race in knowledge/dictionary_test.exs that
passes in isolation.
Pulled three Svelte components forward from the sibling businessos-5
repo into desktop/src/lib/components/knowledge/:
* OptimalGraphView.svelte — Three.js force-directed 3D graph
* NodeDrillDown.svelte — breadcrumb file explorer per node
* GraphControls.svelte — zoom / layout overlay
Matching infrastructure:
* desktop/src/lib/api/base.ts — shim exporting getApiBaseUrl /
getCSRFToken so the ported components' `$lib/api/base` imports
resolve against the engine's Vite proxy in dev and
VITE_ENGINE_URL in Tauri builds. No engine-side CSRF.
* desktop/src/routes/graph/+page.svelte — new route that mounts
OptimalGraphView with NodeDrillDown as a side panel. Fetches
/api/optimal/nodes on mount so the drill-down has its level-0
card grid ready.
* desktop/package.json — three + d3-force added; @types/three dev.
* Layout nav gains a Graph link.
Engine endpoints (api/router.ex), shaped to exactly what the ported
components expect:
GET /api/optimal/graph
→ {entities: [{name, type, connections}],
edges: [{source, target, relation, weight}],
stats: {entity_count, edge_count, edge_types}}
Entities come from DISTINCT context_id per (name, type) in the
entities table. Edges are co-occurrence (two entities sharing a
context), relation = related_to, weight capped at 5.0.
GET /api/optimal/nodes
→ {nodes: [{slug, name, type, signal_count}]}
Pulls from the workspace `nodes` table joined with a context
count. Falls back to DISTINCT node from contexts when the
nodes table is empty.
GET /api/optimal/nodes/:slug/files
→ {files: [{name, path, is_dir, size, genre, modified_at}]}
Demo seeder (mix optimal.seed):
Idempotent task that drops 6 workspace nodes, 10 realistic signals
(transcripts / decisions / plans / notes spanning Roberto, MIOSA,
Lunivate, AI Masters, Agency Accelerants, ContentOS) with 41
extracted entities spanning person / org / product / concept /
operation types, plus one curated wiki page. Rows are tagged
"demo-seed" in metadata so `--reset` can surgically remove them
without disturbing real content.
End-to-end verified: seeder populates, the three endpoints return
the right shapes (27 entities, 66 edges, 6 nodes, 2 files in
04-ai-masters). API test suite 7/7 green.
This is the model-agnostic data-point substrate. The engine stops being "a text-chunk store" and becomes a universal storage layer where any field (text / image / audio / video / time-series / tabular / structured / code / tensor / graph / geo / binary) binds to any processor (LLM / vision model / classical ML / rule engine / agent) by atom. Architecture layer (lib/optimal_engine/architecture/): * Field — typed slot with modality + shape + processor hint * Architecture — composition of fields + granularity + retention * Processor — @behaviour every processor implements * Registry — built-in + tenant-defined architecture catalog * ProcessorRegistry — processor lookup by atom id * Apply — validate + dispatch (pure; no model imports) * 7 built-in architectures covering the dimension space: text_signal, image_asset, audio_transcript, structured_record, time_series_window, code_commit, multimodal_media * 6 built-in processor stubs: text_embedder, image_embedder, audio_embedder, video_embedder, code_embedder, ts_feature_extractor (classical, no-ML) * Facade module OptimalEngine.Architecture Schema (migration 025): * architectures table (spec as JSON — schema-on-read) * contexts.architecture_id column + index * processor_runs table (per-field × per-processor audit row) CLI: * mix optimal.architectures list catalog * mix optimal.architectures show <name> field-level detail * mix optimal.architectures processors registered processors HTTP API (api/router.ex): * GET /api/workspace full node forest (tree shape) * GET /api/signals/:id every granularity for one signal: dimensions + chunks (4 scales) + entities + classification + intent + clusters + citations + architecture binding * GET /api/activity events log, filterable by kind * GET /api/architectures catalog + processors * GET /api/architectures/:id field-level detail Desktop (SvelteKit): * /workspace 3-pane explorer: node tree → signal list → full-granularity detail. Every layer the engine tracks is selectable. * /activity events feed, 5s auto-refresh, filter by kind * /architectures catalog + processors side panel, field table per architecture, modality-colored pills * SignalBadge + SignalDetailPanel ported from businessos-5 for reuse inside workspace panels * $lib/api/workspace.ts typed client + toTree() helper * $lib/stores/osa.ts shim type Seeder expansion (optimal.seed): * 12 workspace nodes with parent/child hierarchy (MIOSA ⇾ platform / agency / pe-deck; AI Masters ⇾ beginner / advanced; Agency Accelerants ⇾ ClinicIQ delivery) * 10 signals decomposed into 68 chunks across 4 scales * 41 entities spanning person/org/product/concept/operation * 10 classifications + 10 intents * 1 ClinicIQ theme cluster with membership * 10 ingest audit events * --reset now cascades: wipes chunks, classifications, intents, cluster_members, clusters, processor_runs, entities, events Docs: docs/architecture/DATA_ARCHITECTURE.md captures the SOTA inspirations (Iceberg schema-on-read, CLIP/ImageBind alignment, hybrid retrieval, memory consolidation, neuro-symbolic, data lineage, versioned data) and how each one maps to engine modules. Suite: 1075 / 1075 green.
…routes
Row for Phase 14 added to the status board. Test count corrected to
1,075 across Quick start, Development, and the seeder command.
New sections:
* HTTP API — full endpoint table including Phase 12 core endpoints
(status / metrics / rag / wiki / graph) and Phase 14 additions
(workspace / signals / activity / architectures).
* Desktop UI — route-by-route description matching the shipped
SvelteKit app (ask / workspace / graph / wiki / architectures /
activity / status).
Mix-task reference restructured into four groups (core ingest,
graph + insight, wiki + architectures, ops + compliance) and now
includes: optimal.seed, optimal.architectures, optimal.status,
optimal.backup, optimal.connector, optimal.compliance, optimal.wiki.
Architecture tree updated: architecture/, compliance/, telemetry.ex,
health.ex, backup.ex, release.ex all listed. Root primitives list
adds architecture.ex + compliance.ex.
Doc map gains DATA_ARCHITECTURE.md.
Public-repo hygiene pass: * Personal names replaced everywhere in lib/, test/, docs/, README, PLAN with a consistent Acme-Corp placeholder set (Alice, Bob, Carol, Dan, Erin, Frank, Grace, Heidi, Ivan, Judy, Karl, Mallory, Nina, Oscar, Peggy, Quinn, Ruth, Sam, Tina, Uma, Victor). Scrub used `[[:<:]] [[:>:]]` word boundaries so substrings were preserved. * Seeder fixtures rewritten as a generic "Acme" demo company — 12 nodes (Founder's Desk / Platform Division / Services Division / Customer Academy / Partner Network / Media & Content plus six project sub-nodes), 10 signals spanning pricing transcripts / partner-close notes / microVM specs / sales-enablement plans / academy course structure / invoices / advisor calls. * The seeded wiki page is now "healthtech-pricing-decision"; the seeded theme cluster is "Healthtech delivery + pricing". * Intake tests updated to match the new fixture strings. Competitor comparisons kept (positioning intact) and expanded to cover a broader slice of the space: vector DBs (Pinecone / Weaviate / LlamaIndex), memory platforms (Mem0 / Letta / Zep), and doc tools (Notion AI) alongside the application-layer rivals already listed. Framing stays factual — no "inspired by" / "same paradigm as" language. Suite: 1075 / 1075 passing.
Reality-check task exposed retrieval was 8 s per query even with
Ollama healthy. Root cause: IntentAnalyzer called Ollama.generate
(LLM completion) on every hybrid search. Fixed:
* Search.do_hybrid_search gates LLM intent expansion behind
`expand_intent: true`; default uses the regex fallback.
* Ollama.embed_healthy?/0 replaces available?/0 on the hot path —
actually probes the embed endpoint via a 1 s Task.async with
brutal_kill on timeout. Cached 60 s.
* Ollama default HTTP timeout dropped 30 s → 5 s.
* Guard: skip vector search when Ollama returns an empty embedding
(happens for degenerate queries).
Reality-check task (`mix optimal.reality_check [--hard] [--ingest N]`)
runs 50+ probes touching every storage table, every pipeline stage,
every retrieval path, every compliance workflow, with OK/WARN/FAIL
+ elapsed ms. 500-row ingest completes in 91 ms. End-to-end wall
clock: 394 ms / 51 probes / 0 failures.
Desktop:
* /graph switched to three.js + d3-force-3d with node drag,
animated pulse particles, OrbitControls (left: rotate, right:
pan, wheel: zoom), theme-reactive edges + pulses.
* app.css with full light + dark palette as CSS variables; layout
gains a header theme toggle with OS-preference fallback.
* full-bleed layout on /graph, /workspace, /activity.
* Ask page migrated to Svelte 5 runes (was breaking on `$:`);
layout migrated to `{@render children()}`.
* Graph dataset expanded — 23 signals, 97 entities, 194 chunks,
23 classifications + intents, cluster membership, denser edge
graph for visualization.
Health check treats "no connectors + no credential key" as :ok
rather than :warn — that's the default dev state, not degraded.
Fleshed-out example of how an organization lays out its knowledge
for the engine to ingest:
sample-workspace/
├── nodes/
│ ├── 01-founder/ context + signal + 2 signals
│ ├── 02-platform/ context + signal + 3 signals (spec, audit, review)
│ ├── 03-services/ context + invoice signal
│ ├── 04-academy/ context + signal + 3 signals (pricing call, curriculum, office-hours)
│ ├── 06-partners/ context + 2 signals (close, renewal pipeline)
│ └── 08-media/ context + 2 signals (handoff, content calendar)
├── .wiki/ SCHEMA.md + 2 curated pages with live {{cite:}} back to Tier 1
├── architectures/ clinical_visit.yaml — user-defined data-point schema example
└── assets/ binary-attachment convention
Every signal file carries YAML frontmatter + body — the documented
convention is in sample-workspace/README.md with field table + usage
examples. Stories interlock (the pricing call feeds the wiki decision,
which references the microVM spec, which references the security
audit, etc.) so the graph / retrieval / wiki-first flow demonstrates
real cross-references rather than disconnected fixtures.
Two new mix tasks:
* `mix optimal.init <target>` — scaffolds a new workspace from
sample-workspace/; refuses to overwrite existing files without
--force.
* `mix optimal.ingest_workspace <path>` — walks a workspace
directory, parses YAML + body for every signal, creates contexts +
entities + audit events, and puts wiki pages into Tier 3.
End-to-end verified: fresh ingest yields 6 nodes / 13 signals / 56
entities / 2 wiki pages; `mix optimal.rag "healthtech pricing
decision" --trace` hits the curated page with 4 citations resolving
to real files on disk.
README quick-start updated to drive off the sample workspace.
New contributors now have a one-command path from clone to working
engine:
* GETTING_STARTED.md — prerequisites table with platform-specific
install commands, troubleshooting section, next-steps.
* mix optimal.bootstrap — single task that verifies the supervision
tree, confirms migrations applied, ingests sample-workspace/,
and prints row counts + next-step hints. Idempotent; safe to
rerun after git pull.
* Makefile — `make install` / `make bootstrap` / `make dev` / `make ui`
/ `make test` / `make reality` / `make clean` for folks who live
in make.
Bootstrap produces a working state in about 20 seconds: 12 nodes, 13
signals, 56 entities, 2 wiki pages ingested from disk, rag queries
answerable, wiki citations resolving back to real files.
The second brain of a company — an Elixir/OTP quad-store that ingests
every signal flowing through an organization, classifies intent,
embeds into a multi-modal aligned vector space, clusters wide by
theme, and delivers permission-gated, receiver-matched context to
any agent runtime.
* 9-stage signal pipeline (parse → decompose → classify → embed
→ route → store → cluster → curate)
* Three tiers: raw sources, machine-maintained derivatives,
LLM-curated wiki with hot citations + executable directives
* Hierarchical chunking at 4 scales, 10-value intent extraction,
768-d aligned text + vision embeddings
* OWL 2 RL reasoner over an SPOG quad-store
* Wiki-first retrieval with principal-aware ACLs and
bandwidth-matched delivery envelopes (plain / markdown /
claude-xml / openai-json)
* Universal data-architecture registry — any field modality,
any processor binding (LLM, vision model, classical ML,
rule engine, agent workflow)
* 14 enterprise connectors (Slack / Gmail / Drive / Notion /
Jira / Linear / GitHub / Zoom / Confluence / Teams / Dropbox /
OneDrive / Salesforce / HubSpot)
* GDPR Art. 15 DSAR + Art. 17 erasure + legal holds + retention
policies + PII detection / redaction
* Telemetry (p50/p95/p99), online backup, release entry points
* Plug + Cowboy HTTP API, Tauri + SvelteKit desktop with 3D
force-directed knowledge graph
* Sample workspace + one-command bootstrap (mix optimal.bootstrap)
This commit only adds the cross-reference doc on top of the subtree
import (a1dfd4c). Now the businessos-5 repo carries:
optimal-engine/ ← Elixir reference (just imported via git subtree
from Miosa-osa/OptimalEngine — 246 .ex files)
desktop/backend-go/internal/connectorbridges/
← BO-specific glue
desktop/backend-go/go.mod
→ require github.com/Miosa-osa/OptimalEngine-go
(the Go port lives in its own sibling repo)
So one git clone of businessos-5 brings:
- the canonical Elixir source (read-only reference)
- the production Go consumer (BusinessOS backend + frontend)
- the BO-specific glue between them
The Go engine itself stays separate (Miosa-osa/OptimalEngine-go) so it
can be released independently and consumed by other projects.
To pull upstream Elixir changes later:
git subtree pull --prefix=optimal-engine optimal-engine-elixir main --squash
Documents (and codifies) the end-to-end Pages ↔ Engine integration that
got lost on first attempt because Elixir uses different env var names
than BusinessOS.
Makefile
optimal-engine Start the Elixir engine with OPTIMAL_ENGINE_ROOT
+ OPTIMAL_ENGINE_DB pointing at the same paths
BusinessOS writes to. Auto-installs deps on
first run.
optimal-reindex Force a filesystem walk after manual MD edits.
optimal-engine-stop Kill any process on :4200.
optimal-engine/RUNNING.md
Step-by-step ops guide. Critical clarification: Elixir reads
OPTIMAL_ENGINE_ROOT / OPTIMAL_ENGINE_DB, BusinessOS reads
OPTIMAL_OS_ROOT / OPTIMAL_DB_PATH. The make target sets BOTH so
they point at the same SQLite — that's the alignment that was
missing in earlier attempts (engine was reading
/Users/rhl/code/OptimalEngine/.optimal/index.db while BusinessOS
wrote to optimal-engine/sample-workspace/.system/index.db).
Also documents:
- Verifying both ends with `curl :4200` vs `curl :8801`
- Lifecycle of a Page save (frontend → contexts table →
PagesEngineSync → SQLite → engine read → graph view)
- lsof check for "are they actually sharing the db"
Verified end-to-end on this machine before commit:
✅ make optimal-engine starts engine on :4200 reading
optimal-engine/sample-workspace/.system/index.db
✅ curl :4200/api/optimal/nodes returns 10-businessos node with
test pages I wrote (proves the businessos-pages → engine
ingest path works)
✅ optimal.Reindex from BusinessOS Go writes to the SAME db
Elixir reads from
Three fixes in the vendored Elixir engine so BusinessOS can drive it
end-to-end:
1. POST /api/reindex — running engine accepts reindex requests instead
of needing `mix optimal.index` (which collides on :4200). 200 ok or
202 already_running both treated as success by callers.
2. /api/optimal/nodes falls back to deriving distinct nodes from the
contexts table when the nodes table is empty (fresh schema, before
anything has populated `nodes` rows). Was returning [] even though
contexts were indexed.
3. Bridge.Knowledge.sync_edges_to_store passed [s,p,o] lists to
Backend.ETS.assert_many which expects {s,p,o} tuples, crashing
post-index OWL materialization with FunctionClauseError.
The desktop shell showed empty modules (nodes, graph, timeline, heatmap) for any real workspace. Two root causes, both the same class of bug: the active workspace scope was dropped before hitting the engine, so requests fell back to the empty "default" compatibility workspace. 1. workspace store: bootstrap()/refreshWorkspaces() called listWorkspaces(orgId), but that function's first arg is `tenant`, not organization. An org id is not a tenant id (every workspace shares tenant "default" but belongs to a specific organization), so selecting a real org (e.g. operatoros) sent ?tenant=operatoros and matched zero workspaces, emptying the whole UI. listWorkspaces now takes an options object and filters by ?organization. bootstrap() also now prefers a real organization over the "default" compatibility org on first load. 2. knowledge graph + node files: loadGraph() and getNodeFiles() called /api/workspace, /api/optimal/graph and /api/optimal/nodes/:slug/files with no workspace param, so they always read the empty default workspace. Node slugs are not globally unique across workspaces, so the scope is mandatory. Both now thread the active workspace through every call, and the /graph route remounts on workspace switch. Verified against a live engine: all 5 workspaces now populate nodes, graph, activity/timeline/heatmap and search. svelte-check: 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…x signal drill-down Follow-up to the workspace-scoping fix, addressing "every module is empty" for returning users. 1. Stuck on the compat "default" org. Before organizations were first-class, the app auto-selected and persisted the "default" org. The previous fix only re-selected when the saved org was *invalid*, but "default" is valid, so returning users stayed parked on the empty compatibility org and every data module (nodes, graph, timeline, heatmap, activity) rendered empty. bootstrap() now also upgrades away from "default" to a real org when one exists, so a returning user self-heals on next load. Verified live: restoring the old default/default localStorage state now recovers to OperatorOS + a populated workspace and renders the node tree. 2. Signal drill-down returned "signal not found". /api/signals/:id is keyed by the context id and is workspace-scoped, but the node-files payload dropped the id it already SELECTs (the UI fell back to parsing the filename, which never matches) and getSignal() omitted the workspace. optimal_node_files now returns `id`; getSignal(id, workspace) passes the scope; the workspace page uses the file's real id. svelte-check 0 errors; engine recompiles clean (the running release must be rebuilt for the node-files id to appear). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bennettxai
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In the desktop shell, the data modules (nodes, graph, timeline, heatmap, activity) rendered empty for a real workspace, and returning users saw every module empty. The signal drill-down also errored with "signal not found."
Root causes (all: the active scope was dropped before reaching the engine)
Org id used as tenant id.
bootstrap()/refreshWorkspaces()calledlistWorkspaces(orgId), but that arg istenant. Selecting a real org (operatoros) sent?tenant=operatoros, matched zero workspaces, and blanked the UI.Graph + node files unscoped.
loadGraph()andgetNodeFiles()hit/api/workspace,/api/optimal/graph,/api/optimal/nodes/:slug/fileswith no workspace param, always reading the emptydefaultworkspace. Node slugs are not globally unique, so scope is mandatory.Returning users stuck on the compat
defaultorg. Before organizations were first-class, the app auto-selected and persisted thedefaultorg. Fix fix(desktop): workspace scoping + returning-user org recovery + signal drill-down #1's re-selection only triggered when the saved org was invalid;defaultis valid, so returning users stayed parked on the empty compatibility org and saw every module empty. This was the reported "nothing works."Signal drill-down 404.
/api/signals/:idis keyed by the context id and workspace-scoped, but the node-files payload dropped theidit already SELECTs (the UI parsed the filename instead, which never matches) andgetSignal()omitted the workspace.Fix
listWorkspaces()takes an options object and filters by?organization; addedorganization_idto theWorkspacetype.bootstrap()prefers a real organization, and now also upgrades away from the emptydefaultorg when a real one exists, so returning users self-heal on next load.loadGraph(workspace)andgetNodeFiles(slug, workspace)thread the active workspace through every/api/workspaceand/api/optimal/*call; the/graphroute remounts on workspace switch.optimal_node_filesnow returns each signal'sid;getSignal(id, workspace)passes the scope; the Nodes page drills in with the real id.Verification (live engine on :4200)
svelte-check: 0 errors (4 pre-existing warnings). Enginemix compile: clean.default/defaultlocalStorage state now recovers to OperatorOS + a populated workspace and renders the node tree./graphrenders 262 nodes / 420 edges for a populated workspace (was 0).id(fix fix(claims): honour :limit as a SQL bound; report totals from a grouped count #4) to appear; the change compiles and the/api/signals/:id?workspace=contract is verified manually.Out of scope / follow-ups
memoryandwikiare empty by data-state (no promoted facts, no curated wiki pages), not a bug.+layout.sveltecreateWorkspace({ tenant: $activeOrgId })has the same org-vs-tenant mixup on the write path and noorganizationfield. Left for a separate pass.🤖 Generated with Claude Code