Build completion on shared query grammar

[Sinity]

Purpose

Build shell completion, fuzzy selection, and query-builder metadata on top of the same query grammar, AST, action contracts, and read-view metadata used by CLI, MCP, daemon, Python, and web routes.

Completion is not a separate language. It is a read-only projection of the parser/lowerer/action/view registries, plus bounded archive-backed value providers.

Why this exists

The query language is now powerful enough that mistyped fields, stale examples, and unsupported future syntax are expensive. Completion should make valid fields, structural units, operators, values, read views, formats, and then actions discoverable without adding a natural-language layer or a second parser.

The product outcome is fewer invalid commands, fewer hidden surface mismatches, and a shared metadata feed that future MCP/web query builders can consume without reimplementing query semantics.

Current implementation reality

Polylogue already has useful completion plumbing:

• polylogue completions emits bash/zsh/fish completion setup.
• polylogue/cli/shell_completion_values.py contains static and archive-backed completion providers.
• polylogue/archive/query/fields.py defines QueryFieldDescriptor, CompletionSource, and query_completion_sources() for spec-field metadata.
• polylogue/archive/query/expression.py owns the Lark-backed query grammar, EXPRESSION_FIELD_REGISTRY, and structural query-unit metadata for exists message(...), exists action(...), and exists block(...).
• Public action contracts live in polylogue.operations.action_contracts; the CLI re-exports them for compatibility. CliActionContract.completion_context and action_completion_contexts() expose action-contract completion contexts.
• SessionViewProfile and the read-view profile APIs from Define evidence-linked session view profiles #1997 expose read-view metadata to CLI, Python, and MCP. read --view choices and completion come from READ_VIEW_PROFILES; read --format choices and completion come from SessionViewProfile.formats (feat(cli): complete read formats from view profiles (#1844) #2034).
• polylogue/cli/select.py owns the read-only query-backed selector with fzf/UI fallback; select is now registered as a query verb and action contract (feat(cli): expose query-backed select verb (#1844) #2036).
• polylogue.archive.query.completions owns the shared QueryCompletionCandidate payload and registry-backed candidate providers. polylogue query-completions, Polylogue.query_completions(...), MCP query_completions, and GET /api/query-completions all expose the same structured query/action completion metadata (feat(cli): expose query completion metadata (#1844) #2037/feat(query): share completion metadata across API and MCP (#1844) #2038/feat(daemon): expose query completion metadata endpoint (#1844) #2039), including registry provenance, danger/stale fields, replacement fields, full descriptions, and readable count/date field discoverability.
• Completion matrix tests bind shell completion coverage to registered action/query contexts.

Stale wording to avoid: there is no long-lived "flat compiler floor." Current compact queries and explicit Boolean/structural predicates are being absorbed into #2006's Lark grammar and AST/lowering path. Completion should follow that canonical path as it lands, not preserve a parallel grammar.

Source-of-truth rule

Completion metadata must come from existing parser/action/view registries wherever possible:

• query field names and examples from EXPRESSION_FIELD_REGISTRY;
• structural units and structural fields from STRUCTURAL_QUERY_UNIT_REGISTRY / structural_query_units() / structural_query_fields();
• completion value source names from QueryFieldDescriptor.completion_source / query_completion_sources();
• action and danger metadata from ACTION_CONTRACTS / CliActionContract;
• read views from SessionViewProfile / read_view_profile_payloads() rather than ad hoc lists;
• parser errors and unsupported syntax from feat(query): implement the full query DSL substrate #2006 typed errors, not completion-only validation.

If completion needs to duplicate a command, field, view, or output contract by hand, treat that as a design smell and either move the metadata to the owning registry or make the duplication mechanically checked.

Query-substrate boundary

Completion must expose the shared query grammar. It may inspect incomplete text to decide what kind of candidate to offer, but it must not implement a second grammar or accept syntax the real parser/lowerer rejects.

Rules:

• Supported syntax may be suggested normally.
• Future feat(query): implement the full query DSL substrate #2006 syntax may appear only as disabled/help metadata marked unsupported, never as an active completion.
• Unknown-field suggestions should be derived from the grammar field registry.
• Structural-unit and structural-field suggestions should be derived from the grammar's structural registry.
• Value suggestions should be tied to the field's declared completion source.
• Unsupported parser/lowerer errors should be surfaced as "why not" messages where the adapter can show them.

Completion should suggest

• fields: origin, repo, tag, title, since, until, tool, action, words, messages, has, near, lineage, etc., as they are supported by the parser/lowerer;
• structural units: message(, action(, block( after exists;
• structural fields inside exists <unit>(...), such as role:, text:, type:, path:, tool:, and command: when supported by that unit;
• values from the live archive: origins, repos, tags, tools, actions, session ids, and path-like values where there is a real projection;
• operators supported by the field and lowerer;
• then actions and valid action-specific options from action contracts;
• read views and formats from view-profile metadata;
• degraded/unavailable states when the archive cannot answer dynamic values;
• danger/confirmation metadata for destructive actions such as delete/mark variants.

Completion candidate record

Candidate records should carry enough structure for shells, fuzzy pickers, and future query builders:

value
insert
replace_start / replace_end
display
kind
group
description
score
source
stale
danger
unsupported_reason optional
preview_command optional

Completion is read-only. It may suggest destructive actions with danger metadata, but it must not execute them or bind them to single-key fuzzy shortcuts.

Implementation program

Implement completion as a usable query/action affordance, not as DTO ceremony.

1. Keep the structured completion candidate model in polylogue.archive.query.completions; CLI shell completion should only adapt that metadata into Click CompletionItems, while polylogue query-completions --format json, Polylogue.query_completions(...), and MCP query_completions expose the shared payload for non-shell consumers (feat(cli): expose query completion metadata (#1844) #2037/feat(query): share completion metadata across API and MCP (#1844) #2038).
2. Expose query field candidates directly from EXPRESSION_FIELD_REGISTRY, plus readable count/date field metadata from COUNT_QUERY_FIELD_REGISTRY and DATE_QUERY_FIELD_REGISTRY, including descriptions and examples.
3. Expose structural unit/field candidates directly from feat(query): implement the full query DSL substrate #2006 grammar metadata.
4. Keep parser-context completion for common partial query states: empty query, partial field name, after field:, after exists, inside exists <unit>(...), after then, and after read --view. Already-covered states must stay tested rather than reimplemented.
5. Keep dynamic archive value providers bounded and graceful: missing/locked/stale archive returns stale or empty candidates, never a traceback.
6. Add action candidates from ACTION_CONTRACTS, carrying destructive/danger metadata.
7. Keep read-view and read-format candidates sourced from SessionViewProfile metadata.
8. Keep the read-only select verb wired through the existing query-backed selector and action contract. Future picker work should enrich candidate metadata rather than bypassing that path.
9. Reuse query_completions / QueryCompletionCandidate payloads from any future interactive query-builder UI; CLI, Python API, MCP, and the daemon web API already consume the shared contract, so new consumers should not reimplement query metadata.

The implementation should keep landing surfaced behavior. Do not stop at a registry or model with no shell/API/MCP consumer.

Acceptance criteria

• Field-name completions are generated from EXPRESSION_FIELD_REGISTRY.
• Structural-unit and structural-field completions are generated from feat(query): implement the full query DSL substrate #2006 grammar metadata.
• Value completions are generated from field completion sources and archive-backed providers where available.
• Read-view completions come from view-profile metadata, not ad hoc lists.
• Completion suggestions match parser support, action contracts, and view profiles.
• Completion endpoints never execute mutations.
• Unknown field typos produce useful suggestions.
• Dynamic value completion handles empty, stale, locked, and unavailable archive state without crashing.
• Dangerous/destructive candidates carry danger/confirmation metadata.
• At least one test proves a deleted/renamed field, read view, action, structural unit, or structural field disappears from completion automatically through the shared registry.
• Completion metadata is consumable by Python API, MCP, daemon web API, and future query-builder clients without reimplementing query semantics.

Non-goals

• Do not build a separate query parser.
• Do not implement natural-language search here.
• Do not expose write actions through a fuzzy picker without explicit confirmation.
• Do not add a web/query-builder facade that bypasses the shared metadata contract.
• Do not suggest feat(query): implement the full query DSL substrate #2006 future syntax as active unless the lowerer can execute it; disabled/help candidates must be marked unsupported.

Related issues

• Full query grammar and AST/lowerers: feat(query): implement the full query DSL substrate #2006.
• View-profile registry for read views: Define evidence-linked session view profiles #1997.
• Shared MCP/Python/API query/read contracts: Keep MCP and Python API on shared query/read contracts #1825.
• Parent product epic: EPIC: Polylogue as local AI-work evidence cockpit #1807.

---

Current state (audited 2026-06-27)

Readiness: ~80% already in code · scope M · conflict risk medium · depends on: #2006

Assessment & remaining work: The core completion infrastructure is complete: polylogue/archive/query/completions.py provides QueryCompletionCandidate and registry-driven candidates from EXPRESSION_FIELD_REGISTRY, QUERY_UNIT_DESCRIPTORS, and ACTION_CONTRACTS; polylogue/cli/shell_completion_values.py wraps these for shells with graceful empty/locked archive degradation; GET /api/query-completions, MCP query_completions, and Polylogue.query_completions() all expose the shared payload. Two ACs remain unimplemented: (1) "unknown field typos produce useful suggestions" requires adding a difflib/levenshtein nearest-match helper in completions.py and surfacing it from query_field_candidates when no candidates match; (2) an explicit registry-binding test in tests/unit/archive/test_query_metadata.py (or a dedicated file) must prove that removing a field from EXPRESSION_FIELD_REGISTRY causes it to disappear from query_field_candidates() output — the design guarantees this but no test currently asserts it. The structural-unit AC referencing "#2006 grammar metadata" is blocked on issue #2006 (open Lark DSL work); current QUERY_UNIT_DESCRIPTORS-backed implementation satisfies the spirit but not the exact letter until #2006 lands.

Primary files: polylogue/archive/query/completions.py, polylogue/archive/query/metadata.py, polylogue/cli/shell_completion_values.py, polylogue/cli/commands/completions.py, tests/unit/archive/test_query_metadata.py, tests/unit/cli/test_completions_contract.py, tests/unit/cli/test_completion_matrix.py

Grounded re-assessment from the 2026-06-27 backlog triage; verify against source before acting.

[Sinity]

Audit addendum (2026-06-13) — compact completion protocol (richest chatlog UX layer, only partially in the body):

Completion candidate records must include enough structure for shell adapters and fuzzy pickers: value, insert, replace_start, replace_end, display, kind, group, description, score, source, stale, danger, and optional preview command. Completion must be grammar-position-aware, live-data-aware, context-aware, privacy-aware, and authority-aware.

Dependent completions are required: previous query filters narrow later value candidates. Shell integration remains read-only. Destructive candidates may be displayed but must be ranked/marked as dangerous and must never be executed by completion or picker single-key shortcuts.

Add tests for partial query states, dependent values, stale-cache fallback, privacy redaction, danger metadata, debug scoring, and shell adapter schema validity.

[Sinity]

Wave 3 UX overlay: completion should be grammar-aware but substrate-honest

Completion/fuzzy/query builder should surface only supported grammar by default. It may preview #1879 future forms as disabled/help text, but must not suggest syntax that current parser/lowering rejects.

First implementation after #1842

• complete fields from actual archive values: origin, repo, tag, tool, action, path prefix;
• complete then actions from registered action contracts (Replace output assurance with code-coupled action contracts #1816 once available);
• expose typed unsupported errors from parser as “why not” messages;
• add --explain/parse preview once #1879 AST exists.