Parallelize full indexing with explicit analyzer concurrency contracts

[marco0560]

Problem

codira index --full currently reparses all indexed files through a sequential analysis loop, then persists results through a single backend write session. This is deterministic but leaves substantial performance on the table for large repositories, especially when all files are selected for reindexing.

Current relevant flow:

• src/codira/indexer.py::_plan_index_run(full=True) selects all current paths for reindexing.
• src/codira/indexer.py::_collect_indexed_file_analyses analyzes indexed_paths sequentially.
• src/codira/indexer.py::_persist_indexed_file_analyses persists analyzed results sequentially.
• src/codira/contracts.py::LanguageAnalyzer.analyze_file is a per-file analyzer contract.
• src/codira/contracts.py::IndexWriteSession owns mutable backend write lifecycle for one index pass.

The natural parallelization boundary is therefore file analysis, not backend persistence.

Goal

Add opt-in parallel analysis for indexing while preserving:

• deterministic index results
• deterministic diagnostics and warnings
• existing backend transaction semantics
• plugin isolation
• reproducible configuration
• safe defaults

The first target should be codira index --full, but the implementation should also work for incremental runs where many changed files are selected.

Non-goals

• Do not parallelize backend writes initially.
• Do not let analyzers own repository-level scheduling policy.
• Do not require third-party analyzers to be thread-safe by default.
• Do not weaken existing deterministic ordering of reports, failures, warnings, or persisted rows.

Proposed architecture

Core-owned scheduling

Parallelism should be implemented in codira core, around the analysis phase in src/codira/indexer.py.

Recommended execution shape:

1. Scan repository and build the existing deterministic IndexPlan exactly as today.
2. Prepare backend storage exactly as today.
3. Analyze selected files concurrently in core.
4. Collect (path, metadata_snapshot, AnalysisResult) plus warnings/failures.
5. Sort/merge results by deterministic path order.
6. Persist through the existing single IndexWriteSession sequentially.
7. Rebuild derived indexes, persist runtime inventory, and commit exactly as today.

This keeps the storage contract stable and avoids concurrent writes into SQLite/DuckDB sessions.

Process pool as the preferred default strategy

Use process-based workers as the preferred strategy for CPU-bound analyzer work:

• avoids Python GIL limits for pure-Python AST walking
• isolates plugin instance state
• protects against analyzer-level thread-unsafety
• reduces risk from module-global mutable state

Thread-based execution may still be useful for analyzers dominated by I/O or C extensions that release the GIL, but it should not be the default until plugin contracts explicitly permit it.

Per-worker analyzer instances

Workers should instantiate and configure analyzers independently rather than sharing analyzer objects across workers.

This matters because first-party analyzers currently store configuration on the analyzer instance, for example path filters and emit flags. Sharing a configured instance across threads might be safe for some analyzers, but the current LanguageAnalyzer protocol does not declare this.

Contract updates

Analyzer concurrency declaration

Extend the analyzer contract with an optional concurrency declaration. Possible shape:

@dataclass(frozen=True)
class AnalyzerConcurrencyDeclaration:
    analyzer_name: str
    analyzer_version: str
    supports_process_workers: bool
    supports_thread_workers: bool
    reentrant_after_configure: bool
    notes: tuple[str, ...] = ()

Add an optional protocol, for example:

@runtime_checkable
class ConcurrencyDeclaringAnalyzer(Protocol):
    def analyzer_concurrency_declaration(self) -> AnalyzerConcurrencyDeclaration: ...

Recommended initial defaults:

• missing declaration means process workers are allowed only if the analyzer can be constructed/configured in the worker from registry/config state
• missing declaration means thread workers are not allowed
• thread workers require explicit reentrant_after_configure=True and supports_thread_workers=True

Capability contract export

Update codira caps --json to include analyzer concurrency metadata.

Suggested per-analyzer fields:

{
  "analyzer_name": "python",
  "analyzer_version": "6",
  "concurrency": {
    "process_workers": true,
    "thread_workers": false,
    "reentrant_after_configure": true,
    "declaration_status": "declared"
  }
}

If undeclared, emit degraded metadata explicitly rather than omitting it.

Runtime inventory and cache boundary

If concurrency mode affects emitted artifacts, ordering, warnings, embedding payloads, or failure behavior, treat it as an index/cache reuse boundary.

If the implementation guarantees identical artifacts independent of concurrency mode, then concurrency should not invalidate existing file content hashes by itself. The tests must prove this.

Third-party plugin behavior

Document that third-party analyzers are not assumed thread-safe. Third-party plugins should be safe under process workers if they can be discovered, constructed, configured, and invoked independently in each worker.

Configuration updates

Add index concurrency configuration to the persistent config system.

Suggested config table:

[index.concurrency]
enabled = false
strategy = "process"
max_workers = 0
min_files = 32
chunk_size = 1

Field semantics:

• enabled: default false initially for safe rollout.
• strategy: one of "off", "process", "thread", "auto".
• max_workers: 0 means auto-detect bounded worker count; positive integer sets an explicit cap.
• min_files: minimum number of indexed paths before parallel scheduling is used.
• chunk_size: optional batching size for worker dispatch; default should preserve responsive failure reporting and deterministic merge order.

CLI overrides:

codira index --jobs N
codira index --concurrency off|process|thread|auto

Recommended behavior:

• --jobs 1 forces serial analysis.
• --concurrency off forces current behavior.
• --concurrency process --jobs N uses process workers if all selected analyzers are process-compatible.
• --concurrency thread --jobs N fails fast if any selected analyzer lacks explicit thread support.
• --concurrency auto chooses process workers unless disabled by capability/config constraints.

Environment override, if desired:

CODIRA_INDEX_JOBS=N
CODIRA_INDEX_CONCURRENCY=off|process|thread|auto

Persistent config surfaces to update:

• config schema
• default config rendering
• codira config init --full
• codira config validate --json
• effective config docs

Implementation strategy

Phase 1: Refactor analysis into serial-compatible worker units

Extract a single-file analysis helper from _collect_indexed_file_analyses.

The helper should accept only serializable inputs where possible:

• root path
• source path
• metadata snapshot or raw metadata
• analyzer identity/config selection information

It should return a structured result object:

• path
• parsed file tuple when successful
• analysis failure when failed
• warning rows collected during analysis

Keep current serial behavior as the baseline path.

Phase 2: Add process-worker scheduler

Add a scheduler that:

• groups paths by selected analyzer identity if needed
• initializes/configures analyzers inside each worker
• runs single-file analysis
• returns structured worker results
• merges by original path order

The parent process should remain responsible for backend session operations.

Phase 3: Add config and CLI controls

Wire persistent config and CLI overrides into index_repo or an index request object.

Avoid passing many loosely related keyword arguments into index_repo; prefer a small request/config dataclass if the option surface grows.

Phase 4: Add analyzer concurrency metadata

Implement concurrency declarations for first-party analyzers.

Initial conservative recommendation:

• process workers: true for first-party analyzers that can be constructed/configured in workers
• thread workers: false unless explicitly stress-tested
• reentrant after configure: true only when analyze_file does not mutate instance or module state

Phase 5: Optional thread scheduler

Implement thread scheduling only after first-party analyzer reentrancy tests exist.

Thread mode should remain opt-in and fail closed for undeclared analyzers.

Reentrancy verification

A plugin is reentrant after configuration if concurrent calls to analyze_file(path, root) on independent paths cannot alter shared state or change another call's result.

Verification should include both static screening and dynamic stress tests.

Static screening

Use static checks to flag suspicious patterns inside analyze_file and functions it calls:

• assignment to self.*
• mutation of self.* containers
• global or nonlocal writes
• mutation of module-level dictionaries/lists/sets
• global parser instances reused across calls
• os.chdir
• writes to os.environ
• fixed temp file paths
• monkeypatching warnings/logging/tokenizer/parser state
• caches without locking or deterministic keys

Semgrep can help here, but it cannot prove reentrancy. It should be used as a guardrail/lint, not as the sole verification.

Potential Semgrep rules:

• flag self.$ATTR = ... inside analyze_file
• flag $OBJ.append(...), $OBJ.clear(...), $OBJ.update(...) where $OBJ is module-level mutable state
• flag global $NAME inside analyzer modules
• flag os.chdir(...)
• flag os.environ[...] = ...
• flag tempfile.NamedTemporaryFile or manual temp paths with fixed names if used unsafely

Dynamic stress tests

Add tests that compare serial and concurrent output exactly.

Required properties:

• same indexed file set
• same symbols
• same declarations/imports/documentation
• same stable IDs
• same call/reference/include edges
• same warnings after deterministic sorting
• same failures after deterministic sorting
• same embedding recompute/reuse counts when embeddings are enabled
• same final query results from sym, symlist, calls, refs, ctx where applicable

Recommended test shape:

1. Build a temporary repo with mixed Python/C/C++/JSON/Markdown/Text/Bash files.
2. Run serial full index into one output directory.
3. Run process-parallel full index into another output directory.
4. Compare backend-visible normalized rows, excluding backend-owned timestamps if any.
5. Repeat the parallel run multiple times to detect nondeterministic ordering.

Tests to add

Unit tests

• index concurrency config parsing and defaults
• CLI override precedence over config
• invalid max_workers, strategy, chunk_size, and min_files
• analyzer concurrency declaration export in caps --json
• undeclared analyzer fails closed for thread strategy
• --jobs 1 preserves serial path
• deterministic result merge by path order
• worker analysis result serialization/deserialization

Analyzer tests

For each first-party analyzer:

• serial vs process-worker analysis equality
• repeated concurrent analysis equality
• configured analyzer behavior preserved in workers
• warnings captured per file without leaking across files

For Python analyzer specifically:

• concurrent analysis of many .py files equals serial output
• module docstring, imports, constants, and type alias toggles are preserved under worker initialization
• shadowed module stable-id rebasing remains deterministic

Integration tests

• codira index --full --concurrency off
• codira index --full --concurrency process --jobs 2
• codira index --full --concurrency auto --jobs 2
• incremental index with a subset of changed files under parallel mode
• full index with one analyzer intentionally failing on one file
• full index with warnings from multiple files
• backend commit/abort behavior when a worker failure occurs
• output JSON/explain mode remains stable

Regression tests

• serial and parallel full index produce identical query results
• running parallel index repeatedly produces identical backend row ordering after normalized comparison
• thread mode fails closed for analyzers without thread declaration
• process mode works with independent analyzer instances and does not share configured instance state

Performance test or benchmark

Add a benchmark-style test or script that measures:

• serial full index wall time
• process parallel wall time with jobs=2, jobs=4, and auto
• overhead threshold for small file counts
• large repo behavior with many changed files

This should be outside normal fast unit tests unless bounded tightly.

Documentation updates

Update developer and user documentation:

• docs/configuration.md: document [index.concurrency] and precedence rules.
• relevant ADR or new ADR: document concurrency architecture and why persistence remains serial.
• CLI help text for codira index.
• plugin author docs: document analyzer concurrency declarations and reentrancy requirements.
• capability contract docs/schema: document new caps --json concurrency fields.
• testing/process docs: document required serial-vs-parallel equivalence tests for analyzer changes.

Docs must explicitly state:

• process workers are preferred for CPU-bound Python analysis because of the GIL
• thread workers require explicit analyzer support
• Semgrep/static checks are insufficient proof by themselves
• backend writes remain serial in the initial design
• deterministic output is a hard requirement

Validation

Required validation for implementation PR:

uv run codira index --full --concurrency off
uv run codira index --full --concurrency process --jobs 2
uv run codira caps --json
uv run codira config init --full --output <tmp-config>
uv run codira config validate --path <tmp-config> --json
uv run pre-commit run --all-files
uv run pytest -q

If Semgrep rules are added:

uv run semgrep --config <rules-path> packages/ src/

Also validate that serial and parallel indexes produce equivalent normalized backend results on a fixture repo.

Acceptance criteria

• codira index --full supports serial and process-parallel analysis modes.
• Parallel mode is configurable through persistent config and CLI overrides.
• Defaults are conservative and reproducible.
• Analyzer concurrency support is declared in the contract and exported via caps --json.
• Third-party analyzers fail closed for unsupported thread mode.
• Backend writes remain serial unless a later issue explicitly changes backend contracts.
• Serial and parallel full indexes produce identical normalized results.
• Repeated parallel runs are deterministic.
• Documentation explains GIL implications, process-vs-thread strategy, plugin reentrancy requirements, Semgrep limitations, and validation requirements.
• Tests cover configuration, contracts, worker scheduling, analyzer declarations, deterministic merge behavior, failure/warning handling, and integration-level serial-vs-parallel equivalence.

Open questions

• Should enabled default to false for one release, then move to auto after benchmark data and plugin declarations are stable?
• Should process-worker support require explicit analyzer declaration, or should constructible/configurable analyzers be process-compatible by default?
• Should worker initialization reuse the existing registry/config loader, or should core pass a minimized serialized plugin configuration payload?
• Should parallel mode affect runtime inventory metadata for diagnostics even if it does not invalidate file reuse?
• Should there be a hard upper bound on auto workers to avoid oversubscribing memory-heavy analyzers?