refactor(engine): rework pipeline module#58
Merged
Conversation
Remove run eviction (max_completed_runs) — runs are in-memory and lost on restart; explicit DELETE endpoints handle cleanup. Flatten RunState inner type from RunStateInner struct to plain HashMap. Remove config field and builder method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ConcurrencyPolicy as a graph-level policy alongside RetryPolicy and TimeoutPolicy. Available on both Graph.concurrency and EngineSection.concurrency, with graph taking precedence. Remove max_completed_runs eviction — runs are in-memory and lost on restart; explicit DELETE endpoints handle cleanup. Simplify RunState inner type to plain HashMap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrite module-level docs across all pipeline files. Document the 5-phase run lifecycle, DAG scheduling model, CompletionGuard semantics, plan compilation steps, config merge behavior, and run state concurrency model. Add doc comments to all public and important private types, methods, and fields. Fix stale references and typos. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…paths Standardize doc style across all pipeline submodules: add module-level summaries, `# Errors`/`# Panics` sections on public methods, one-line docs on private executor methods, and replace inline qualified paths with proper imports (RunStatus, AudialExtraction, Entities, RunOutput, ConcurrencyPolicy, Error/ErrorKind, HashSet). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…y crate High-priority fixes: - Preserve Cancelled status in finalize() instead of overwriting it with the orchestrator's computed status - Detect cancellation in run_status() — return Cancelled when all node errors are cancellation errors - Fail-fast: cancel sibling nodes when any node fails, avoiding wasted LLM/OCR calls on a doomed pipeline Medium-priority fixes: - Validate ConcurrencyPolicy.max_nodes >= 1 to prevent deadlocks from a zero-permit semaphore - Concurrent fan-in via futures::select_all instead of sequential receiver draining, preventing backpressure stalls - Fix RuntimeConfig::merge to preserve base version instead of unconditionally taking the override's version - Implement explicit Default for RuntimeConfig with version=1, matching the serde default - Validate graph before injecting engine-level policy defaults so validation errors reflect user input Low-priority fixes: - Use Arc<RuntimeConfig> in NodeExecutor to avoid cloning the full config tree per node - Extract fan_out() helper, eliminating duplicated send logic in process_envelopes and execute_import Cleanup: - Remove dead nvisy-registry crate (already merged into nvisy-engine, not in workspace members, no dependents) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace intra-doc links to private submodules (config, plan, orchestrator, executor, runs, analytics) with plain text in mod.rs, and remove broken self-link in default.rs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace broken intra-doc links to private or removed items with plain backtick text across 6 crates: - nvisy-engine: SharedData, CryptoService, Import, ParallelContext, SequentialContext, Redaction, OperationContext, ContentHandle, ContextHandle - nvisy-pattern: CsvDictionaryError, DictionaryLoadError, from_bytes, RegexPattern::confidence, DictionaryPattern::confidence - nvisy-ocr: OcrEngine (3 occurrences) - nvisy-rig: BaseAgent (2), CvPromptBuilder, GenPromptBuilder, NerPromptBuilder, OcrPromptBuilder - nvisy-codec: AudioHandler, PdfRenderer, parallel_render, Handler, TextOutput - nvisy-server: ApiRouter, request/response modules, ErrorResponse Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Upgrades pyo3, pyo3-async-runtimes, and pythonize to 0.28 to support Python 3.14. Migration changes: - Python::with_gil() → Python::attach() - PyObject → Py<PyAny> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Group run_timeout_ms and channel_buffer into a dedicated ResourceLimits struct with sensible defaults (channel_buffer=256, run_timeout_ms=None). Flattened via serde so the TOML shape is unchanged. This removes scattered Option unwrapping in Engine::run and makes compile() take a plain usize for channel_buffer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.