feat(ingest): unified ingest queue for dreaming and agentic dreaming (#913)

[NicholaiVogel]

Summary

Draft PR for issue #913. Unifies pipeline dreaming and agentic dreaming into one ingest operation with interchangeable inference executors. Both runners draw from the same queue, share deterministic lease/context and validate/apply phases, and differ only in where the inference comes from.

Work in progress. This branch captures the current state of the implementation.

Changes

• Adds ingest queue table and migration 088-ingest-queue (replaces legacy 044-dreaming-state)
• Adds IngestPlan contract and ontology operation batch (apply, plan, lease)
• Adds format normalizers: markdown, HTML, JSON
• Adds CLI surface: signet ingest command
• Adds API surface: POST /api/ingest routes
• Updates dreaming skill runbook prompt with temporal claim and unified queue guidance
• Retires legacy dreaming state migration in Rust schema
• Various supporting changes: extraction queue, stale lease handling, config

Type

• feat -- new user-facing feature (bumps minor)
• fix -- bug fix
• refactor -- restructure without behavior change
• chore -- build, deps, config, docs
• perf -- performance improvement
• test -- test coverage

Packages affected

• @signet/core
• @signet/daemon
• @signet/cli / dashboard
• @signet/sdk
• @signet/connector-*
• @signet/web
• predictor
• Other: platform/daemon-rs (Rust schemas)

Screenshots

N/A (no UI changes)

PR Readiness (MANDATORY)

• Spec alignment validated (INDEX.md + dependencies.yaml)
• Agent scoping verified on all new/changed data queries
• Input/config validation and bounds checks added
• Error handling and fallback paths tested (no silent swallow)
• Security checks applied to admin/mutation endpoints
• Docs updated for API/spec/status changes
• Regression tests added for each bug fix
• Lint/typecheck/tests pass locally

Migration Notes (if applicable)

• Migration is idempotent
• Daemon Rust parity reviewed or explicitly N/A
• Rollback / compatibility note included in PR description

Migration 088-ingest-queue replaces legacy 044-dreaming-state. The new table is additive and idempotent. Legacy dreaming state table is dropped (044 removed from both TS and Rust migrations).

Testing

• bun test passes
• bun run typecheck passes
• bun run lint passes
• Tested against running daemon
• N/A (draft, not ready for testing)

AI disclosure

• No AI tools were used in this PR
• AI tools were used (see Assisted-by tags in commits)

Notes

This is an early WIP draft. Many tests still need to be written and the normalizers need refinement. The IngestPlan contract and queue lease mechanism are the most stable pieces.