feat: add external archive provider sync

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+### Changes
+
+- Added generic external archive providers with an explicit `provider:<name>` sync source, a local JSONL subprocess protocol, scoped resumable checkpoints, bounded validation imports, and source-priority safeguards.
+
+### Performance
+
+- Batched unchanged-message checks and aligned search-index row IDs during external archive replays to avoid per-message database round trips and full-index replacement scans.
+
+### Maintenance
+
 - Standardized the Makefile's build, check, snapshot, and fail-closed release targets across the crawler repositories.
 - Refreshed terminal detection and Unicode display-width dependencies.
 - Updated CrawlKit to 0.14.4, SQLite to 1.55.0, `golang.org/x/net` to 0.57.0, and replaced the retracted libc 1.74.3 with 1.74.4.

README.md

@@ -19,13 +19,14 @@
 
 Slack search is convenient until you need your own workflow, your own retention, or your own queries. `slacrawl` is a Go-based CLI that pulls Slack workspace metadata and message history into SQLite so you can inspect it without depending on the Slack UI.
 
-Data stays on your machine. You can run it in API mode, MCP connector mode, desktop mode, or a hybrid workflow. That covers one-shot syncs, live tailing over Socket Mode, connector-backed fetching, and local desktop recovery or "wiretap" style inspection from Slack Desktop artifacts already on your machine.
+Data stays on your machine. You can run it in API mode, MCP connector mode, desktop mode, external-provider mode, or a hybrid workflow. That covers one-shot syncs, live tailing over Socket Mode, connector-backed fetching, local archive imports, and local desktop recovery or "wiretap" style inspection from Slack Desktop artifacts already on your machine.
 
 ## Included
 
 - local SQLite storage with full-text search backed by SQLite FTS5
 - workspace, channel, user, and message sync
 - MCP connector sync through Codex's HTTP Slack connector or the reference Slack MCP server
+- generic external archive providers over a local JSONL subprocess protocol
 - thread reply backfill when a user token is available
 - DM and MPIM sync when a user token is available
 - incremental API history sync by default, with `--full` reserved for deliberate backfills
@@ -171,8 +172,9 @@ Treat one `slacrawl` config/database as one Slack visibility boundary. The archi
 
 - `--source bot` is an alias for `--source api`; it crawls Slack through configured bot/user tokens
 - `--source mcp` fetches through Codex's HTTP Slack connector or the reference Slack MCP server over stdio and requires a workspace ID for archive ownership
+- `--source provider:<name>` imports a configured external archive through a local subprocess and requires a workspace ID for archive ownership
 - `--source wiretap` is an alias for `--source desktop`; it reads the local Slack Desktop cache
-- `--source all` runs API first, then desktop enrichment
+- `--source all` runs API first, then desktop enrichment; external providers remain explicit
 - `[share]` is a backup/restore target for the current DB, not a second Slack source
 
 For separate company and personal archives, use separate configs with separate `db_path` and `[share].remote` values.
@@ -183,6 +185,7 @@ Choose the path that matches your setup:
 - use `sync --source bot --full` only when you want a deliberate full backfill
 - use `sync --source bot --latest-only` when you only want fresh deltas on channels that already have local history
 - use `sync --source mcp --workspace T01234567` when a configured HTTP or stdio MCP connector can read the workspace
+- use `sync --source provider:archive --workspace T01234567` when a configured local provider should import another archive
 - use `sync --source wiretap` when you want local desktop recovery only
 - use `watch` when you want desktop-local state to refresh into SQLite continuously
 
@@ -194,7 +197,7 @@ Choose the path that matches your setup:
 - `publish` exports the local SQLite archive into a git repo as compressed JSONL shards plus a manifest
 - `subscribe` configures a git-backed reader that can run without Slack credentials
 - `update` safely merges the latest git snapshot; `update --restore` performs explicit exact replacement, including historical tag/ref restores
-- `sync` performs a one-shot crawl from bot/API, MCP connector, wiretap/desktop, or both
+- `sync` performs a one-shot crawl from bot/API, MCP connector, an external provider, wiretap/desktop, or the combined API-plus-desktop mode
 - `import` imports a Slack export ZIP or extracted export directory
 - `purge` previews or deletes messages and message-owned records older than a cutoff, with optional retained-event compaction
 - `tail` listens for live events through Socket Mode, including one tail per configured workspace
@@ -362,6 +365,65 @@ go run ./cmd/slacrawl files fetch --missing --max-bytes 104857600
 go run ./cmd/slacrawl sync --source bot --latest-only --with-media
 ```
 
+## External Archive Providers
+
+An external provider is a trusted local executable that streams workspace,
+channel, user, and message records into the normal SQLite archive. Configure it
+with a unique name, then select it explicitly with `provider:<name>`:
+
+```toml
+[[providers]]
+name = "archive"
+command = "/usr/local/bin/archive-provider"
+args = ["provide", "--format", "jsonl"]
+env_allowlist = ["ARCHIVE_DB_PATH"]
+source_rank = 5
+batch_size = 1000
+```
+
+```bash
+slacrawl sync --source provider:archive --workspace T01234567
+slacrawl sync --source provider:archive --workspace T01234567 --latest-only
+slacrawl sync --source provider:archive --workspace T01234567 --full
+slacrawl sync --source provider:archive --workspace T01234567 --limit 100
+```
+
+`command` must be an absolute path; `~` and `~/...` are expanded before
+validation. The
+configured `args` are passed directly without a shell. Only a minimal runtime
+environment and variables named by `env_allowlist` reach the process, so put
+secret values in the environment rather than in TOML. `source_rank` must be
+greater than `2`; lower numeric ranks win, keeping higher-priority native data
+authoritative. Equal ranks may replace, so use a larger number for a
+lower-priority provider.
+
+`batch_size` defaults to `1000`, must be between `1` and `100000`, and is sent to
+the provider as a preferred upstream batch size.
+
+`slacrawl` writes one JSON request line to provider stdin. The request identifies
+protocol `slacrawl-provider-v1` and forwards the workspace, `since`, `full`,
+`latest_only`, channel filters, opaque saved checkpoint, batch size, and optional
+validation limit. Provider stdout must be JSONL: one `hello` record, zero or more
+`workspace`, `channel`, `user`, `message`, or `checkpoint` records, then one
+`done` record. A successful run must emit `done` and exit zero. `slacrawl`
+validates workspace ownership and the message limit, builds normalized search
+text, updates FTS and mentions, and does not replace a higher-priority canonical
+message.
+
+Checkpoints are isolated by invocation scope. A normal incremental run reuses
+the workspace checkpoint; filters, `--since`, `--full`, `--latest-only`, and
+`--limit` use separate checkpoints so bounded or partial runs cannot advance the
+normal cursor. A completed unbounded full run promotes its last checkpoint to
+the matching incremental scope. Providers should emit a checkpoint only after
+the records it covers and must honor the forwarded filters; `slacrawl` cannot
+infer provider-specific filtering.
+
+Treat provider executables as part of the archive's trust boundary. They can
+read allowlisted environment values, run with the caller's OS permissions, and
+emit private Slack data. See
+[Configuration](./docs/configuration.md#external-archive-providers) for the
+complete request and response contract.
+
 ## Git Archive Sharing
 
 Use git-share mode when one machine has Slack credentials and should publish snapshots, while other machines only need a local read-only archive.

SPEC.md

@@ -26,6 +26,7 @@ V1 scope:
 - FTS5 search
 - raw SQL access
 - desktop-local Slack discovery on macOS and Linux
+- external archive ingestion through a local JSONL provider protocol
 
 Out of scope for V1:
 
@@ -43,7 +44,7 @@ Out of scope for V1:
 - language: Go
 - schema: single-workspace default, multi-workspace-ready
 - search: FTS5 first, embeddings later
-- source precedence: user-token API, then bot-token API and slack-export imports, then desktop-local cache
+- source precedence: user-token API, then bot-token API and slack-export imports, then desktop-local cache; external providers must use a numeric rank greater than `2`, and equal ranks may replace
 - files: metadata only in DB for V1
 - future file-blob backup must store Git-share media as gzip-compressed files, import those files back into raw local cache layout, and keep legacy raw-media import compatibility
 - desktop-local source: supported Slack Desktop cache paths on macOS and Linux
@@ -155,13 +156,14 @@ Purpose:
 
 Expected flags:
 
-- `--source api|desktop|all`
+- `--source api|bot|desktop|wiretap|mcp|connector|all|provider:<name>`
 - `--workspace <id>`
 - `--channels <csv>`
 - `--exclude-channels <csv>`
 - `--since <timestamp>`
 - `--full`
 - `--latest-only`
+- `--limit <messages>` for bounded external-provider validation imports
 - `--concurrency <n>`
 - `--auto-join=<bool>`
 
@@ -310,6 +312,15 @@ Credential model:
 - `[sync].auto_join` defaults to `true` and controls whether API sync attempts to join public channels before retrying history
 - `[sync].exclude_channels` is an optional case-insensitive list of channel names to skip during API sync and merges with `--exclude-channels`
 
+External provider config:
+
+- each `[[providers]]` entry has a unique lowercase `name` without whitespace, slashes, or colons
+- `command` is required, expands `~` or a leading `~/`, and must resolve to an absolute path
+- `args` are passed directly to the command without a shell
+- `env_allowlist` names additional environment variables forwarded alongside the minimal runtime environment
+- `source_rank` is required and must be greater than `2`; lower numeric ranks win during message reconciliation, while equal ranks may replace
+- `batch_size` defaults to `1000`, must be between `1` and `100000`, and is forwarded as an upstream batching hint
+
 Share config:
 
 - `[share].remote` points at the git remote that stores compressed archive snapshots
@@ -352,6 +363,44 @@ Share config:
 14. update FTS rows and mentions
 15. write checkpoints, channel skips, and join attempts
 
+### External provider sync
+
+1. resolve `provider:<name>` against `[[providers]]` and require a workspace ID
+2. choose a checkpoint key from the provider name, workspace, and normalized invocation scope
+   - the unfiltered incremental run uses the workspace checkpoint
+   - `--since`, `--full`, `--latest-only`, channel filters, exclusions, and `--limit` use isolated scope checkpoints
+3. start the absolute command directly with configured args and a minimal environment plus `env_allowlist`
+4. send one `slacrawl-provider-v1` JSON request on stdin containing `workspace_id`, `since`, `full`, `latest_only`, channel filters, saved opaque `checkpoint`, `batch_size`, and optional positive `limit`
+5. consume JSONL from stdout
+   - the first record must be `hello` with the matching protocol
+   - data records may be `workspace`, `channel`, `user`, or `message`
+   - `checkpoint` records must identify the requested workspace and contain a nonempty opaque value
+   - the terminal `done.records` count must equal the number of data records consumed; checkpoints are not counted
+6. reject cross-workspace records, missing required identities or message channels, records after `done`, and more messages than `limit`
+7. build normalized message search text, extract mentions, update FTS, and preserve existing messages with a lower numeric source rank
+8. when a checkpoint is present, atomically commit it with the pending record batch; committed batches and checkpoints remain resumable if the process later fails
+9. require `done` plus a zero exit status for overall success
+10. after a successful unbounded `--full`, promote the final full checkpoint to the matching incremental scope
+
+Provider v1 response records:
+
+- `hello`: `type`, `protocol`, optional provider implementation name
+- `workspace`: `type`, requested workspace `id`, optional metadata and `raw_json`
+- `channel`: `type`, `workspace_id`, `id`, optional metadata and `raw_json`
+- `user`: `type`, `workspace_id`, `id`, optional profile data and `raw_json`
+- `message`: `type`, `workspace_id`, `channel_id`, Slack `ts`, optional `user_id`, `thread_ts`, `text`, and `raw_json`
+- `checkpoint`: `type`, `entity_type = "workspace"`, requested workspace `entity_id`, and opaque nonempty `value`
+- `done`: `type`, data-record count in `records`, plus optional provider quality counters
+
+The provider owns the interpretation of `since`, `full`, `latest_only`,
+`channels`, and `exclude_channels`; the consumer enforces workspace ownership
+and the positive message limit. A message channel must already exist locally or
+be emitted earlier in the stream. Unknown nonempty message user IDs are reserved
+as sparse workspace-bound profiles that a later real user record can enrich.
+Incremental imports enforce stored retention floors; `--full` or an explicit
+`--since` older than the floor is a deliberate restore that may reintroduce
+purged history.
+
 ### Git share sync
 
 1. clone or open the configured share repo
@@ -384,6 +433,7 @@ Share config:
 cmd/slacrawl/
 internal/cli/
 internal/config/
+internal/provider/
 internal/share/
 internal/slackapi/
 internal/slackdesktop/

config.example.toml

@@ -55,6 +55,16 @@ max_pages = 250
 # command = "npx"
 # args = ["-y", "@modelcontextprotocol/server-slack"]
 
+# Generic local archive provider (selected with --source provider:archive).
+# Keep this block commented until the executable is installed.
+# [[providers]]
+# name = "archive"
+# command = "/usr/local/bin/archive-provider"
+# args = ["provide", "--format", "jsonl"]
+# env_allowlist = ["ARCHIVE_DB_PATH"]
+# source_rank = 5 # must be greater than 2; lower ranks win; equal ranks may replace
+# batch_size = 1000 # default 1000; valid range 1..100000
+
 [sync]
 concurrency = 4
 repair_every = "30m"