feat(packages): add voc-data-pull, the VoC raw-data package with connect-time auto-install and self-activating daily sync

[Eric Smith (motionBuilderEric)]

What this is

voc-data-pull is an installer package that pulls raw voice-of-customer (VoC) data - product reviews, support conversations, community posts, and ad comments - from a connected platform into standardized files in the org brain: one file per item, a human-readable header and content with the full machine-parseable metadata record in a collapsed block. Once installed, the package activates itself: a standing instruction has Runneth create one daily sync routine per connected platform, so the backfill and all subsequent pulls run with zero human prompting. Creative strategy's VoC packages build on these files. Tracking: PDEC-9193 (PR 1 of 2; PR 2 in agent-builder makes the backend feed package intent on connect and wake the conversation so the activation instruction runs).

Contents

• packages/voc-data-pull/package.json - installer manifest (schema v1): a directory resource installing the skill to agent_skills/voc-data-pull, and a package_instruction resource (instructions/activation.md) that lands in the agent's standing instructions. installPolicy: auto so a matching intent installs with no manual step; updatePolicy: auto so recipe fixes roll out on later syncs.
• packages/voc-data-pull/instructions/activation.md - the self-activation contract. When a covered platform is connected and its voc-sync-<platform> routine does not exist, Runneth creates it (pinned cron 0 6 * * *, silent-success/delivered-failure delivery) and kicks the first sync run in the background. Routine absence - not folder state - is the setup trigger, so cancel-and-reconnect re-sets-up cleanly. Pulls never run inside the user's conversation; the routine prompt defers all run behavior to the skill. Junip is gated on a working key check before its routine is created.
• packages/voc-data-pull/skill/SKILL.md - the pull workflow: resolve the connection path (Pipedream OAuth via the integrations CLI, stored customer key via secure-fetch, Motion native for Meta ad comments), follow the platform recipe, write id-keyed files, report results. A "Recurring sync runs" section owns routine-run behavior: the incremental window (empty folder → trailing 12 months backfill; otherwise newest item's created_at minus 2 days, updated_at bound for tickets - daily runs pull only new data, never re-page history), multi-account iteration, and disconnect handling (no-op run, never self-pause/cancel). Per-source write policy: reviews skip existing files, tickets overwrite on fresher updated_at, comments/posts overwrite inside the pull window (engagement mutates). Hard boundaries: read-only against platforms; bounded pulls with the 12-month window as the coverage contract (page caps are per-run batching, never a coverage limit); never write into /agent/brain/integrations/<source>/; author_contact stays null and raw payloads are not persisted (both flagged halves of the same pending PII/template policy call).
• File layout: everything lands under /agent/brain/data-sources/voc/<platform>/ - one flat folder per platform, type-prefixed id-keyed filenames (review-<id>.md, ticket-<id>.md, comment-<id>.md, post-<id>.md). The id-keyed path is the contract that re-pull dedupe, ticket overwrite, and the incremental window depend on. Each file is three parts: H1 headline, bold-label human header, readable content, then the unified metadata as a collapsed fenced-YAML block (one flat shape for every item, all fields always present, null when the source lacks the concept).
• skill/references/platform-recipes.md - per-platform endpoints, pagination, discovery steps, and unified-template field mappings (including source_url and reply_count semantics per platform), with honest evidence levels: Judge.me/Gorgias/Intercom live-verified; Trustpilot/Yotpo doc-grounded pending first connects; Reddit needs-setup with org-specific pull targets; Junip blocked on a dead key; Okendo/Stamped secrets-path (no Pipedream app exists - customer API key + secure-fetch).
• skill/templates/ - copyable skeletons for all four output shapes: review, support conversation (header top, full conversation below), ad comment (replies are their own files via parent_ref), and community post.
• package-index.json - the first indexed package. Categories list one integration:<slug> per VoC platform (judge_me, trustpilot, yotpo, junip, gorgias_oauth, intercom, reddit, okendo, stamped) because the VM reconciler matches literal connected slugs - no invented family category. Slugs verified against the Builder integration registry. Source type backend-github (fetched through the backend's package cache). indexRevision uses a date-sequence scheme (2026-07-22-01).
• scripts/validate-runneth-package-index.mjs - aligned with the current package-manager contract (package.json manifests, installPolicy, packageManagerVersion 1|2, manifest v2 tasks/workflows, github/backend-github sources), with shared manifest assertions (semver required for v1 and v2), a header pinning the agent-builder schema files it mirrors, and packageManagerVersion included in the fleet-gate fingerprint so a manifest-contract flip cannot bypass approval. Overlaps fix(package-index): validate current package contract #152 by design; whichever merges first, the other rebases clean.

This is an installer package under packages/, not a use-case-library card: no site catalog changes, and validate-catalog.mjs is untouched.

How it installs and activates

Connected-integration package intent selects this package via its integration:<slug> categories and the desired-state installer installs it. Two triggers: the connect-time dispatch shipping in agent-builder (automatic - and its post-connect turn is what makes the activation instruction run with zero human words), or package intent add-integration <slug> in a sandbox (manual path, works today - also the route for the secrets-path platforms, which have no Pipedream connect). On the manual path, activation happens on the next conversation turn or prompt.

Validation

• node --test scripts/validate-runneth-package-index.mjs - passes except the intentional fleet-approval gate (see CI note).
• node --test scripts/validate-catalog.mjs - passes, untouched surface.
• Manifest and index entry are field-for-field consistent (enforced by the validator's mirror test).
• Live E2E on a fresh sandbox with the agent-builder PR: Judge.me connect → auto-install → activation turn → voc-sync-judge_me created (cron 6am) → backfill ran and wrote correctly shaped files → a manual re-run used the incremental window (newest-minus-2-days, 0 re-writes). Idempotency verified: re-asking and reconnecting found the existing routine and created nothing.

Open items (surfaced, not solved here)

• The unified metadata template is proposed pending Eric/Rachel sign-off; the package builds against it as the contract.
• PII policy call: author_contact is nulled and raw platform payloads are not persisted - both halves of the same pending decision, flagged in the skill rather than silently resolved.
• Trustpilot/Yotpo recipes are doc-grounded until their first live connects; Junip needs a working key; Reddit pull targets are org-specific and need setup.
• Package installs on Azure-mounted staging VMs currently hit a pre-existing chmod/EPERM bug in the runneth-tools installer (mounted-filesystem contract); fix lands separately in agent-builder.

CI note

The fleet-approval gate fires (new package with installPolicy: auto) - needs the runneth-fleet-change-approved label after core engineering approval.