Skip to content

scarnyc/moltworker-poc

Repository files navigation

moltworker

A security-hardened AI agent runtime on Cloudflare Workers — production-grade defense-in-depth for autonomous AI agents.

50 MCP tools | 13 security invariants | 2300+ tests | 3 LLM providers | Cloudflare Workers

This project is archived. Moltworker shipped as a production AI agent runtime on Cloudflare Workers before being permanently torn down on April 11, 2026. The code, architecture, and documentation remain as a reference implementation for defense-in-depth AI agent security. See Project Status for details.

What Is This?

Moltworker is a production-grade reference implementation of defense-in-depth for autonomous AI agents. It ran real Telegram conversations, daily briefings, and integrations with Jira, Google Workspace, Granola, and other services using real credentials across 28 development waves over 4 weeks. The codebase, architecture, and 2300+ tests remain as a blueprint for building secure agent runtimes.

The key architectural insight is strict separation of concerns across trust boundaries: the container where the agent reasons has zero real secrets, and the Worker where tools execute has zero agent reasoning. The container's OPENAI_API_KEY is actually a sandbox authentication token — when the container makes an LLM call, the Worker intercepts it, swaps the sandbox token for the real API key, and forwards the request through Cloudflare's AI Gateway. The same pattern applies to every tool call: the container requests, the Worker authorizes, injects credentials, executes, audits, and returns sanitized results.

Moltworker is built on Cloudflare's full platform stack: Workers, Containers (Firecracker VMs), D1, R2, KV, Vectorize, AI Gateway, Zero Trust Access, Browser Rendering, Workers AI, Durable Objects, and Cron Triggers. Every external action flows through a 16-phase security pipeline before execution.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                        CLOUDFLARE EDGE                                  │
│                                                                         │
│  Layer 1: Zero Trust Access (JWT)  +  AI Gateway (rate limits, cache)   │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │  Layer 2: WORKER (Hono)                                           │  │
│  │                                                                   │  │
│  │  Auth + routing ──> 16-phase MCP pipeline ──> Tool execution      │  │
│  │  LLM proxy ──> AI Gateway ──> OpenAI / Anthropic / Google         │  │
│  │  Credential vault ── D1 audit logger (SHA-256 chain) ── PII scrub │  │
│  │                                                                   │  │
│  │  ┌─────────────────────────────────────────────────────────────┐  │  │
│  │  │  Layer 3: OPENCLAW CONTAINER (Firecracker VM)               │  │  │
│  │  │                                                             │  │  │
│  │  │  Agent reasoning ── Skill runtime ── R2 FUSE workspace      │  │  │
│  │  │  Local MCP tools (shell, git, CLI, web crawl)               │  │  │
│  │  │                                                             │  │  │
│  │  │  NO real API keys ── NO direct external access              │  │  │
│  │  │  All egress through Worker proxy                            │  │  │
│  │  └─────────────────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────────────────┘  │
│                                                                         │
│  Layer 4: CODE CONTAINER                                                │
│  Isolated code execution with dual HITL approval.                       │
└─────────────────────────────────────────────────────────────────────────┘

Data flow:
  Telegram ──> Worker webhook ──> Delegation ──> OpenClaw Container
  Container ──> Worker MCP server ──> [16-phase pipeline] ──> External APIs
  Container ──> Worker LLM proxy ──> AI Gateway ──> LLM providers

The container calls the Worker for everything. The Worker injects credentials, enforces policy, and audits every action before forwarding. The container never sees real API keys — it authenticates to the Worker with a sandbox token, and the Worker swaps it for real credentials at the proxy boundary. This is the same trust model as a browser (untrusted code) talking to a backend (trusted execution), applied to AI agents.

Request Lifecycle

flowchart LR
    TG[Telegram] --> WH[Webhook Handler]
    CR[Cron] --> SC[Scheduled Handler]
    WH --> DEL[Delegation]
    SC --> DEL
    DEL --> OC[OpenClaw Container]
    OC --> LLM[LLM Proxy]
    LLM --> GW[AI Gateway]
    GW --> P[OpenAI / Anthropic / Google]
    OC --> MCP[MCP Server]
    MCP --> HITL{HITL Gate}
    HITL -->|allow| EXEC[Execute Tool]
    HITL -->|ask| TGA[Telegram Approval]
    TGA --> EXEC
    EXEC --> FILT[Credential Filter + PII Scrub]
    FILT --> OC
Loading

Tool Execution Pipeline

Every MCP tool call passes through a 16-phase security pipeline before execution and result delivery. The pipeline is fail-closed — unknown tools are denied, and any phase failure aborts the request.

flowchart TD
    REQ[JSON-RPC Request] --> REG[Tool Registry Lookup]
    REG --> INJA[Scan Args for Prompt Injection]
    INJA --> AUD[Audit Log: fail-closed]
    AUD --> RATE[Rate Limit]
    RATE --> LOOP[Loop Guard]
    LOOP --> POL[Policy Gate: deny / ask / allow]
    POL --> HITL{HITL Confirmation}
    HITL -->|denied| ABORT[Abort]
    HITL -->|approved| CRED[Validate Credential Scope]
    CRED --> BUD[Budget Check]
    BUD --> EXEC[Execute with Credential Injection]
    EXEC --> RCPT[Receipt Audit]
    RCPT --> SCAN[Scan Result for Injection]
    SCAN --> FILTER[Filter Credentials + Scrub PII]
    FILTER --> SUM[Result Summarization]
    SUM --> RESP[Output Cap + Trust-Level Delimiters]
Loading

Full 16 phases: (1) Parse JSON-RPC, (2) Tool registry lookup, (3) Scan args for prompt injection, (4) Audit log (fail-closed), (5) Rate limit, (6) Loop guard, (7) Policy gate (deny/ask/allow), (8) HITL confirmation, (9) Credential scope validation, (10) Budget check, (11) Execute with credential injection, (12) Receipt audit, (13) Scan result for injection, (14) Credential filter + PII scrub, (15) Result summarization, (16) Output cap + trust-level delimiters.

Defense in Depth

Most agent frameworks give the LLM direct access to credentials and tools. Moltworker enforces a strict separation — the container where the agent reasons has zero real secrets, and the Worker where tools execute has zero agent reasoning. This is the same pattern as a browser (untrusted code) talking to a backend (trusted execution).

Trust Boundaries

There are four concentric trust boundaries, each enforcing independent security controls:

Cloudflare Edge — Zero Trust Access validates JWTs on admin routes. AI Gateway enforces rate limiting, guardrails (prompt/response blocking), caching, and logging on all LLM traffic. DDoS protection is automatic.

Worker (Hono) — The inner trust boundary. Three auth layers (Telegram webhook secret, sandbox Bearer token, CF Access JWT). The 16-phase MCP pipeline validates, authorizes, executes, and sanitizes every tool call. All 14 Worker secrets live here — credentials never enter the container environment.

Container (Firecracker VM) — The agent sandbox. OpenClaw drives the agent loop with enableInternet: true but no real API keys. fs.workspaceOnly: true confines file operations. exec.security: "deny" blocks shell commands. Browser access is disabled — browser tools route through the audited Worker MCP pipeline. The container's "API keys" are actually sandbox auth tokens.

CF Access (Identity) — Channel identity linking ties Telegram users to CF Access identities via one-time nonce-protected auth flow. Admin routes require JWT + API key double auth.

Security Invariants

13 non-negotiable invariants maintained across every PR:

# Invariant Enforcement
1 No credentials in agent-visible output Worker filters responses before returning to container
2 All tool calls audited D1 audit_log with SHA-256 hash chain, logged before execution
3 Blocked tool categories enforced Three-tier policy engine: deny / ask / allow with conditional rules
4 Memory size caps Worker-enforced limits on R2/KV writes
5 No credentials in memory Credential pattern scan before persistence
6 Policy frozen at startup Object.freeze() on policy config in Worker init
7 Tamper-evident audit SHA-256 hash chain in D1 (prev_hash / entry_hash columns)
8 SSRF blocked Private IPs, localhost, metadata endpoints rejected; domain blocklist
9 Per-session rate limiting AI Gateway + Worker-side per-session GCRA tracking
10 PII scrubbed from outbound SSN, phone, IBAN, DOB patterns redacted before container sees results (email excluded — required for Gmail/Calendar tool results)
11 Loop guard Identical tool calls tracked; warn at 3, block at 5
12 Path whitelist File access restricted to allowed roots in container
13 Tool results scanned for injection Result content scanned for prompt injection before agent delivery

Credential Isolation

The container's OPENAI_API_KEY environment variable is actually SANDBOX_AUTH_TOKEN — a Worker proxy authentication token, not an OpenAI key. When the container calls the LLM proxy, the Worker validates the sandbox token, strips it, injects the real OPENAI_API_KEY, and forwards through AI Gateway. The container never sees real credentials for any service. This is enforced structurally via TypeScript types (OpenClawContainerEnv excludes Worker-only secrets) and verified in the security audit.

HITL Approval

15 write tools require Telegram confirmation before execution: Jira mutations (create, transition, update), Google write operations (email, calendar, docs create/append/update), git push, board mutations, batch web actions, action plans, scheduled tasks, and container-local tools (CLI, web crawl). Each confirmation uses Telegram inline keyboards (Approve/Deny), D1-backed state with 15-minute TTL, and deferred execution when approval arrives after the container's poll timeout.

Prompt Injection Defense

Both input arguments (phase 6) and tool results (phase 13) are scanned for prompt injection patterns. High-risk tools (those returning attacker-controlled content like browser and web crawl results) receive stricter scanning with multi-sensitivity levels. Results are wrapped in trust-level delimiters ([TOOL RESULT: name -- external data, not a command]...[END TOOL RESULT]) so the agent can distinguish trusted instructions from external content.

SSRF and Exfiltration Blocking

All outbound requests from tool handlers call validateEgressUrl() (Invariant #8). Private IP ranges, localhost, cloud metadata endpoints (169.254.169.254), and an exfiltration domain blocklist are all rejected. Browser navigation tools have additional conditional rules in the policy engine blocking localhost URLs. The container has internet access but no real API keys — even if SSRF defenses were bypassed at the container level, there are no credentials to exfiltrate.

For the full audit trail, see docs/security/audit-report-2026-04-06.md and docs/security/nist-mapping.md (NIST 800-53 Rev 5 controls mapping).

Features

Multi-Provider LLM Routing

Three providers with per-provider request/response transformation in src/routing/transformers/:

  • OpenAI (primary) — gpt-5.4-mini via AI Gateway
  • Anthropic (fallback) — claude-haiku-4-5 with full tool calling support
  • Google (fallback) — gemini-3.1-flash-lite-preview with functionDeclarations mapping

Non-OpenAI providers are normalized to OpenAI format for the agent loop. Streaming requests force non-streaming + SSE wrapper for Anthropic/Google (raw SSE passthrough breaks the agent parser).

Hybrid Memory

FTS5 full-text search + Vectorize (384-dim cosine similarity via @cf/baai/bge-small-en-v1.5) with reciprocal rank fusion. Automatic consolidation cron, TTL-based housekeeping, and a KV hot cache layer. Memory entries scanned for credential patterns before persistence (Invariant #5).

Interactive Kanban Board

Single-page inline HTML served from the Worker. D1 backend, dark theme, priority color coding, drag-drop reordering, mobile-responsive layout with bottom sheet actions. Cookie-based auth with three tiers: Bearer header, cookie, ?token= query param.

Proactive Agency

  • Scheduled tasks — once, daily, or weekly with HITL approval via Telegram
  • Proactive alerts — stale board tasks (3+ days), overdue scheduled tasks (2+ hours)
  • Quick-scan cron — daily proactive-check container spawn at 2 PM ET
  • KV mode toggleset_proactive_mode tool to pause/resume

Telegram Integration

Webhook-based with HITL inline keyboards, message chunking (4096-char Telegram limit with pre-chunk credential/PII filtering), typing indicators, access control via passphrase + CF Access identity linking, and deferred tool execution when approval arrives after poll timeout.

OTel Tracing

@microlabs/otel-cf-workers auto-instrumentation on the Worker + OpenClaw diagnostics.otel in the container. Custom spans for tool execution, LLM proxy, HITL flows, and container lifecycle. Axiom backend (OTLP/protobuf for container, OTLP/JSON for AI Gateway).

Skills System

Three-tier hierarchy: KV registry (metadata) -> R2 bundle (prompt + config) -> container runtime. Per-skill tool whitelisting controls which MCP tools are available per session type. Worker-managed skills at /root/skills/ (read-only) + OpenClaw workspace skills at /root/.openclaw/workspace/skills/ (agent-created, read-write).

Session Continuity

Durable Object-backed turn history (SessionDO), context enrichment injection (critical board tasks, scheduled tasks, memory principles), and memory flush on compaction. Session isolation prevents briefing sessions from contaminating ad-hoc Telegram conversations.

Additional Capabilities

  • Web Crawling — Crawl4AI 0.8.6 + Browserbase CDP in container, HITL-gated
  • Google Docs — Read, create, append, update via OAuth (separate from Google Read tools)
  • Browser Automation — 7 tools via Cloudflare Browser Rendering (navigate, snapshot, click, fill, select, screenshot, evaluate)
  • Git Push — GitHub Git Data API (blob -> tree -> commit -> ref update, non-force), HITL-gated
  • Action Plans — Multi-step HITL approval bundles (max 5 steps, stop-on-failure)
  • Context7 — Live library documentation via external MCP gateway
Full Tool Catalog (50 tools)
# Tool Service HITL Description
1 search_jira Jira No Search issues using JQL
2 get_jira_issue Jira No Get a single issue by key
3 create_jira_issue Jira Yes Create a new issue
4 transition_jira_issue Jira Yes Transition issue status
5 update_jira_issue Jira Yes Update issue fields
6 add_jira_comment Jira No Add comment to issue
7 rank_jira_issue Jira No Reorder issue on board
8 get_jira_board Jira No Get board configuration
9 list_granola_meetings Granola No List meetings for a date
10 get_granola_meeting Granola No Get meeting with transcript
11 query_granola_meetings Granola No Search meetings by query
12 brave_search Brave No Web search via Brave API
13 google_gmail_list Google No List Gmail messages
14 google_calendar_events Google No List calendar events
15 google_calendar_list Google No List calendars
16 google_drive_list Google No List Drive files
17 send_email Google Yes Send email via Gmail
18 create_calendar_event Google Yes Create calendar event
19 google_docs_read Google Docs No Read document content
20 google_docs_create Google Docs Yes Create new document
21 google_docs_append Google Docs Yes Append to document
22 google_docs_update Google Docs Yes Update document content
23 send_telegram_message Telegram No Send message to user
24 git_push Git Yes Push commits via GitHub API
25 batch_web_action Browser Yes Orchestrate search + browser actions
26 browser_navigate Browser No Navigate to URL
27 browser_snapshot Browser No Get page accessibility snapshot
28 browser_click Browser No Click element by ref
29 browser_fill Browser No Fill input field
30 browser_select Browser No Select dropdown option
31 browser_screenshot Browser No Capture page screenshot
32 browser_evaluate Browser No Execute JavaScript on page
33 mem_search Memory No Hybrid FTS + vector search
34 mem_observe Memory No Store memory observation
35 workspace_ls Workspace No List R2 workspace directory
36 check_confirmation Confirmation No Check HITL confirmation status
37 get_runtime_info Runtime No Get runtime configuration
38 get_current_datetime Runtime No Get current date and time
39 propose_action_plan Planning Yes Multi-step approval bundle
40 schedule_task Scheduling Yes Schedule a future task
41 cancel_scheduled_task Scheduling No Cancel scheduled task
42 list_scheduled_tasks Scheduling No List scheduled tasks
43 set_proactive_mode Scheduling No Pause/resume proactive agency
44 resolve_library_id Context7 No Resolve library to Context7 ID
45 query_library_docs Context7 No Query library documentation
46 list_board_tasks Board No List Kanban board tasks
47 create_board_task Board Yes Create board task
48 update_board_task Board Yes Update board task
49 move_board_task Board No Move task between columns
50 complete_board_task Board No Mark task complete

Plus 4 container-local tools (not in Worker MCP registry): container_shell, container_git, container_openclaw_cli, container_web_crawl.

Tech Stack

Service Purpose Binding
Workers Application runtime (Hono) --
Containers OpenClaw sandbox (Firecracker VM) CONTAINERS
D1 Audit log, memory, confirmations, board, sessions, scheduled tasks DB
R2 Sessions, workspace, skills, storage (4 buckets) R2, R2_SKILLS, R2_SESSIONS, R2_WORKSPACE
KV Config, skill metadata, feature flags, session cache KV
Vectorize Memory embeddings (384-dim cosine) VECTORIZE
AI Gateway LLM rate limiting, logging, guardrails, caching, multi-provider routing --
Zero Trust Access JWT-based admin route protection --
Browser Rendering Cloudflare Playwright for browser tools BROWSER
Durable Objects SessionDO (turn history), OpenClawContainer (lifecycle) SESSIONS_DO, CONTAINERS
Workers AI Embedding generation (@cf/baai/bge-small-en-v1.5) AI
Cron Triggers Daily briefing, memory consolidation, scheduled task dispatch --

Project Structure

src/
  worker.ts                     # Hono app entry point, routes, middleware chain
  types.ts                      # Env bindings, policy types, shared interfaces
  errors.ts                     # Typed error classes
  agent/                        # Agent execution loop (callModel, tool dispatch)
  agui/                         # AG-UI SSE event types
  board/                        # Kanban board HTML renderer + seed data parser
  container/                    # Container spawning, lifecycle, Durable Object
  handlers/                     # HTTP route handlers (~20 files)
    mcp-server.ts               #   16-phase MCP pipeline
    llm-proxy.ts                #   LLM proxy to AI Gateway
    telegram-webhook.ts         #   Telegram bot webhook + HITL auto-execute
    telegram-delegation.ts      #   Container delegation via Telegram
    scheduled.ts                #   Cron handlers (briefing, consolidation, dispatch)
    context-enrichment.ts       #   Pre-session context injection
    delegation-callback.ts      #   Container completion callback
    control-proxy.ts            #   Control UI gateway proxy
    board-api.ts                #   Kanban board REST API
    channel-link.ts             #   CF Access identity linking
  mcp-client/                   # External MCP server gateway (Streamable HTTP)
  memory/                       # Hybrid search, embeddings, indexing, hot cache
  middleware/                   # Auth, audit, policy freeze, security headers
  observability/                # OTel tracing (auto-instrumentation + custom spans)
  routing/                      # Provider config, model resolution
    transformers/               #   Per-provider request/response transformation
  schemas/                      # Zod validation schemas
  security/                     # All 13 invariant implementations (17 files)
    tool-policy.ts              #   Three-tier deny/ask/allow with conditional rules
    prompt-injection.ts         #   Input + result injection scanning
    credential-filter.ts        #   Credential pattern stripping
    credential-scope.ts         #   Domain-scoped credential binding
    network-policy.ts           #   Domain allowlist + exfiltration blocklist
    ssrf-guard.ts               #   Private IP + metadata endpoint blocking
    pii-filter.ts               #   SSN, phone, email redaction
    rate-limiter.ts             #   Per-session GCRA rate limiting
    loop-guard.ts               #   Duplicate tool call detection
    confirmation-gate.ts        #   HITL Telegram confirmation flow
  services/                     # OAuth, audit logger (hash chain), tool summarizer
  session/                      # Durable Object session management
  skills/                       # Skill loader (KV registry + R2 bundles)
  tools/                        # All 50 MCP tool implementations (23 files)
container/                      # Dockerfile, entrypoint.sh, local-tools.js, config
workspace/                      # Agent workspace files (SOUL.md, TOOLS.md, etc.)
test/                           # 2300+ tests across 117 files (unit + integration)
docs/                           # Specs, plans, security audit, teardown
migrations/                     # 14 D1 schema migrations
scripts/                        # Setup, deployment, backup scripts

Getting Started

These instructions are preserved for reference. The original Cloudflare deployment has been torn down. To deploy your own instance, you would need a Cloudflare Workers Paid plan and the listed API keys.

# 1. Clone and install
git clone https://github.com/scarnyc/moltworker-poc.git
cd moltworker-poc
pnpm install

# 2. Provision Cloudflare resources (idempotent)
./scripts/setup-cloudflare.sh
# Creates D1, R2 buckets, KV namespace, runs migrations,
# uploads skill files, updates wrangler.toml with real IDs

# 3. Set secrets (14 total)
wrangler secret put SANDBOX_AUTH_TOKEN
wrangler secret put OPENAI_API_KEY
wrangler secret put ANTHROPIC_API_KEY
wrangler secret put ATLASSIAN_API_TOKEN
wrangler secret put ATLASSIAN_EMAIL
wrangler secret put ATLASSIAN_DOMAIN
wrangler secret put GOOGLE_CLIENT_ID
wrangler secret put GOOGLE_CLIENT_SECRET
wrangler secret put GOOGLE_FULL_SCOPE_REFRESH_TOKEN
wrangler secret put GOOGLE_REFRESH_TOKEN_READONLY
wrangler secret put GRANOLA_API_KEY
wrangler secret put TELEGRAM_BOT_TOKEN
wrangler secret put BRAVE_SEARCH_KEY
wrangler secret put BROWSERBASE_API_KEY

# 4. Configure wrangler.toml [vars]
#    ENVIRONMENT       = "production"
#    TELEGRAM_CHAT_ID  = "<your-telegram-chat-id>"
#    AI_GATEWAY_URL    = "https://gateway.ai.cloudflare.com/v1/<your-cf-account-id>/<gateway-name>"
#    WORKER_URL        = "https://moltworker.<your-subdomain>.workers.dev"

# 5. Deploy
npx wrangler deploy

# 6. Set Telegram webhook
# curl https://api.telegram.org/bot<TOKEN>/setWebhook?url=https://moltworker.<subdomain>.workers.dev/api/telegram/webhook

Development

pnpm install          # Install dependencies
pnpm test             # Vitest (2300+ tests across 117 files)
pnpm typecheck        # tsc --noEmit
pnpm lint             # biome check .
pnpm format           # biome format --write .
pnpm dev              # wrangler dev (local)

Tests use @cloudflare/vitest-pool-workers with dangerouslyDisableSandbox: true (miniflare binds TCP ports). vi.mock does not work in this pool — all testable code uses dependency injection instead.

Documentation

Document Description
docs/state-machine-v1.md 16-phase request pipeline, all flows, security invariants (Mermaid diagrams)
docs/security/audit-report-2026-04-06.md Full 13-invariant security audit with findings and remediations
docs/security/nist-mapping.md NIST 800-53 Rev 5 controls mapping
docs/specs/ 20 design specifications (Telegram, OpenClaw, MCP, sessions, git, context engineering, proactive agency, Kanban, Crawl4AI, Control UI, and more)
docs/plans/ 8 implementation plans (foundation through proactive agency)
docs/teardown/ MOL-134 teardown audit trail and rotation log

Project Status

Permanently archived. 28 waves of development over 4 weeks (March 15 -- April 11, 2026). 74 PRs merged. 2300+ tests across 117 files.

What Was Destroyed (April 11, 2026)

  • 1 Worker + 3 cron triggers
  • 1 D1 database (7.2 MB, moltworker-db) with 14 migrations
  • 5 R2 buckets (1,889 objects total)
  • 1 KV namespace
  • 1 Vectorize index (moltworker-embeddings, 384-dim)
  • 111 container registry images (4 moltworker-codecontainer + 107 moltworker-openclawcontainer)
  • 1 CF Access application (/control/* + /auth/channel-link)
  • 1 AI Gateway (moltworker) + openai provider route
  • 1 R2 S3 API token
  • Cloudflare One subscription (downgraded)

Vendor Keys Rotated/Revoked

OpenAI, Anthropic, Google AI Studio, Atlassian (Jira), Granola, Brave Search, Browserbase, Google OAuth refresh tokens (OAuth client preserved -- shared across services), Telegram bot token.

Recovery

Git tag teardown/mol-134 at commit 5189af3 — last known-good deployed source. Full rotation audit trail at docs/teardown/MOL-134-rotation-log.md.

License

MIT -- see LICENSE.

Packages

 
 
 

Contributors