Skip to content

feat(hitl): PostgreSQL 持久化 pending store + structured_form - #3027

Open
labspc wants to merge 7 commits into
TencentCloud:mainfrom
labspc:feat/hitl-postgres-store
Open

labspc wants to merge 7 commits into
TencentCloud:mainfrom
labspc:feat/hitl-postgres-store

Conversation

@labspc

@labspc labspc commented Nov 18, 2026

Copy link
Copy Markdown

PR Note:feat(hitl) PostgreSQL 持久化 pending store + structured_form 生命周期

仓库:TencentCloud/Octop
分支:labspc:Octop feat/hitl-postgres-store → main
Base:octop v1.0.3b5(merge-base cd17e8d3 即 v1.0.3b5 本身,上游对涉及文件无额外提交,可干净 rebase 无冲突)
关联:octop-harness feat/interaction-lifecycle(request_structured_form 工具,协议 octop.structured_form.v1 两端对齐)

⚠️ 本分支内含三份 PR 描述文档,对应三个逻辑子集,建议合入顺序:先功能(表单协议 → 生命周期),后基础设施(PG store)。

What

  1. structured_form 网关侧协议(structured_form.py 新增 358 行):request_structured_form 的请求/响应双向校验、四类字段(integer/string/boolean/enum)的逐字段级错误定位
  2. 表单生命周期补全:timeout_seconds 超时、提交幂等(consumed_submission_ids)、主动取消(cancel_structured_form)
  3. HITL pending 记录 PostgreSQL 持久化:HitlPendingPostgresStore(pg_store.py 新增 604 行)+ 迁移 026(双方言 DDL)+ 运行时工厂 + 冒烟脚本

变更总量:16 个文件,+2,358 行,-5 行(相对于 merge-base cd17e8d3)

Why

  • HITL pending 记录原先只存内存:网关重启即丢失所有『等待用户审批/响应』状态;多实例部署下各实例状态互不可见
  • structured_form 协议缺少生命周期:无超时(表单挂起永久占坑)、无幂等(重复提交无法去重)、无主动取消入口
  • 顺带修复一个安全漏洞:原 if not allowed: continue 逻辑会放行未声明的中断类型,改为 if allowed and kind not in allowed 正确拦截

How

PG store 与工厂

  • pg_store.py:HitlPendingPostgresStore,19 个方法与内存版 HitlPendingStore 签名完全一致(直接替换实现);psycopg v3 + dict_row + ConnectionPool(min=1, max=4)
  • 迁移 026 双方言 DDL(.sql SQLite / .pg.sql PG):hitl_pending_records 表 18 列 + 9 索引(含 (form_context->>'form_id') 部分索引);沿用 021–025 双方言约定,migrate.py sorted(iterdir()) 自动发现,无需改注册表
  • factory.py build_pending_store():PG 控制面且 026 已应用 → PG store;探测方式 psycopg.connect(conninfo, connect_timeout=5) 执行 SELECT 1 FROM hitl_pending_records LIMIT 1
  • 失败降级而非抛错:无 config.json 且无 OCTOP_DATABASE_* → 内存版;非 PG 控制面 → 内存版;PG 不可达/缺表 → warning + 回落内存版(HITL 记录是会话级状态,启动失败的代价远大于用内存版)
  • __init__.py PEP 562 __getattr__ 惰性导出 PG store:pg_store.py 模块级导入 psycopg/psycopg_pool,惰性解析避免无 PG 驱动的纯 SQLite 安装导入失败

表单生命周期

  • HitlPendingRecord 新增 form_context(form_id/schema_version/timeout_seconds/context)与 consumed_submission_ids: set[str]
  • coordinator.register_from_request:注册时将 form_context 传入 store.register() 同一 INSERT 落库(修复『只给返回对象赋值、持久化不落库』,否则 cancel_pending_by_form_id 无法匹配)
  • decision_rejection_reason 新增校验链:请求校验 → timeout 过期 → 已消费表单禁 cancel → respond 响应校验 → 重复 submission_id 幂等拒绝
  • 新方法:mark_form_consumed(幂等,重复返回 False,仅 pending 态有效)、cancel_pending_by_form_id(仅过期未消费表单,PG 版按 form_id+agent_id(+user_id) 批量置 expired)
  • append_ask_answer 使用 FOR UPDATE 行锁

网关侧协议校验(structured_form.py)

  • 常量:STRUCTURED_FORM_PROTOCOL = "octop.structured_form.v1"、SCHEMA_VERSION = 1、MAX_STRUCTURED_FORM_BYTES = 32768、MAX_STRUCTURED_FORM_FIELDS = 64
  • validate_structured_form_request:MVP 仅 integer 字段;timeout ∈ [30,3600];required 不可同时 nullable;step 必须为 1;context ≤16 键
  • validate_structured_form_response:校验 respond JSON 的 protocol/version/form_id/submission_id/values,逐字段校验 required/nullable/int 类型/min-max 范围

Test

测试平台:本地 + PostgreSQL 实库(连接串经环境变量 OCTOP_TEST_DSN 传入;依赖 psycopg[binary]>=3.2、psycopg_pool>=3.2)

测试项 结果
PG 冒烟 scripts/smoke_hitl_pg_store.py(13 项,需先应用迁移 026) 13/13 通过,连续两次
工厂三场景实测(无配置→内存 / SQLite→内存 / PG 可达→PG store) 通过
回归 tests/unit/gateway 599 passed
HITL 专项 84 passed
迁移测试 46 passed
structured_form 协议单测 tests/unit/gateway/test_structured_form.py 63 passed(四类字段校验 + 字段级错误定位)
ruff 全过(3 个模块因缺 PIL/fastapi 无法收集,与本改动无关)
新增 test_hitl_form_context_registration.py(4 测试:form_context 注册落库 / 普通审批传 None / memory store 兼容 / slot 复用刷新) 通过

Limitations

  • 未含端到端集成测试(需完整 Octop 运行时 + LLM 后端),建议后续补
  • PG 版 _gc() 无定时任务:过期记录按 created_at 在读写路径触发清理;多实例下各实例各自触发,非集中调度

Dependencies

  • 上游依赖 PR3 feat/db-migrate-to-pg:PG store 需要 PG 控制面先行就位;迁移 026 双方言 DDL 依赖 migrate.py 的自动发现机制(sorted(iterdir())),该机制随 PR3 的 db.py 命令组一同引入

Related

  • octop-harness feat/interaction-lifecycle:Agent 侧 request_structured_form 工具(本 PR 提供其网关侧校验与持久化)
  • Octop feat/db-migrate-to-pg:SQLite → PG 迁移工具(同属控制面 PG 化方向)

labspc and others added 7 commits November 18, 2026 15:10
- Add gateway-side structured_form.py with request/response validation
- Tighten decision_rejection_reason: skip only when allowed decisions
  are explicitly configured and the kind is not in the allowlist
- Add structured form validation branch in decision_rejection_reason

Based on upstream v1.0.3b5
feat(hitl): add structured form request/response validation
…ncy and cancel

Timeout: timeout_seconds (30-3600) validated and enforced on both respond and cancel.

Idempotency: submission ids recorded in consumed_submission_ids and rejected on replay.

Cancellation: cancel_structured_form expires every pending pause matching a form_id.
Migration 026 creates hitl_pending_records in both SQLite and PostgreSQL dialects.

HitlPendingPostgresStore mirrors the in-memory API so the coordinator needs no branching.

build_pending_store() picks the implementation from the control-plane config and degrades to the in-memory store on any failure, so a half-configured host still boots.

pg_store is exported lazily from the hitl package: SQLite installs do not ship psycopg and must keep importing the package successfully.
…rsist it

Before this fix, coordinator.register_from_request() called
store.register() without the form_context keyword, then assigned
record.form_context afterwards.  The in-memory store appeared to work
because it returns the same object reference, but the PostgreSQL store
had already INSERTed the row — the assignment never reached the database.

Consequence: cancel_pending_by_form_id (which queries
form_context->>form_id) never matched, and timeout/idempotency checks
were equally broken on PostgreSQL deployments.

Fix:
- store.py: accept resume_thread_id and form_context in register(),
  write them into the HitlPendingRecord at construction time, and
  refresh form_context when a re-register hits the same slot.
- coordinator.py: parse the structured-form action request before
  calling register() and hand form_context in the call, so both the
  in-memory and PostgreSQL stores receive it in the same INSERT.

Test: 4 new regression tests in
  tests/unit/gateway/test_hitl_form_context_registration.py
All 566 existing HITL-related tests continue to pass.
…elds

Extend the gateway-side structured_form validator beyond integer-only fields so it matches the harness four-field-type schema:

- Add string field validation: min_length/max_length (≤4096), pattern (pre-compiled with re.error guard), multiline toggle

- Add boolean field validation: rejects nullable=true because null carries no boolean meaning

- Add enum field validation: non-empty unique option list (≤64 entries, ≤120 chars each), default must be an option

- Add default value validation for every field type (default outside range/enum/pattern is rejected at request time)

- Scope every field-level error with the field key so a renderer can highlight the offending control instead of rejecting the whole form

- Align required=False / nullable=True defaults with the harness side (previously required=True / nullable=False was the gateway default)

- Add 63 gateway-side unit tests covering all four field types, constraints, defaults, response validation and edge cases
Copilot AI balanced review requested due to automatic review settings November 18, 2026 07:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

10 open issues
What changed in this PR

This PR adds a structured HITL form protocol (request/response validation + lifecycle controls) and introduces an optional PostgreSQL-backed durable pending store selected at runtime, enabling pending HITL records to survive restarts and support multi-instance deployments.

Changes:

  • Add structured_form protocol validators + comprehensive unit tests for four field types and timeout handling.
  • Extend HITL lifecycle: timeout expiration, submission idempotency tracking, and form cancellation by form_id.
  • Add PostgreSQL pending store (migration 026 + runtime factory + verification/smoke scripts) and wire it into the gateway processor.
File Description
tests/​unit/​gateway/​test_structured_form.py Adds unit tests covering structured_form request/response validation rules.
tests/​unit/​gateway/​test_hitl_form_context_registration.py Ensures structured form context is passed into store.register for durability.
src/​octop/​infra/​gateway/​process/​processor.py Uses runtime factory to pick pending store implementation when building the coordinator.
src/​octop/​infra/​gateway/​hitl/​structured_form.py Implements structured_form protocol validators and size/shape constraints.
src/​octop/​infra/​gateway/​hitl/​store.py Extends pending record with form_context + consumed_submission_ids and adds lifecycle store methods.
src/​octop/​infra/​gateway/​hitl/​pg_store.py Adds PostgreSQL-backed pending store implementation.
src/​octop/​infra/​gateway/​hitl/​factory.py Adds runtime selection logic with graceful fallback to in-memory store.
src/​octop/​infra/​gateway/​hitl/​coordinator.py Adds structured_form lifecycle enforcement and cancellation entrypoint.
src/​octop/​infra/​gateway/​hitl/​__init__.py Lazily exports the PG store to avoid optional dependency import failures.
src/​octop/​infra/​db/​migrations/​026_hitl_pending_store.sql Adds SQLite dialect migration 026 for hitl_pending_records.
src/​octop/​infra/​db/​migrations/​026_hitl_pending_store.pg.sql Adds PostgreSQL dialect migration 026 with JSONB columns and indexes.
scripts/​verify_hitl_pg_schema.py Adds helper script to apply/verify migration 026 schema objects.
scripts/​smoke_hitl_pg_store.py Adds a PG smoke test script for the new store behaviors.
PULL_REQUEST_PG_STORE.md Documentation describing the PG store design and validation steps.
PULL_REQUEST_BODY.md Documentation describing the structured_form protocol motivation and validation rules.
PULL_REQUEST.md Documentation describing structured_form lifecycle changes (timeout/idempotency/cancel).

Review effort: Lite

Comment on lines 254 to 262
allowed_by_action[name] = {str(item) for item in allowed}
for action, decision in zip(actions, decisions, strict=True):
allowed = allowed_by_action.get(str(action.get("name")))
if not allowed:
continue
kind = str(decision.get("type"))
if kind not in allowed:
if allowed and kind not in allowed:
return (
f"decision '{kind}' is not allowed for tool "
f"'{action.get('name')}' (allowed: {', '.join(sorted(allowed))})"
)
Comment on lines +294 to +295
*,
pending_id: str | None = None,
Comment on lines +518 to +527
def mark_form_consumed(self, pending_id: str, submission_id: str) -> bool:
now = self._now()
with self._pool.connection() as conn:
conn.execute("BEGIN")
try:
r = conn.execute(
"SELECT consumed_submission_ids FROM hitl_pending_records"
" WHERE pending_id = %s FOR UPDATE",
(pending_id,),
).fetchone()

sql = SQL_PATH.read_text(encoding="utf-8")
with psycopg.connect(dsn, autocommit=True) as conn:
conn.execute("CREATE TABLE IF NOT EXISTS _schema_version (version INTEGER)")
Comment on lines +6 to +12

from octop.infra.gateway.hitl.store import HitlPendingStore

_log = logging.getLogger(__name__)


def build_pending_store() -> HitlPendingStore:
Comment on lines +49 to +53
except Exception:
_log.warning(
"hitl pending store: postgres unavailable, using in-memory store",
exc_info=True,
)
(session_key, thread_id or "", resume_agent_id or ""),
)

pending_id = secrets.token_hex(2)
Comment on lines +224 to +235
conn.execute(
"""
INSERT INTO hitl_pending_records
(pending_id, thread_id, agent_id, user_id, session_key,
channel_type, action_requests, review_configs, status,
created_at, updated_at, resume_agent_id, resume_thread_id,
form_context, consumed_submission_ids, ask_question_index,
ask_answers)
VALUES
(%s, %s, %s, %s, %s, %s, %s, %s, 'pending',
%s, %s, %s, %s, %s, %s, 0, %s)
""",


def _json_size(value: Any) -> int:
return len(json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8"))
if request_error:
return request_error
if not isinstance(message, str):
return "structured form response must be a JSON object"

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants