Repository navigation
Conversation
- Add gateway-side structured_form.py with request/response validation - Tighten decision_rejection_reason: skip only when allowed decisions are explicitly configured and the kind is not in the allowlist - Add structured form validation branch in decision_rejection_reason Based on upstream v1.0.3b5
feat(hitl): add structured form request/response validation
…ncy and cancel Timeout: timeout_seconds (30-3600) validated and enforced on both respond and cancel. Idempotency: submission ids recorded in consumed_submission_ids and rejected on replay. Cancellation: cancel_structured_form expires every pending pause matching a form_id.
Migration 026 creates hitl_pending_records in both SQLite and PostgreSQL dialects. HitlPendingPostgresStore mirrors the in-memory API so the coordinator needs no branching. build_pending_store() picks the implementation from the control-plane config and degrades to the in-memory store on any failure, so a half-configured host still boots. pg_store is exported lazily from the hitl package: SQLite installs do not ship psycopg and must keep importing the package successfully.
…rsist it Before this fix, coordinator.register_from_request() called store.register() without the form_context keyword, then assigned record.form_context afterwards. The in-memory store appeared to work because it returns the same object reference, but the PostgreSQL store had already INSERTed the row — the assignment never reached the database. Consequence: cancel_pending_by_form_id (which queries form_context->>form_id) never matched, and timeout/idempotency checks were equally broken on PostgreSQL deployments. Fix: - store.py: accept resume_thread_id and form_context in register(), write them into the HitlPendingRecord at construction time, and refresh form_context when a re-register hits the same slot. - coordinator.py: parse the structured-form action request before calling register() and hand form_context in the call, so both the in-memory and PostgreSQL stores receive it in the same INSERT. Test: 4 new regression tests in tests/unit/gateway/test_hitl_form_context_registration.py All 566 existing HITL-related tests continue to pass.
…elds Extend the gateway-side structured_form validator beyond integer-only fields so it matches the harness four-field-type schema: - Add string field validation: min_length/max_length (≤4096), pattern (pre-compiled with re.error guard), multiline toggle - Add boolean field validation: rejects nullable=true because null carries no boolean meaning - Add enum field validation: non-empty unique option list (≤64 entries, ≤120 chars each), default must be an option - Add default value validation for every field type (default outside range/enum/pattern is rejected at request time) - Scope every field-level error with the field key so a renderer can highlight the offending control instead of rejecting the whole form - Align required=False / nullable=True defaults with the harness side (previously required=True / nullable=False was the gateway default) - Add 63 gateway-side unit tests covering all four field types, constraints, defaults, response validation and edge cases
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
10 open issues
This check still treats an empty allowed-decision set as \"no restriction\" becauseallowedis… · NewHitlPendingPostgresStore.resolve_for_sessionis not signature-compatible with… · Newmark_form_consumeddoes not enforcestatus = 'pending', so it can record consumption for… · New The docstring says the script is safe to re-run, but it inserts a new_schema_versionrow every… · Newbuild_pending_store()is annotated to returnHitlPendingStore, but it can return… · New Logging withexc_info=Trueon DB connection failures can inadvertently leak sensitive connection… · New The PG store generates a 4-hex-characterpending_idand relies on a UNIQUE constraint, but it… · New The PG store generates a 4-hex-characterpending_idand relies on a UNIQUE constraint, but it… · New_json_size()can raiseTypeErrorifargscontains non-JSON-serializable objects (the public… · New The error message says the response must be a \"JSON object\", but the validator actually requires… · New
What changed in this PR
This PR adds a structured HITL form protocol (request/response validation + lifecycle controls) and introduces an optional PostgreSQL-backed durable pending store selected at runtime, enabling pending HITL records to survive restarts and support multi-instance deployments.
Changes:
- Add structured_form protocol validators + comprehensive unit tests for four field types and timeout handling.
- Extend HITL lifecycle: timeout expiration, submission idempotency tracking, and form cancellation by form_id.
- Add PostgreSQL pending store (migration 026 + runtime factory + verification/smoke scripts) and wire it into the gateway processor.
| File | Description |
|---|---|
| tests/unit/gateway/test_structured_form.py | Adds unit tests covering structured_form request/response validation rules. |
| tests/unit/gateway/test_hitl_form_context_registration.py | Ensures structured form context is passed into store.register for durability. |
| src/octop/infra/gateway/process/processor.py | Uses runtime factory to pick pending store implementation when building the coordinator. |
| src/octop/infra/gateway/hitl/structured_form.py | Implements structured_form protocol validators and size/shape constraints. |
| src/octop/infra/gateway/hitl/store.py | Extends pending record with form_context + consumed_submission_ids and adds lifecycle store methods. |
| src/octop/infra/gateway/hitl/pg_store.py | Adds PostgreSQL-backed pending store implementation. |
| src/octop/infra/gateway/hitl/factory.py | Adds runtime selection logic with graceful fallback to in-memory store. |
| src/octop/infra/gateway/hitl/coordinator.py | Adds structured_form lifecycle enforcement and cancellation entrypoint. |
| src/octop/infra/gateway/hitl/__init__.py | Lazily exports the PG store to avoid optional dependency import failures. |
| src/octop/infra/db/migrations/026_hitl_pending_store.sql | Adds SQLite dialect migration 026 for hitl_pending_records. |
| src/octop/infra/db/migrations/026_hitl_pending_store.pg.sql | Adds PostgreSQL dialect migration 026 with JSONB columns and indexes. |
| scripts/verify_hitl_pg_schema.py | Adds helper script to apply/verify migration 026 schema objects. |
| scripts/smoke_hitl_pg_store.py | Adds a PG smoke test script for the new store behaviors. |
| PULL_REQUEST_PG_STORE.md | Documentation describing the PG store design and validation steps. |
| PULL_REQUEST_BODY.md | Documentation describing the structured_form protocol motivation and validation rules. |
| PULL_REQUEST.md | Documentation describing structured_form lifecycle changes (timeout/idempotency/cancel). |
Review effort: Lite
Comment on lines
254
to
262
| allowed_by_action[name] = {str(item) for item in allowed} | ||
| for action, decision in zip(actions, decisions, strict=True): | ||
| allowed = allowed_by_action.get(str(action.get("name"))) | ||
| if not allowed: | ||
| continue | ||
| kind = str(decision.get("type")) | ||
| if kind not in allowed: | ||
| if allowed and kind not in allowed: | ||
| return ( | ||
| f"decision '{kind}' is not allowed for tool " | ||
| f"'{action.get('name')}' (allowed: {', '.join(sorted(allowed))})" | ||
| ) |
Comment on lines
+294
to
+295
| *, | ||
| pending_id: str | None = None, |
Comment on lines
+518
to
+527
| def mark_form_consumed(self, pending_id: str, submission_id: str) -> bool: | ||
| now = self._now() | ||
| with self._pool.connection() as conn: | ||
| conn.execute("BEGIN") | ||
| try: | ||
| r = conn.execute( | ||
| "SELECT consumed_submission_ids FROM hitl_pending_records" | ||
| " WHERE pending_id = %s FOR UPDATE", | ||
| (pending_id,), | ||
| ).fetchone() |
|
|
||
| sql = SQL_PATH.read_text(encoding="utf-8") | ||
| with psycopg.connect(dsn, autocommit=True) as conn: | ||
| conn.execute("CREATE TABLE IF NOT EXISTS _schema_version (version INTEGER)") |
Comment on lines
+6
to
+12
|
|
||
| from octop.infra.gateway.hitl.store import HitlPendingStore | ||
|
|
||
| _log = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def build_pending_store() -> HitlPendingStore: |
Comment on lines
+49
to
+53
| except Exception: | ||
| _log.warning( | ||
| "hitl pending store: postgres unavailable, using in-memory store", | ||
| exc_info=True, | ||
| ) |
| (session_key, thread_id or "", resume_agent_id or ""), | ||
| ) | ||
|
|
||
| pending_id = secrets.token_hex(2) |
Comment on lines
+224
to
+235
| conn.execute( | ||
| """ | ||
| INSERT INTO hitl_pending_records | ||
| (pending_id, thread_id, agent_id, user_id, session_key, | ||
| channel_type, action_requests, review_configs, status, | ||
| created_at, updated_at, resume_agent_id, resume_thread_id, | ||
| form_context, consumed_submission_ids, ask_question_index, | ||
| ask_answers) | ||
| VALUES | ||
| (%s, %s, %s, %s, %s, %s, %s, %s, 'pending', | ||
| %s, %s, %s, %s, %s, %s, 0, %s) | ||
| """, |
|
|
||
|
|
||
| def _json_size(value: Any) -> int: | ||
| return len(json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8")) |
| if request_error: | ||
| return request_error | ||
| if not isinstance(message, str): | ||
| return "structured form response must be a JSON object" |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



PR Note:feat(hitl) PostgreSQL 持久化 pending store + structured_form 生命周期
What
structured_form.py新增 358 行):request_structured_form的请求/响应双向校验、四类字段(integer/string/boolean/enum)的逐字段级错误定位timeout_seconds超时、提交幂等(consumed_submission_ids)、主动取消(cancel_structured_form)HitlPendingPostgresStore(pg_store.py 新增 604 行)+ 迁移 026(双方言 DDL)+ 运行时工厂 + 冒烟脚本Why
if not allowed: continue逻辑会放行未声明的中断类型,改为if allowed and kind not in allowed正确拦截How
PG store 与工厂
pg_store.py:HitlPendingPostgresStore,19 个方法与内存版HitlPendingStore签名完全一致(直接替换实现);psycopg v3 +dict_row+ConnectionPool(min=1, max=4).sqlSQLite /.pg.sqlPG):hitl_pending_records表 18 列 + 9 索引(含(form_context->>'form_id')部分索引);沿用 021–025 双方言约定,migrate.pysorted(iterdir())自动发现,无需改注册表factory.pybuild_pending_store():PG 控制面且 026 已应用 → PG store;探测方式psycopg.connect(conninfo, connect_timeout=5)执行SELECT 1 FROM hitl_pending_records LIMIT 1OCTOP_DATABASE_*→ 内存版;非 PG 控制面 → 内存版;PG 不可达/缺表 → warning + 回落内存版(HITL 记录是会话级状态,启动失败的代价远大于用内存版)__init__.pyPEP 562__getattr__惰性导出 PG store:pg_store.py模块级导入 psycopg/psycopg_pool,惰性解析避免无 PG 驱动的纯 SQLite 安装导入失败表单生命周期
HitlPendingRecord新增form_context(form_id/schema_version/timeout_seconds/context)与consumed_submission_ids: set[str]coordinator.register_from_request:注册时将form_context传入store.register()同一 INSERT 落库(修复『只给返回对象赋值、持久化不落库』,否则cancel_pending_by_form_id无法匹配)decision_rejection_reason新增校验链:请求校验 → timeout 过期 → 已消费表单禁 cancel → respond 响应校验 → 重复 submission_id 幂等拒绝mark_form_consumed(幂等,重复返回 False,仅 pending 态有效)、cancel_pending_by_form_id(仅过期未消费表单,PG 版按 form_id+agent_id(+user_id) 批量置 expired)append_ask_answer使用FOR UPDATE行锁网关侧协议校验(structured_form.py)
STRUCTURED_FORM_PROTOCOL = "octop.structured_form.v1"、SCHEMA_VERSION = 1、MAX_STRUCTURED_FORM_BYTES = 32768、MAX_STRUCTURED_FORM_FIELDS = 64validate_structured_form_request:MVP 仅 integer 字段;timeout ∈ [30,3600];required 不可同时 nullable;step 必须为 1;context ≤16 键validate_structured_form_response:校验 respond JSON 的 protocol/version/form_id/submission_id/values,逐字段校验 required/nullable/int 类型/min-max 范围Test
测试平台:本地 + PostgreSQL 实库(连接串经环境变量
OCTOP_TEST_DSN传入;依赖psycopg[binary]>=3.2、psycopg_pool>=3.2)scripts/smoke_hitl_pg_store.py(13 项,需先应用迁移 026)tests/unit/gatewaytests/unit/gateway/test_structured_form.pytest_hitl_form_context_registration.py(4 测试:form_context 注册落库 / 普通审批传 None / memory store 兼容 / slot 复用刷新)Limitations
_gc()无定时任务:过期记录按created_at在读写路径触发清理;多实例下各实例各自触发,非集中调度Dependencies
feat/db-migrate-to-pg:PG store 需要 PG 控制面先行就位;迁移 026 双方言 DDL 依赖migrate.py的自动发现机制(sorted(iterdir())),该机制随 PR3 的db.py命令组一同引入Related
feat/interaction-lifecycle:Agent 侧request_structured_form工具(本 PR 提供其网关侧校验与持久化)feat/db-migrate-to-pg:SQLite → PG 迁移工具(同属控制面 PG 化方向)