Skip to content

BPMN eval: smart-triage + high-volume-batch flake on intrinsic weight after the hunt is removed (#3228) #3233

Description

@nikhil-maryala

Context

smart-triage and high-volume-batch regressed to MAX_TURNS / turn_timeout in the eu nightly. The registry-hunt cause is fixed at the shared layer in #3228 (verified: the timed-out smart-triage run on that branch does 0 uip or / uip is connections / registry search commands, down from 16). But both tasks still flake over the 1800s turn budget with hunting fully removed — the agent deliberates through a large shape (extract/classify/two gateways/HITL/handlers/ends/diagram/package metadata for smart-triage; multi-instance + aggregate + escalation for high-volume-batch) and does not author in time on some runs.

Evidence: on #3228 (carve-out only), a smart-triage run timed out at 1803s with 0 hunt commands and 0 writes — 92 slow turns, never reached authoring. This is intrinsic task weight, not hunting.

Options to evaluate

  1. Decompose the eval — split the heavy authoring into smaller graded steps, or provide a partial fixture so the agent authors a delta rather than the whole shape from scratch.
  2. Right-size the budget — but test(maestro-bpmn): generous task/turn timeouts for 15 slow BPMN tasks #2940/test(uipath-maestro-bpmn): raise turn/max-turn budgets for 4 verified BPMN eval tasks #3150 already bumped these twice; a further raise chases eu per-turn latency (see [[coder-eval-task-timeout-config]]).
  3. Tighten the authoring recipe — a point-of-use "author the whole shape in one pass from this skeleton" prompt/guide nudge (the fix(uipath-maestro-bpmn): scope metadata regeneration away from IS draft handoffs #3152 lesson: point-of-use changes behavior more than a referenced file).

Acceptance

smart-triage and high-volume-batch reach SUCCESS reliably (e.g. ≥4/5) on claude/sonnet-5, with the mechanism documented (decomposition vs recipe vs budget).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions