Skip to content

[DRAFT] docs: PR #387 Kiro-vs-Claude evaluation + reconciliation plan (independent Linux/Py3.12 run)#16

Open
plauzy wants to merge 1 commit into
feat/agentic-protocols-generative-uifrom
docs/pr387-reconciliation-2
Open

[DRAFT] docs: PR #387 Kiro-vs-Claude evaluation + reconciliation plan (independent Linux/Py3.12 run)#16
plauzy wants to merge 1 commit into
feat/agentic-protocols-generative-uifrom
docs/pr387-reconciliation-2

Conversation

@plauzy

@plauzy plauzy commented Jul 6, 2026

Copy link
Copy Markdown
Owner

This pull request was created by @kiro-agent on behalf of @plauzy 👻

Comment with /kiro fix to address specific feedback or /kiro all to address everything.
Learn about Kiro Web


What this is

An independent, evidence-based head-to-head evaluation of the two remediations of upstream awslabs/cli-agent-orchestrator#387, plus a reconciliation plan. Adds one file: docs/reviews/pr387-reconciliation-plan.md. Plan only — no reconciliation code.

  • Kiro: PR-A #14 kiro/pr387-agui-core, PR-B #13 kiro/pr387-a2a-hardened
  • Claude: PR-A #11 claude/pr387-agui-core, PR-B #12 claude/pr387-a2a-hardened

Why a second branch

A concurrent run already pushed docs/pr387-reconciliation (a complete plan). This PR is a separate, independently-derived pass on a different toolchain, deliberately not clobbering that branch. The two can be diffed against each other. Base is feat/agentic-protocols-generative-ui; please mark as draft (the create step could not set the draft flag directly).

How it was verified (this run)

All four branches + base fetched as source; every gate executed on Linux, Python 3.12.13, uv sync --all-extras --dev, full pytest (including e2e):

Branch mypy pytest (full) Verdict
kiro/pr387-agui-core 132 files clean 3484 passed, 0 failed GREEN
claude/pr387-agui-core 132 files clean 3523 passed, 2 failed RED (telemetry [otel] subprocess test uses the legacy find_module hook, ignored on Py≥3.12)
kiro/pr387-a2a-hardened 142 files clean 3597 passed, 0 failed GREEN
claude/pr387-a2a-hardened 142 files clean 3592 passed; mount-guard test flaky AMBER

(Note: this differs from the macOS/-m 'not e2e' concurrent run, which attributed a failure to kiro-a2a and none to claude-a2a — the flakiness is environment/selection-sensitive, which the plan documents.)

Highlights of the verdict

  • Claude stronger on: AC1 default-off strength (module-absent + both endpoints 404 + enablement matrix), the real PWA ?since= cursor-loss fix (Kiro's api.ts is byte-identical to base), higher-fidelity live reconnect (real server SIGKILL→replay), real-JWT/JWKS auth tests + the reset_jwks_cache latent-break fix, and enforcement-before-parse.
  • Kiro stronger on: green on the specified toolchain, demo-artifact hygiene (no committed webm), skill-parity wiring (SHIPPED_SKILLS), endpoint-replay depth, the extracted testable _should_mount_a2a() guard, and the correct "no false [a2a] extra" dependency call.
  • Both missed: the single-use ticket handshake (I2), emit_ui rate limiting, STATE_DELTA debounce.

Full scorecard, "what both missed", and the per-contrast reconciliation decisions (two reconcile/* branches, gates, constraints, updated upstream reply, retarget sequencing) are in the file.

…ion plan

Evidence-based head-to-head of the Kiro (#13/#14) and Claude (#11/#12)
remediations of upstream awslabs#387: executed gate
results on all four branches (Python 3.12.13), a per-finding scorecard, a
'what both missed' list, and a step-by-step reconciliation plan for two
reconcile/* branches. Plan only — no reconciliation code.

Co-authored-by: Kiro Agent <244629292+kiro-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant