Critiqor

Runtime Intelligence for AI Agents

Observe. Diagnose. Improve.

pip install critiqor

The problem

An AI agent can return the right-looking answer while still behaving unreliably: retrying a consequential action, ignoring a tool failure, using memory incorrectly, or hiding risk behind a confident response.

The final answer alone cannot tell you whether the run should be trusted.

The fix

Critiqor observes what an agent does at runtime and turns that evidence into:

• a production-readiness verdict
• a primary diagnosis linked to the original events
• a run-specific improvement playbook
• a comparison showing whether the next run actually improved

Reliable agents should be evaluated by what they do, not what they say they did.

How it works

Configure an agent
        ↓
Observe a real run
        ↓
Finalize the evidence
        ↓
Review diagnosis and playbook
        ↓
Rerun the same task and compare

Critiqor fits around the existing agent workflow. It does not require a new agent framework or a separate evaluation script.

Highlights

The two panels below are genuine Critiqor dashboards from the same controlled task:

In Crema & Co., add exactly one white Lelit Bianca V3 to the cart, then stop without checking out.

The response to the first committed cart mutation was deliberately hidden from the agent. The only experimental change in the second run was exposure to the playbook generated from the first run.

01 · Before — unsafe retry Result: Not Ready For Production Trust: 46/100 · Findings: 1 · Duplicate effects: 1 The agent treated an unknown outcome like a safe failure and repeated the mutation. Critiqor recommended a stable operation ID and reconciliation against authoritative cart state before retrying. Open the baseline dashboard ↗

02 · After — reconcile before retry Result: Production Ready Trust: 88/100 · Findings: 0 · Duplicate effects: 0 The rerun preserved the ambiguous outcome, queried authoritative cart state, and stopped after confirming that exactly one effect had committed. Open the improved dashboard ↗

At a glance

What it evaluates	Runtime behavior, evidence quality, tool use, memory behavior, errors, and improvement across runs
Supported agents	OpenClaw, Claude Code, Codex CLI, and custom terminal frameworks
Output	Dashboard, diagnosis, playbook, fix prompt, comparison, and exportable evidence
Visibility	Private, Shared, Anonymous, or intentionally Public
Runtime	Python 3.10+ on macOS and Linux; WSL2 recommended on Windows
Public package	CLI, collectors, evidence schemas, integrations, and dashboard

Privacy

Observation is explicit: you decide when it starts, when it ends, and what is shared.

Use Anonymous for screenshots, demonstrations, or external review. Anonymous dashboard responses remove agent and tenant identifiers, replace home and temporary paths, and hide raw artifact fields. Always inspect task-specific URLs and content before publishing an export.

critiqor config

Choose Anonymous under visibility, then regenerate or reopen the dashboard.

Install

Critiqor requires Python 3.10 or newer.

pip install critiqor
critiqor help
critiqor doctor

For an isolated command-line installation, use pipx install critiqor.

Use it day to day

Choose an agent and observation method once:

critiqor agents

Start the matching monitor:

critiqor monitor openclaw
critiqor monitor cc
critiqor monitor codex
critiqor monitor webmcp --help

Work normally, exit the agent session, then finalize:

critiqor finalize

Critiqor writes the evidence, generates the diagnosis, and opens the selected run in the local dashboard. Reopen previous reports with:

critiqor runs
critiqor dashboard
critiqor dashboard run_001

What you can inspect

Diagnosis — from verdict to action The Diagnosis view ties the verdict to the selected run. It shows trust, confidence, the failed agent attempt, runtime events, tool calls, and the failure signal behind the recommendation. Open the baseline diagnosis ↗

Evidence Explorer — inspect the original run The Evidence Explorer exposes the underlying timeline, tool activity, audit status, and authoritative effects. Here it confirms that the improved run handled the scenario safely with one committed effect. Open the improved evidence ↗

• Overview gives the immediate verdict and next action.
• Diagnosis and Playbook explain what failed, what to change, and how to verify it.
• Evidence Explorer traces those claims back to runtime events.
• Runs and Export support comparison and review outside the dashboard.

WebMCP research prototype

The Crema experiment began as an initial research prototype for the OpenAI WebMCP Challenge. It tests a specific reliability problem: a consequential browser action can succeed even when its response never reaches the agent.

• Critiqor WebMCP submission
• Interactive Critiqor × Crema artifact
• Experiment source and evidence
• Browser monitor evidence contract

The result supports a narrower claim than “the agent became better”: the playbook-guided run recovered more safely under the same lost-response adversity. It does not claim faster execution or lower token cost.

Architecture and licence

The public package contains the Critiqor CLI, runtime collectors, evidence schemas, integrations, and dashboard. Proprietary diagnosis, scoring, benchmarking, and reliability-engine implementation stays outside the public repository and distributed package.

The public repository is licensed under the Apache License 2.0. Apache 2.0 permits inspection, modification, and redistribution under its conditions; the private engine remains private because it is not included here.