Reference

Agents

client.agents.verify_connection(...) -> VerifyConnectionResponse

📝 Description

Verify an agent's connection and persist the result when successful

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.verify_connection(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent whose connection to verify

model: typing.Optional[str] — Model to verify. Omit for a basic connection check. Provide it for a model-specific check before benchmarking that model

messages: typing.Optional[typing.List[typing.Dict[str, str]]] — Sample chat messages to send during verification. Omit to use the default probe

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.resolve(...) -> ResolveAgentNamesResponse

📝 Description

Get the IDs for your agents by their names

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.resolve(
    names=[
        "my-agent",
        "support-bot"
    ],
)

⚙️ Parameters

names: typing.List[str] — Agent names to resolve to IDs

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.list(...) -> PaginatedResponseAgentSummary

📝 Description

Get the list of all your agents

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.create(...) -> AgentCreateResponse

📝 Description

Create an agent to test inside Calibrate or connect your existing agent to Calibrate

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.create(
    name="Support Agent",
    type="agent",
    config={
        "system_prompt": "You are a helpful assistant.",
        "llm": {"model": "google/gemini-2.5-flash"},
        "stt": {"provider": "google"},
        "tts": {"provider": "google"},
        "settings": {"agent_speaks_first": True, "max_assistant_turns": 50}
    },
)

⚙️ Parameters

name: str — Agent name, unique within the workspace

type: typing.Optional[AgentCreateType]

agent: built inside Calibrate
connection: your existing agent connected to Calibrate

config: typing.Optional[typing.Dict[str, typing.Any]]

Agent behavioral config. The keys depend on type.

type=agent, built inside Calibrate:

system_prompt: the agent's instructions
llm.model: provider/model, e.g. openai/gpt-4.1 or google/gemini-2.5-flash
stt.provider: deepgram, openai, cartesia, elevenlabs, google, sarvam, or smallest
tts.provider: cartesia, openai, google, elevenlabs, sarvam, or smallest
settings.agent_speaks_first, settings.max_assistant_turns
system_tools.end_call: let the agent end the call
data_extraction_fields: [{name, type, description, required}]

{
  "system_prompt": "You are a helpful support agent.",
  "llm": {"model": "openai/gpt-4.1"},
  "stt": {"provider": "deepgram"},
  "tts": {"provider": "elevenlabs"},
  "settings": {"agent_speaks_first": true, "max_assistant_turns": 50}
}

type=connection, your own HTTP endpoint:

agent_url: public HTTP(S) endpoint your agent is called at
agent_headers: headers sent on each request, e.g. auth
benchmark_provider: openrouter by default. Other values: openai, google, anthropic, meta-llama, mistralai, deepseek, x-ai, cohere, qwen, or ai21

{
  "agent_url": "https://api.example.com/agent",
  "agent_headers": {"Authorization": "Bearer <token>"},
  "benchmark_provider": "openrouter"
}

For type=agent, omitted keys inherit managed defaults. Omit config entirely to use all defaults. For type=connection, config is stored as-is and must contain agent_url

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.get(...) -> RoutersAgentsAgentResponse

📝 Description

Get one agent by its ID

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.get(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.update(...) -> RoutersAgentsAgentResponse

📝 Description

Update an agent's configuration

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.update(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent to update

name: typing.Optional[str] — New agent name. Omit to leave the name unchanged

config: typing.Optional[typing.Dict[str, typing.Any]]

Agent behavioral config. The keys depend on type.

type=agent, built inside Calibrate:

system_prompt: the agent's instructions
llm.model: provider/model, e.g. openai/gpt-4.1 or google/gemini-2.5-flash
stt.provider: deepgram, openai, cartesia, elevenlabs, google, sarvam, or smallest
tts.provider: cartesia, openai, google, elevenlabs, sarvam, or smallest
settings.agent_speaks_first, settings.max_assistant_turns
system_tools.end_call: let the agent end the call
data_extraction_fields: [{name, type, description, required}]

{
  "system_prompt": "You are a helpful support agent.",
  "llm": {"model": "openai/gpt-4.1"},
  "stt": {"provider": "deepgram"},
  "tts": {"provider": "elevenlabs"},
  "settings": {"agent_speaks_first": true, "max_assistant_turns": 50}
}

type=connection, your own HTTP endpoint:

agent_url: public HTTP(S) endpoint your agent is called at
agent_headers: headers sent on each request, e.g. auth
benchmark_provider: openrouter by default. Other values: openai, google, anthropic, meta-llama, mistralai, deepseek, x-ai, cohere, qwen, or ai21

{
  "agent_url": "https://api.example.com/agent",
  "agent_headers": {"Authorization": "Bearer <token>"},
  "benchmark_provider": "openrouter"
}

Replaces the stored config. Omit to leave unchanged

For type=connection, changing agent_url or agent_headers resets the connection and benchmark verification flags

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.list_evaluators(...) -> PaginatedResponseEvaluatorResponse

📝 Description

List evaluators linked to an agent

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.list_evaluators(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — The agent whose evaluators to list

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agents.link_evaluators(...) -> RoutersAgentsEvaluatorLinkResponse

📝 Description

Link one or more existing evaluators to an agent, skipping any already linked

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agents.link_evaluators(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    evaluator_ids=[
        "f47ac10b-58cc-4372-a567-0e02b2c3d479"
    ],
)

⚙️ Parameters

agent_uuid: str — The agent to link the evaluators to

evaluator_ids: typing.List[str] — The evaluators to link to the agent. Ones that are already linked are skipped. Each must be one you created or a built-in default

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

Tests

client.tests.bulk_create(...) -> BulkTestUploadResponse

📝 Description

Create many test cases at once and link them to your agents

🔌 Usage

from calibrate import Calibrate, BulkTestItem, ChatMessage
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.bulk_create(
    type="response",
    tests=[
        BulkTestItem(
            name="name",
            conversation_history=[
                ChatMessage(
                    role="user",
                )
            ],
        )
    ],
)

⚙️ Parameters

type: BulkTestUploadType

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

Applied to every test in the batch

tests: typing.List[BulkTestItem] — Test items to create, at most 500 per request, with names unique within the batch

agent_uuids: typing.Optional[typing.List[str]] — IDs of agents to link every created test to. Omit to link none

language: typing.Optional[str] — Language written to each test's config.settings.language. Omit to leave unset

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.list(...) -> PaginatedResponseTestListResponse

📝 Description

List all the test cases for your agents

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.create(...) -> TestCreateResponse

📝 Description

Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.create(
    name="name",
    type="response",
)

⚙️ Parameters

name: str — Name of the test, unique within the workspace

type: TestCreateType

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

config: typing.Optional[typing.Dict[str, typing.Any]]

The calibrate test config. Three top-level keys.

history: the required conversation up to the agent's turn. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
evaluation: the required {type, ...}, where type matches the test's type below.
settings: an optional object, e.g. {"language": "en"}.

evaluation by test type:

response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
conversation: append the reply and judge the whole conversation. {"type": "conversation"}
tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.

For tool_call, each expected argument value is one of:

{"match_type": "exact", "value": <any>}: must equal value
{"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
{"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

Evaluators are linked via the separate evaluators field, not inside config.

Omit to create the test with no config and fill it in later via update

evaluators: typing.Optional[typing.List[RoutersTestsEvaluatorRef]] — Evaluators to link. Used by response and conversation tests

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.get(...) -> TestResponse

📝 Description

Get an agent test case by its ID

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.get(
    test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)

⚙️ Parameters

test_uuid: str — Test to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.tests.update(...) -> TestResponse

📝 Description

Update an agent test case

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.tests.update(
    test_uuid="b1c2d3e4-f5a6-7890-bcde-f12345678901",
)

⚙️ Parameters

test_uuid: str — Test to update

name: typing.Optional[str] — New test name. Omit to leave unchanged

type: typing.Optional[TestUpdateType]

What the test judges:

response: judges the generated reply
tool_call: diffs the generated tool calls
conversation: judges the full conversation

Immutable. Omit it, or send the current value

config: typing.Optional[typing.Dict[str, typing.Any]]

The calibrate test config. Three top-level keys.

history: the required conversation up to the agent's turn. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
evaluation: the required {type, ...}, where type matches the test's type below.
settings: an optional object, e.g. {"language": "en"}.

evaluation by test type:

response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
conversation: append the reply and judge the whole conversation. {"type": "conversation"}
tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.

For tool_call, each expected argument value is one of:

{"match_type": "exact", "value": <any>}: must equal value
{"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
{"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

Evaluators are linked via the separate evaluators field, not inside config.

Replaces the stored config. Omit to leave unchanged

evaluators: typing.Optional[typing.List[RoutersTestsEvaluatorRef]] — New evaluator links for the test. Omit to leave unchanged. An empty list clears them, except on conversation tests, which must keep at least one

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

AgentTests

client.agent_tests.link(...) -> AgentTestsCreateResponse

📝 Description

Link one or more tests to an agent. Tests that are already linked are skipped.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.link(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    test_uuids=[
        "b1c2d3e4-f5a6-7890-bcde-f12345678901"
    ],
)

⚙️ Parameters

agent_uuid: str — Agent to link tests to

test_uuids: typing.List[str] — Tests to link. Any that are already linked are skipped

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.list_for_agent(...) -> PaginatedResponseTestListResponse

📝 Description

List the tests linked to an agent.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.list_for_agent(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent whose linked tests to list

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.list_runs_for_agent(...) -> PaginatedResponseAgentTestRunListItem

📝 Description

List an agent's test runs with their results

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.list_runs_for_agent(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent whose test runs to list

type: typing.Optional[ListRunsForAgentAgentTestsRequestType]

Filter by run type. Omit to return both:

llm-unit-test: single runs of an agent's tests
llm-benchmark: multi-model comparisons

status: typing.Optional[TaskStatus] — Filter by run status. Omit for all statuses

has_failures: typing.Optional[bool] — Filter by whether the run has any failing test case or model. true returns only runs with failures (or errors), false only clean runs. Omit for both

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.run(...) -> AgentTestRunCreateResponse

📝 Description

Run an agent's linked tests as a background job, returning a task ID to poll.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.run(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

agent_uuid: str — Agent to test

test_uuids: typing.Optional[typing.List[str]] — Tests to run. Omit to run all tests linked to the agent

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.run_batch(...) -> BatchTestRunResponse

📝 Description

Run agent tests for every agent, or for a selected set.

🔌 Usage

from calibrate import Calibrate, BatchRunRequest
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.run_batch(
    request=BatchRunRequest(),
)

⚙️ Parameters

request: typing.Optional[BatchRunRequest]

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.get_run(...) -> TestRunStatusResponse

📝 Description

Poll a test run for its status and evaluation results.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.get_run(
    task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)

⚙️ Parameters

task_id: str — Test run to poll for status and results

only_failed: typing.Optional[bool] — Return only failing test cases. Omit to return every case

compact: typing.Optional[bool] — Return a compact response that omits heavy detail fields (results.output, results.test_case, results.judge_results, results.reasoning, evaluators.output_config), keeping only the lightweight decision fields. Omit for full detail

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.benchmark(...) -> AgentTestRunCreateResponse

📝 Description

Run a multi-model benchmark on an agent's linked tests as a background job.

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.benchmark(
    agent_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    models=[
        "openai/gpt-4.1",
        "anthropic/claude-sonnet-4"
    ],
)

⚙️ Parameters

agent_uuid: str — Agent to benchmark

models: typing.List[str] — Model names to benchmark

test_uuids: typing.Optional[typing.List[str]] — A subset of the agent's linked tests to benchmark. Each ID must be linked to the agent. Omit to run all linked tests

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.agent_tests.get_benchmark(...) -> BenchmarkStatusResponse

📝 Description

Get the results of a benchmark run

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.agent_tests.get_benchmark(
    task_id="a3b2c1d0-e5f4-3210-abcd-ef1234567890",
)

⚙️ Parameters

task_id: str — Benchmark run to poll for status and results

only_failed: typing.Optional[bool] — Return only failing test cases for each model. Omit to return every case

compact: typing.Optional[bool] — Return a compact response that omits heavy detail fields (model_results.test_results, evaluators.output_config), keeping only the lightweight decision fields. Omit for full detail

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

Evaluators

client.evaluators.list(...) -> PaginatedResponseEvaluatorResponse

📝 Description

List your evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.list()

⚙️ Parameters

evaluator_type: typing.Optional[ListEvaluatorsRequestEvaluatorType] — Filter by what the evaluator judges. Omit for all types

data_type: typing.Optional[ListEvaluatorsRequestDataType] — Filter by modality. Omit for all

include_defaults: typing.Optional[bool] — Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.create(...) -> EvaluatorCreateResponse

📝 Description

Create an evaluator along with its first version, which is set live

🔌 Usage

from calibrate import Calibrate, EvaluatorVersionCreate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.create(
    name="name",
    version=EvaluatorVersionCreate(
        judge_model="judge_model",
        system_prompt="system_prompt",
    ),
)

⚙️ Parameters

name: str — Evaluator name, unique within your workspace

version: EvaluatorVersionCreate — The evaluator's first version. Set as live when you create the evaluator

description: typing.Optional[str] — Description. Omit to leave blank

evaluator_type: typing.Optional[EvaluatorCreateEvaluatorType]

What the evaluator judges:

tts: TTS audio
stt: one transcript
llm: a reply with its conversation history
llm-general: a standalone input and output pair
conversation: a full conversation

data_type: typing.Optional[EvaluatorCreateDataType]

The modality the judge reads:

text
audio

output_type: typing.Optional[EvaluatorCreateOutputType]

How the evaluator scores:

binary: pass or fail
rating: a numeric score, using the scale in output_config

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.get(...) -> EvaluatorDetailResponseCompact

📝 Description

Get one evaluator with its full version history

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.get(
    evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

evaluator_uuid: str — Evaluator to retrieve

compact: typing.Optional[bool] — Return a compact response that omits heavy detail fields (versions.system_prompt, versions.output_config, versions.variables), keeping only the lightweight decision fields. Omit for full detail

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.evaluators.create_version(...) -> VersionCreateResponse

📝 Description

Add a new version to an evaluator you created

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.evaluators.create_version(
    evaluator_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    judge_model="judge_model",
    system_prompt="system_prompt",
)

⚙️ Parameters

evaluator_uuid: str — Evaluator to add a version to

judge_model: str — The model that runs the judge, named the way its provider does, for example openai/gpt-4.1 or anthropic/claude-sonnet-4

system_prompt: str — Judge system prompt. May contain {{variable}} placeholders

output_config: typing.Optional[OutputConfig] — The scale points and their labels. Required for a rating evaluator. A binary evaluator uses the default Correct/Wrong labels unless you set your own

variables: typing.Optional[typing.List[VariableSpec]] — Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable's description or default, but not add, remove, or rename one

make_live: typing.Optional[bool] — When true, immediately point the evaluator's live version at this new version

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

AnnotationTasks

client.annotation_tasks.list(...) -> PaginatedResponseAnnotationTaskResponse

📝 Description

List annotation tasks with linked evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.list()

⚙️ Parameters

q: typing.Optional[str] — Case-insensitive substring search on name. Blank is a no-op

limit: typing.Optional[int] — Maximum number of items to return. Omit for no limit (all items)

offset: typing.Optional[int] — Number of items to skip before returning results

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.create(...) -> AnnotationTaskCreateResponse

📝 Description

Create an annotation task for annotators to label items against evaluators

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.create(
    name="name",
    type="stt",
)

⚙️ Parameters

name: str — Task name, unique within your workspace

type: AnnotationTaskCreateType

Task type. Determines the shape of each item's payload.

stt: judge a transcript on its own
llm: judge one response with its conversation history
llm-general: judge a standalone input -> output pair
conversation: judge a full conversation

description: typing.Optional[str] — A description for the task. Omit for none

evaluator_ids: typing.Optional[typing.List[str]] — IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get(...) -> AnnotationTaskResponse

📝 Description

Get one annotation task with linked evaluators, items, and labelling jobs

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Task to retrieve

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.set_evaluators(...) -> EvaluatorSetResponse

📝 Description

Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.set_evaluators(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    evaluator_ids=[
        "f47ac10b-58cc-4372-a567-0e02b2c3d479"
    ],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

evaluator_ids: typing.List[str] — The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.add_items(...) -> BulkCreateItemsResponse

📝 Description

Bulk-create annotation items in a task, optionally seeding human annotations

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.add_items(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    items=[],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

items: typing.List[AnnotationItemPayload] — Items to insert. Insertion order is preserved

annotator_id: typing.Optional[str] — Annotator these initial annotations belong to. Required when any item carries annotations

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.update_items(...) -> BulkUpdateItemsResponse

📝 Description

Bulk-update item payloads in a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.update_items(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    updates=[],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

updates: typing.List[ItemUpdatePayload] — The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.create_evaluator_run(...) -> EvaluatorRunLaunchResponse

📝 Description

Run evaluators on task items as a background job

🔌 Usage

from calibrate import Calibrate, EvaluatorRunRequestEntry
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.create_evaluator_run(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    evaluators=[
        EvaluatorRunRequestEntry(
            evaluator_id="f47ac10b-58cc-4372-a567-0e02b2c3d479",
        )
    ],
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

evaluators: typing.List[EvaluatorRunRequestEntry] — The evaluators to run. Each must be linked to the task

item_ids: typing.Optional[typing.List[str]] — Item IDs to run on. Required when select_all=false. Ignored when select_all=true

select_all: typing.Optional[bool] — When true, run on every item in the task. Set q to run only items whose name matches it

q: typing.Optional[str] — Case-insensitive substring filter on payload.name. Applies only when select_all=true

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_evaluator_run(...) -> EvaluatorRunResponse

📝 Description

Get one evaluator-run job with results and human-agreement summary

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_evaluator_run(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
    job_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

job_uuid: str — The evaluator run to act on

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_agreement(...) -> TaskAgreementResponse

📝 Description

Get human-vs-human and human-vs-evaluator agreement metrics for a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_agreement(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

bucket: typing.Optional[GetAgreementAnnotationTasksRequestBucket] — How to bucket points in the trend series

days: typing.Optional[int] — Trailing window in days for the trend series

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

client.annotation_tasks.get_summary(...) -> TaskSummaryResponse

📝 Description

Get a paginated summary table of items, evaluator runs, and human annotations for a task

🔌 Usage

from calibrate import Calibrate
from calibrate.environment import CalibrateEnvironment

client = Calibrate(
    api_key="<value>",
    environment=CalibrateEnvironment.DEFAULT,
)

client.annotation_tasks.get_summary(
    task_uuid="f47ac10b-58cc-4372-a567-0e02b2c3d479",
)

⚙️ Parameters

task_uuid: str — Annotation task to act on

item_id: typing.Optional[str] — Filter rows to a single item. The full task-wide annotator union is still returned in annotators

live_only: typing.Optional[bool] — When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded

disagreement_only: typing.Optional[bool] — When true, keep only rows where the evaluator disagreed with at least one annotator

q: typing.Optional[str] — Case-insensitive substring search on payload.name. Blank is a no-op

sort_by: typing.Optional[str] — Sort key for the results

order: typing.Optional[GetSummaryAnnotationTasksRequestOrder] — Sort direction

limit: typing.Optional[int] — Maximum number of items to return

offset: typing.Optional[int] — Number of items to skip before returning results

compact: typing.Optional[bool] — Return a compact response that omits heavy detail fields (rows.payload, rows.evaluator_reasoning, rows.annotations.reasoning, evaluators.versions.system_prompt, evaluators.versions.output_config, evaluators.versions.variables, item_comments), keeping only the lightweight decision fields. Omit for full detail

request_options: typing.Optional[RequestOptions] — Request-specific configuration.

FilesExpand file tree

reference.md

Latest commit

History

reference.md

File metadata and controls

Reference

Agents

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

Tests

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

AgentTests

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters

Evaluators

📝 Description

🔌 Usage

⚙️ Parameters

📝 Description

🔌 Usage

⚙️ Parameters