Skip to content

Latest commit

 

History

History
103 lines (79 loc) · 5.94 KB

code-experiments.md

File metadata and controls

103 lines (79 loc) · 5.94 KB

Test a code fix against a recorded failure

A supplier changed its response field from items to products. The original workflow fetched the response successfully, then failed in its normalizer. An input experiment can show what the next step needs. A code experiment tests the implementation you actually want to keep, using the response that caused the failure.

Try the included example

Start Checkpoint 0.3.0, choose Investigate the example, and open the failed Supplier feed.

  1. Choose Test code change. The included revised workflow is example:supplier_fixed.
  2. Choose Review change. Check the original and revised fingerprints, selected files and source comparison. The normalizer accepts either field name; the graph still has four steps.
  3. Choose Run code experiment. All four steps execute again, including Fetch. Its supported HTTP response comes from the original recording, so the fixture request counter stays one.
  4. Open View source change and Compare outcomes. The original still contains the failed normalizer. The experiment has three normalized inventory items and a completed preview.
  5. Export a reproduction after reviewing its files. Extract it elsewhere and run python reproduce.py. It executes the revised code offline and checks the observed outcome.

The relevant code change is small:

payload = inputs["fetch"]
payload = dict(payload, items=payload.get("items", payload.get("products")))
if payload["items"] is None:
    raise ValueError("The supplier response has no inventory items")

This does not send the prepared inventory update to a real destination. The entire supplier case is fictional. The recorded website shows saved executions and cannot run your local code.

Use your own revised workflow

Record a workflow following the own-workflow guide. Keep the revised module in a separate directory with its selected data and imports, or edit the original file after the recording. In Test code change, enter its absolute path and review it.

The revised workflow must expose the same ordered step IDs and dependencies. Names, source, defaults, timeout/retry settings and exact requirement pins can differ, but execution still starts with the original run's input. Revised defaults do not replace that input. There is no inherited prefix or replacement-input boundary in a code experiment.

Inspection imports trusted local Python. Keep imports free of effects, and install the revised workflow's declared requirements in the selected Python environment first.

The equivalent CLI flow, with the local engine running:

checkpoint plan-code --source ORIGINAL_RUN_ID --entry /path/to/revised/workflow.py
checkpoint experiment --source ORIGINAL_RUN_ID --entry /path/to/revised/workflow.py
checkpoint compare --source ORIGINAL_RUN_ID --other EXPERIMENT_RUN_ID
checkpoint review-export --source EXPERIMENT_RUN_ID --mode reproduction
checkpoint export --source EXPERIMENT_RUN_ID --mode reproduction

On Windows use checkpoint.exe and an appropriate local path. Each command returns JSON. experiment obtains a fresh plan and binds creation to that revised fingerprint. The API exposes the reviewed hash separately so the interface can reject a file changed after review.

What stays strict

Only ctx.http() and the explicitly configured HTTPX transports are recorded. Requests match the original tape by step, attempt, ordinal and supported request identity. A changed or additional request fails without a live fallback. A step that returns successfully while leaving an expected request unused fails with ReplayTapeMismatch. A step that already failed retains that failure; it is not described as a successful tape match.

The source observation may not have reached every step. Revised code that introduces an HTTP request in a previously skipped step has no recording to use and stops. Make a separate live recording when you intentionally need different external input. An experiment based on another code experiment continues using the original tape.

Ordinary Replay still checks the selected code and workflow settings. It cannot become a code experiment just because a file changed. V0.2 observations can replay through the compatible v0.3 runner only when their selected code and all workflow settings still match; the SDK-version metadata change is handled explicitly.

Files, environment variables, time, randomness, unconfigured clients and arbitrary Python effects remain under your code's control. Strict supported-HTTP replay is not network isolation or a sandbox. Do not use a code experiment to run untrusted modules.

Source retention and portable evidence

New observations retain the exact selected source/data bytes in content-addressed artifacts. Workers recheck the reviewed definition before each step. Editing a local file later does not change the stored source comparison or the code selected for an export. If the file changes before execution, that worker fails instead of silently running a different revision.

Older observations without source snapshots remain readable. The interface labels their original source as unavailable and keeps the recorded hashes; it does not substitute today's file as historical evidence. Such observations still require their recorded files for executable export, and an older SDK observation uses its original release for a compatible reproduction.

Code-experiment bundles contain the revised executable workflow, available original source, both identities, actual operations and the original tape needed by the verifier. Source and response bytes are covered by the manifest hashes and the existing review digest. Inspection imports do not execute code and can be exported again without the original engine. Format, worker protocol and SQLite schema remain 1 through additive optional metadata.

Selected source, data, logs and responses can contain sensitive content. Inspect the actual export inventory before sharing. Nothing is automatically redacted.