Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,11 @@ jobs:
# else: no npm ci anywhere else gets slower, and a contributor who has not run this line gets
# a skip rather than a failure, exactly as the pip line above behaves.
- run: npm install --no-save @mastra/core @ai-sdk/openai
# From the tree, not PyPI: the `openai-agents-handoff` check exists to prove the tracing layer
# in *this* commit works, and installing a published copy would test the last release instead.
# From the tree, not PyPI: the `openai-agents-handoff` and `langgraph-nodes` checks exist to
# prove the adapters in *this* commit work, and installing a published copy would test the
# last release instead.
- run: pip install -e python-openai-agents
- run: pip install -e python-langgraph
# `--require-all`, because installing them above was never what enforced it. Two checks were
# added without a line here and CI stayed green on the skips, while the README said "in CI"
# about both. A skip is now a failure on this runner, so forgetting the line above fails
Expand Down Expand Up @@ -110,6 +112,11 @@ jobs:
# of its tests are about how `install()` registers with it.
- run: pip install ./python-openai-agents[dev]
- run: python -m pytest python-openai-agents/ -q
# The third. Its `[dev]` extra brings langgraph and langchain-core, because the half of its
# suite that matters builds real graphs and invokes them — the unit tests assert what
# langchain-core is believed to pass, and only a real graph keeps that belief honest.
- run: pip install ./python-langgraph[dev]
- run: python -m pytest python-langgraph/ -q

plugin-api-neutrality:
name: plugin API neutrality
Expand Down
81 changes: 81 additions & 0 deletions .github/workflows/release-langgraph.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# `orcareplay-langgraph` records a LangGraph run's own structure — which node ran, in which
# superstep, and how it ended — into an OrcaReplay trace. It ships on its own tag for the same
# reason `orca-trace` and `orcareplay-openai-agents` do: a Python-only fix must not force a version
# bump across twelve npm packages, and the npm release must not fail because PyPI had a bad day.
#
# A *fourth* release workflow rather than another job beside the others, because the three Python
# packages are independent. They share a repository and nothing else: different sources, different
# versions, different reasons to cut a release. One tag that published several would make every fix
# to one reissue the rest, and PyPI does not allow reusing a version number.
#
# Publishing uses PyPI Trusted Publishing — GitHub's OIDC identity is exchanged for a short-lived
# upload token, so there is no PyPI secret in this repository to leak or rotate. It needs a
# one-time registration at pypi.org (Publishing -> Add a pending publisher) naming:
#
# PyPI project orcareplay-langgraph
# owner Continuum-AI-Corp
# repository OrcaReplay
# workflow release-langgraph.yml
# environment pypi
#
# Until that exists the upload step fails and nothing else is affected.
name: Release (orcareplay-langgraph)

on:
push:
tags: ['langgraph-v*']
workflow_dispatch:
inputs:
dry-run:
description: 'Build and check without uploading'
type: boolean
default: true

jobs:
release-langgraph:
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write # Trusted Publishing: this is the whole credential
contents: read
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.12'

# A tag that disagrees with pyproject would publish a version nobody can reproduce from the
# tree. Same check the other three releases make, for the same reason.
- name: Tag matches package version
if: startsWith(github.ref, 'refs/tags/langgraph-v')
run: |
set -euo pipefail
tag="${GITHUB_REF#refs/tags/langgraph-v}"
pkg=$(python -c "import tomllib;print(tomllib.load(open('python-langgraph/pyproject.toml','rb'))['project']['version'])")
[ "$tag" = "$pkg" ] || { echo "tag langgraph-v$tag does not match pyproject $pkg"; exit 1; }

# Neither langgraph nor langchain-core is a dependency — this package is inert without them,
# and `orca record` puts it in front of every Python process a recording starts, so importing
# it has to cost nothing. The `dev` extra brings them, because half the suite builds real
# graphs: the unit tests assert what langchain-core is believed to pass, and only a real graph
# keeps that belief honest.
- name: Test
working-directory: python-langgraph
run: |
python -m pip install --upgrade pip
pip install -e '.[dev]'
python -m pytest -q

- name: Build
working-directory: python-langgraph
run: |
pip install --upgrade build twine
python -m build
twine check dist/*

- name: Publish to PyPI
if: ${{ github.event.inputs.dry-run != 'true' }}
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages-dir: python-langgraph/dist
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@ Three more layers catch what the protocol cannot see: an exit code, a real durat
byte came out of, a file written without telling anyone. A fifth exists for the agents that read no
base-URL variable at all — see [which agents](#which-agents). A sixth reads the agent's own account
of its structure, for a harness that has one: which sub-agent ran, which handed off to which,
whether a guardrail tripped — none of which reaches the wire.
whether a guardrail tripped, which graph node produced what and which of them called no model at
all — none of which reaches the wire.

```mermaid
%%{init: {'theme':'neutral'}}%%
Expand Down Expand Up @@ -474,7 +475,7 @@ whether orca understands the wire format it speaks once it arrives.
| **OpenClaw** | `orca record openclaw` — the hook for the gateway, inherited variables for the agents it spawns | works |
| **opencode** | `orca record opencode` | adapter shipped, both origins redirected |
| **goose** (Block) | `orca record goose` — `OPENAI_HOST` **and** `OPENAI_BASE_URL`, `ANTHROPIC_HOST` → Responses API | works — driven end to end against goose 1.49.0, [what is different about it](#the-harness-that-reads-different-variables) |
| **LangGraph / LangChain** | `OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL` | works — a two-node graph, streaming and with a tool, records and replays at `exact=2` and forks live, [in CI](test/integrations/) |
| **LangGraph / LangChain** | `OPENAI_BASE_URL`, `ANTHROPIC_BASE_URL` | works — a two-node graph, streaming and with a tool, records and replays at `exact=2` and forks live, [in CI](test/integrations/); `pip install orcareplay-langgraph` [also records the graph](python-langgraph/README.md) — which node ran, in which superstep, including the ones that call no model |
| **OpenHands** | `orca record generic-openai -- python your_agent.py` — its SDK wraps LiteLLM and reads `OPENAI_API_BASE` | works — the SDK's own LLM layer records and replays at `exact=1`, [in CI](test/integrations/) |
| **CrewAI** | `orca record generic-openai -- python your_crew.py` — since 1.x its own provider, reading `OPENAI_API_BASE` and `OPENAI_BASE_URL` | works — a real `Agent`, `Task` and `Crew` records and replays at `exact=1`, [in CI](test/integrations/); [what 1.x changed](docs/integrations.md#crewai) |
| **Aider** | `orca record generic-openai -- python your_agent.py` — routes through LiteLLM, which reads `OPENAI_API_BASE` | works — the LiteLLM layer records and replays at `exact=1`, [in CI](test/integrations/) |
Expand Down
14 changes: 8 additions & 6 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sequence of `npm publish` calls run by hand at the end of a long day.
- An **`NPM_TOKEN`** repository secret, from an npm account that owns the `orcareplay` name and the
`@orcareplay` scope. An automation token, not a personal one.
- **Actions enabled** for the repository, and Actions billing active on the org.
- For the two Python packages, a **pending publisher** registered at pypi.org under
- For the three Python packages, a **pending publisher** registered at pypi.org under
*Publishing → Add a pending publisher*, and a repository **environment named `pypi`**. There is
no PyPI secret: Trusted Publishing exchanges GitHub's OIDC identity for a short-lived upload
token, so there is nothing here to leak or rotate. One registration per package, all four other
Expand All @@ -19,6 +19,7 @@ sequence of `npm publish` calls run by hand at the end of a long day.
| --- | --- | --- | --- | --- |
| `orca-trace` | `Continuum-AI-Corp` | `OrcaReplay` | `release-python.yml` | `pypi` |
| `orcareplay-openai-agents` | `Continuum-AI-Corp` | `OrcaReplay` | `release-openai-agents.yml` | `pypi` |
| `orcareplay-langgraph` | `Continuum-AI-Corp` | `OrcaReplay` | `release-langgraph.yml` | `pypi` |

A pending publisher does not reserve the name — it is only honoured on the first upload, and
another account registering that name first invalidates it. So publish reasonably soon after
Expand Down Expand Up @@ -129,14 +130,15 @@ registry at a version the rest do not name — which is the one state npm will n
To rehearse without sending anything: **Actions → Release → Run workflow**, leaving *dry run*
checked. It packs and validates every tarball and publishes nothing.

## The two Python packages
## The three Python packages

They are not part of the npm release and do not share its version. Each ships on its own tag:

| package | source | tag | workflow |
| --- | --- | --- | --- |
| `orca-trace` — the read-only reader for the trace format | `python/` | `py-v<version>` | `release-python.yml` |
| `orcareplay-openai-agents` — records the Agents SDK's own run structure | `python-openai-agents/` | `agents-v<version>` | `release-openai-agents.yml` |
| `orcareplay-langgraph` — records which LangGraph node ran, and in which superstep | `python-langgraph/` | `langgraph-v<version>` | `release-langgraph.yml` |

```console
# edit the version in the package's pyproject.toml, merge that, then:
Expand All @@ -148,11 +150,11 @@ builds, and `twine check`s before it uploads — the same gate the npm release a
reason. **Actions → Release (…) → Run workflow** with *dry run* left checked does everything except
the upload, which is how to find out whether a release would work without making one.

> **Why three workflows rather than one.** A Python-only fix must not force a version bump across
> twelve npm packages, and the npm publish must not fail because PyPI had a bad day. The two Python
> **Why four workflows rather than one.** A Python-only fix must not force a version bump across
> twelve npm packages, and the npm publish must not fail because PyPI had a bad day. The Python
> packages are separated from each other for the same reason: they share a repository and nothing
> else, so one tag publishing both would make every `orca-trace` fix reissue the adapter — and PyPI
> does not allow reusing a version number.
> else, so one tag publishing several would make every `orca-trace` fix reissue the adapters — and
> PyPI does not allow reusing a version number.

## Why order matters

Expand Down
42 changes: 42 additions & 0 deletions docs/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,48 @@ orca's, but a recorded run will differ from an unrecorded one in exactly that fi
a replay rebuilds it by re-running the graph. A database-backed checkpointer is a different matter
and is not covered by the checks here.

### Recording the graph itself

Everything above is about the traffic. The *graph* — which node ran, in which superstep, and how it
ended — is not on the wire at all, and one package puts it in the trace:

```console
pip install orcareplay-langgraph
```

No code change: `orca record` attaches it to a graph you have not edited, the same way it attaches
the fetch hook and the shell shim. Measured on `plan → validate → answer` against a stub origin:

```
graph.node.start plan
model.request
model.response
graph.node.end plan
graph.node.start validate ← makes no request of any kind
graph.node.end validate
graph.node.start answer
graph.node.end answer
```

`validate` is the point. Validators, state reducers, routers and writers call no model, so without
this a proxy sees a graph with those nodes and a graph without them as the same run. The same goes
for a parallel fan-out, which reaches the wire as two ordinary consecutive turns, and for a node
that raised — `graph.node.end` carries the exception's class, which no request can.

Two records per node and nothing else: `node`, `step`, the run ids that pair them, the instants,
and `error`. No state, no inputs, no outputs, and no exception message. The model exchanges stay
with the proxy, which already has them byte for byte.

**One caveat, and it is orca's rather than the package's.** A node record carries the instant the
callback fired, measured exact against the agent's own clock. A `model.request` is stamped when
orca persists it — after the response and after a workspace snapshot, measured 95 ms late on one
call and 31 ms on the next. So the node boundaries are trustworthy, and a model call within about a
tenth of a second of one can sit on the wrong side of it.

See [`python-langgraph/`](../python-langgraph/README.md) for how a node is told apart from an inner
runnable, a conditional edge and the graph itself — which is the whole design, and also what keeps a
`run_name` interpolated from user data out of the trace.

---

## CrewAI
Expand Down
4 changes: 3 additions & 1 deletion examples/traces/run_9f2c14a03b71/events.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,6 @@
{"seq":25,"ts":"2026-08-29T10:14:14.680Z","mono_us":36680000,"turn":5,"type":"agent.start","actor":"harness","attrs":{"name":"Triage","handoffs":"Billing Specialist","tools":0,"output_type":"str"}}
{"seq":26,"ts":"2026-08-29T10:14:14.960Z","mono_us":36960000,"turn":5,"type":"agent.guardrail","actor":"harness","attrs":{"name":"not_empty","triggered":false}}
{"seq":27,"ts":"2026-08-29T10:14:15.240Z","mono_us":37240000,"turn":5,"type":"agent.handoff","actor":"harness","attrs":{"from":"Triage","to":"Billing Specialist"}}
{"seq":28,"ts":"2026-08-29T10:14:15.800Z","mono_us":37800000,"turn":5,"type":"run.end","actor":"orca","attrs":{"exit_code":1}}
{"seq":28,"ts":"2026-08-29T10:14:15.430Z","mono_us":37430000,"turn":5,"type":"graph.node.start","actor":"harness","attrs":{"node":"summarise_invoice","step":1,"run_id":"3f6a1c8e-0b2d-4f51-9c77-1a5be2d40c93","parent_run_id":"9d4c7b12-5e83-4a06-b1f2-77c0ae35d918","started_at":"2026-08-29T10:14:15.430Z"}}
{"seq":29,"ts":"2026-08-29T10:14:15.610Z","mono_us":37610000,"turn":5,"type":"graph.node.end","actor":"harness","attrs":{"node":"summarise_invoice","step":1,"run_id":"3f6a1c8e-0b2d-4f51-9c77-1a5be2d40c93","ended_at":"2026-08-29T10:14:15.610Z"}}
{"seq":30,"ts":"2026-08-29T10:14:15.800Z","mono_us":37800000,"turn":5,"type":"run.end","actor":"orca","attrs":{"exit_code":1}}
6 changes: 3 additions & 3 deletions examples/traces/run_9f2c14a03b71/manifest.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"schema_version": "0.2.0",
"schema_version": "0.3.0",
"run_id": "run_9f2c14a03b71",
"created_at": "2026-08-29T10:13:38.000Z",
"ended_at": "2026-08-29T10:18:29.000Z",
Expand Down Expand Up @@ -28,7 +28,7 @@
"node": "v22.22.2"
},
"counts": {
"events": 29,
"events": 31,
"blobs": 5
},
"redaction": {
Expand All @@ -40,7 +40,7 @@
},
"exit_code": 1,
"integrity": {
"events_sha256": "f5664c53bd40fcddf3ab763eef2f02247278cfccd41b77af9ad9a9b3df3f984e",
"events_sha256": "07aa293d6af485d1637a64e0004453282bcca43f2b4cb32970eda4cabbdbd0de",
"blob_count": 5
}
}
Loading
Loading