diff --git a/research/instrumentation-providers.md b/research/instrumentation-providers.md new file mode 100644 index 0000000000..99f391b18a --- /dev/null +++ b/research/instrumentation-providers.md @@ -0,0 +1,679 @@ +--- +issue: TBD +status: proposed +last_updated: "2026-08-07" +--- + +# Instrumentation providers + +eve should give you tracing out of the box: local traces you can read with +`eve trace` in development, and Agent Runs in production. It mostly does. What +it cannot do is keep those on once you bring a backend of your own. Adding an +`agent/instrumentation.ts` switches the local trace writer off completely, and +not for any interesting reason — your `setup` callback calls `registerOTel`, a +process only gets one OpenTelemetry tracer provider, and yours took it. eve's +[Honeycomb integration](https://eve.dev/integrations/honeycomb-instrumentation) +is ten lines and eight of them are that call; most of the other +[instrumentation integrations](https://eve.dev/integrations?filter=instrumentation) +are the same file with a different URL. + +One of them is not an OpenTelemetry backend at all, and it is the more +interesting failure. [Braintrust](https://eve.dev/integrations/braintrust) +records rows rather than spans, so there is no provider for it to take and no +exporter for it to configure, and eve offers it nothing else. Its integration is +built on the runtime event stream instead — the protocol that renders a +conversation — and reconstructs agent execution from it. + +So eve should own the pipeline and expose the event bus underneath it. Then +every backend an agent wants runs at once and none of them clobbers another: in +development and in production alike, local tracing and Agent Runs are always on, +an OpenTelemetry backend is processors you add, and anything else is a provider +you add. + +## Current problems + +### One global slot + +OpenTelemetry has one global tracer provider slot per process. You can build as +many providers as you like, but only one is reachable through +`trace.getTracer()`, which is where the model SDK, auto-instrumentation, and +your own code all go looking. The second registration is refused: it returns +false, which `registerOTel` discards, and logs through `diag`, which goes +nowhere unless `OTEL_LOG_LEVEL` is set. The exporter you configured never +receives anything and nothing tells you. You also cannot add to a provider after +it is built — that method was removed in the current major version of the SDK. + +### One destination for spans + +Because the slot holds one provider, an agent gets one place its spans can go. +Put Honeycomb and Datadog in the same `setup` and only the first `registerOTel` +call reaches anything; the second exports nothing. eve's local trace writer is +in exactly that position — another `registerOTel` caller with its own processor, +which starts a probe span to confirm the registration took and throws if it did +not. eve notices the collision; nobody else does. + +### Adding a backend turns local tracing off + +Rather than let that error fire, eve avoids the collision. When an +`agent/instrumentation.ts` exists, eve does not install the local tracing plugin +at all, and it never installs it in production regardless of what you wrote. You +get your exporter and lose `.eve/traces` and `eve trace`, and you lose the +lifecycle bus underneath them along with it. The defaults and the extension are +mutually exclusive, and the defaults lose. + +### A backend that does not speak OpenTelemetry has nowhere to go + +There is no provider surface, so Braintrust infers execution structure from the +runtime event stream — a protocol built to render a conversation, not to +describe execution. It ends up carrying its own pairing of requests to results, +its own ordering, its own replay-stable identity, and its own way of holding +state across a Workflow step boundary, because eve exposes none of those. + +eve already has the contract that answers all of it: a lifecycle bus carrying +session, turn, step, and call events, with state kept per provider and per +operation and failures isolated to the provider that caused them. eve's own OTel +mapping is one provider on it. The whole thing is internal, and an authored +`instrumentation.ts` turns it off. + +### Agent Runs is assembled from Workflow's telemetry, not eve's + +Agent Runs renders a run by pre-processing Workflow's OpenTelemetry data, so it +can only show what Workflow spans imply. eve already produces a better +description of a run than that — the `agent.*` spans behind `eve trace` — and +cannot get them there, because it does not hold the pipeline in production. + +## The bus + +eve owns the pipeline and a provider is the unit of it. `instrumentation/` +becomes a directory and each file is one provider, the way `tools/` and `hooks/` +already work, so a backend is a file named after itself and the directory is the +list. A provider handles the lifecycle events it cares about; nothing about it is +specific to how those events are recorded. + +```mermaid +flowchart TD + H["harness execution"] --> B["lifecycle bus — eve-owned
session · turn · step.attempt · action · model.call · tool.call"] + B --> O["otel()"] + B --> BT["braintrust()"] + B --> AU["authored provider"] + O --> TP["tracer provider — one global slot"] + TP --> EX["your exporters"] + TP --> LT["localTraces()"] + TP --> AR["agentRuns()"] +``` + +Only `otel` builds a tracer provider. Everything under it is a span processor, +eve's own included, which is why they are values you place rather than defaults +appended behind you. + +```typescript +// --------------------------------------------------------------------------- +// eve/instrumentation +// --------------------------------------------------------------------------- + +// `agent/instrumentation.ts` is gone. One file per provider under +// `agent/instrumentation/`, and `defineInstrumentation` makes one of them +// rather than holding the whole configuration. `functionId` and +// `traceChannelRequests` were only ever OpenTelemetry settings, so they move +// onto `otel()`; `recordInputs` and `recordOutputs` move onto the destinations. + +export function defineInstrumentation(provider: InstrumentationProvider): InstrumentationProvider; + +/** + * Turns off one of eve's two built-in integrations. Only meaningful as the default + * export of `instrumentation/local.ts` or `instrumentation/agent-runs.ts`. + */ +export function disableInstrumentation(): InstrumentationProvider; + +export interface InstrumentationProvider { + readonly events?: ProviderEvents; + /** Once per process. eve accepts no request until it resolves. */ + readonly setup?: (context: ProviderSetupContext) => void | PromiseLike; + readonly flush?: () => PromiseLike; + readonly shutdown?: () => PromiseLike; +} + +export interface ProviderSetupContext { + readonly agentName: string; + readonly environment: "development" | "preview" | "production"; + readonly frameworkVersion: string; + /** Present when this process is running an `eve eval` case. */ + readonly evaluation?: EvaluationRef; +} + +export interface ProviderEvents { + readonly "session.started"?: Handler; + readonly "session.completed"?: Handler; + readonly "session.failed"?: Handler; + // Not terminal. The session may resume with a later turn, so a provider + // holds session state across this and releases it on completed or failed. + readonly "session.waiting"?: Handler; + + readonly "turn.started"?: Handler; + readonly "turn.completed"?: Handler; + readonly "turn.failed"?: Handler; + readonly "turn.cancelled"?: Handler; + + // One actual model attempt, not one step. A step retried three times + // dispatches three of these; the protocol stream's `step.started` fires once + // per step and keeps that name. + readonly "step.attempt.started"?: Handler; + readonly "step.attempt.completed"?: Handler; + readonly "step.attempt.failed"?: Handler; + readonly "step.attempt.metadata"?: Handler; + + // eve's dispatch unit, and the one to reach for: a tool call, a skill load, + // or a subagent, correlated by call id. + readonly "action.started"?: Handler; + readonly "action.completed"?: Handler; + readonly "action.failed"?: Handler; + + // The model SDK's own view of one execution inside one attempt. An ordinary + // tool call fires these *and* `action.*`, so handling both records it twice. + readonly "tool.call.started"?: Handler; + readonly "tool.call.completed"?: Handler; + readonly "tool.call.failed"?: Handler; + readonly "model.call.started"?: Handler; + readonly "model.call.completed"?: Handler; + readonly "model.call.failed"?: Handler; +} + +type Handler = (event: TEvent, ctx: ProviderContext) => void | PromiseLike; + +/** Every event payload above extends this. */ +export interface LifecycleEvent { + /** + * Derived from identity eve already holds and stable across replay, and a + * `started` carries the same one as its terminal. What you key by it is + * written once however many times a worker replays the work that wrote it. + */ + readonly idempotencyKey: string; +} + +export interface ProviderContext { + /** + * Serialized per provider and per operation, handed back on the terminal and + * released there. A `completed` handler reads what its `started` handler + * wrote even when the two run in different processes. + * + * `set` is synchronous because it is not a write to a backend: it stages a + * value in the durable context eve commits once the dispatch settles. So a + * handler cannot await its own persistence, and a `set` from a handler that + * resolves after release is dropped rather than committed. + */ + readonly state: { get(): JsonValue | undefined; set(value: JsonValue): void }; +} +``` + +### What the bus carries + +These are the events eve's own OTel mapping already consumes, with one rename: +`attempt.*` becomes `step.attempt.*`. Not `step.*`, which is taken: the protocol +stream's `step.started` fires once per logical step, and a step retried three +times dispatches three `step.attempt.started` against that one. The span is +still `agent.step`, because a span is one attempt and the `agent.step.attempt` +attribute on it says which. + +`action.*` is the addition. eve's protocol already treats an action as the unit +— a tool call is one kind, alongside a skill load and a subagent — but the bus +carries only `tool.call`, so the OTel provider builds `agent.action` out of it +and hardcodes the kind. A skill load or a subagent call gets no span at all, and +neither does a tool that resolves after an approval, because the model SDK's +callbacks never fire in the process that resolves it. + +It does not replace `tool.call.*`: an ordinary in-process tool call fires both, +so handle one or record it twice. Reach for `action.*` — it is every dispatch eve +makes, including the three the model SDK never reports, and its `kind` says +which. `tool.call.*` is the SDK's own execution boundary, for when that is +specifically what you want; `model.call.*` is the other half of that view and +overlaps nothing. + +### State that outlives the handler + +`ctx.state` is what a provider would otherwise hand-roll. eve keys it by provider +and operation, hands it back on the terminal, and releases it there. For a +session, turn, step attempt, or action it is serialized, so a `completed` handler +reads what a `started` handler wrote even in a different process. + +This is not hypothetical: eve already keeps exactly this for itself. The store +that carries session and turn trace context across a step boundary now also holds +a pre-allocated span id and a start time, so the worker that settles a turn can +emit a span the descendants of an earlier worker already parented to — which is +what turned `agent.turn` from a zero-duration marker into a span with a real +duration. `agent.session` stays a marker because an idle session never closes. A +provider outside eve has neither the store nor the option to degrade its own +output instead. + +### One provider cannot stop the others + +A provider that throws is logged and skipped and the ones after it still run. A +provider that hangs would stop the bus rather than only itself, because dispatch +is sequential and awaited — a `started` handler has to finish before the events +for its children arrive. So each handler gets a timeout; exceeding it abandons +the handler and the bus moves on. A `started` that exceeds it also marks the +provider failed for that operation and skips its terminal handler, so `completed` +never fires against state `started` never got to write. + +Release does not depend on any of that, because releasing state is eve's, not the +handler's. eve releases a scope once its terminal dispatch settles — returned, +threw, or timed out — so an abandoned handler cannot leak serialized state into +the durable context. The handle is stamped with the scope it was issued for; a +`ctx.state.set` from a handler that resolves after release is dropped rather than +resurrecting a closed scope. Serialized state also survives step cancellation, +which discards the rest of a step's context changes: eve already carves out its +own trace state there by name, and the provider namespace is carved out the same +way — the namespace is eve's, the values inside it are the provider's. + +### Identity comes from the event + +A handler gets its identity from the event and nothing from ambient context. eve +cannot run handlers inside an OpenTelemetry span context: the span for a boundary +exists only because some provider built it, so making it current would give a +provider-neutral bus a dependency on OpenTelemetry and on which provider ran +first. `trace.getActiveSpan()` inside a handler reaches whatever the harness +happened to be under, which is not the span for the event it is handling. + +`idempotencyKey` does what state alone cannot. State survives a boundary, but a +replaying worker re-runs the `started` handler, and only a key stable across +replays makes that second write land where the first one did. eve's own OTel +provider gets there by allocating a span id and persisting it — available to it +only because it owns the store. + +### Flushing and eval runs + +`flush` is on the provider because a batching backend cannot see when the process +stops existing: on Vercel Functions it can freeze the moment the response goes +out, and an unflushed batch is lost. eve already flushes at the end of a step and +when a turn settles after cancellation; those become calls to every provider's +`flush`, plus a third when a session suspends. For `otel` that is `forceFlush` on +the tracer provider rather than anything per destination, which reaches every +processor under it — including the `BatchSpanProcessor` each wrapped exporter +sits behind, whose default batching is what loses spans on a frozen function in +[#679](https://github.com/vercel/eve/issues/679). + +`evaluation` on the setup context is there so a provider running under `eve eval` +can attach what it records to the eval run, rather than leaving the two +unrelated. + +## OpenTelemetry + +OpenTelemetry is one provider on that bus, and the only one that builds a tracer +provider. Because a process holds exactly one, its settings split in two. + +One file per integration, named for the integration. `instrumentation/sentry.ts` +says where spans go and nothing else, and eve knows what it is from what it +returns rather than from where it sits. + +A telemetry type with process-wide singletons gets one more file, named for the +type. `instrumentation/otel.ts` owns the things one tracer provider can only have +one of — the sampler, the resource, the propagators — and it is the +`registerOTel` call itself, made once after every destination has been collected. +It exists only when you have one of those to name, so reaching for a sampler +later adds a file rather than moving the ones you already wrote. Because the +singletons and the destinations are separate functions, they are separate types. + +eve has two built-in integrations: local traces, on in development, and Agent +Runs, on in production. You do not write a file for either one, and both are on +whether or not the directory exists, because a default you have to opt into is +not a default. Two filenames are reserved so you can say otherwise: +`instrumentation/local.ts` and `instrumentation/agent-runs.ts`. +Each accepts either `disableInstrumentation()` to turn that one off or the +matching `localTraces()` / `agentRuns()` value to keep it and narrow it. The +default export there is a union, and eve tells the two apart by a discriminant on +the value rather than by the filename. + +Agent Runs is production-only, and not by choice. On Vercel, `@vercel/otel` +reports spans through the request context, which is how eve gets them there for +free. + +**Today** + +```typescript +import { OTLPHttpProtoTraceExporter, registerOTel } from "@vercel/otel"; +import { defineInstrumentation } from "eve/instrumentation"; + +export default defineInstrumentation({ + setup: ({ agentName }) => + registerOTel({ + serviceName: agentName, + traceExporter: new OTLPHttpProtoTraceExporter({ + url: "https://api.honeycomb.io/v1/traces", + headers: { "x-honeycomb-team": process.env.HONEYCOMB_API_KEY! }, + }), + }), +}); +``` + +**Proposed** + +```typescript +// agent/instrumentation/honeycomb.ts +import { OTLPHttpProtoTraceExporter } from "@vercel/otel"; +import { otelIntegration } from "eve/instrumentation/otel"; + +export default otelIntegration({ + traceExporter: new OTLPHttpProtoTraceExporter({ + url: "https://api.honeycomb.io/v1/traces", + headers: { "x-honeycomb-team": process.env.HONEYCOMB_API_KEY! }, + }), +}); +``` + +There is no `otel.ts` in that example. eve registers the pipeline either way; the +file is only necessary when you have a sampler, a resource, or propagators to +name. + +Making `otel` a value a file exports rather than a side effect inside a callback +also deletes the machinery a side effect would need. There is nothing to scope to +a callback, no second call to detect, and no call that can escape through an +unawaited promise, because a value cannot be called twice or escape. + +eve holding the `registerOTel` call is also what makes real durations possible. A +span whose lifetime crosses a Workflow step boundary is emitted by a different +worker than the one that started it, so eve threads its own id generator into the +tracer provider and primes it, letting that span carry the id its descendants +already parented to. An author cannot supply that generator — it is eve's, keyed +to eve's state — which is why `idGenerator` is absent from `OtelOptions` despite +being something a process can only have one of. + +```typescript +// --------------------------------------------------------------------------- +// eve/instrumentation/otel +// --------------------------------------------------------------------------- + +/** + * The pipeline itself: the settings one tracer provider can only have one of, + * and the `registerOTel` call, made once after every `otelIntegration` beside + * it has been collected. Belongs in `instrumentation/otel.ts`. + */ +export function otel(options?: OtelOptions): InstrumentationProvider; + +export interface OtelOptions { + /** + * The function identifier on every span (`ai.telemetry.functionId`). + * Defaults to the agent name. + */ + readonly functionId?: string; + /** + * Emit the inbound HTTP `SERVER` span that wraps each channel request — the + * parent of the turn trace and of any outgoing HTTP spans. Defaults to + * `false`. + */ + readonly traceChannelRequests?: boolean; + /** Merged into eve's resource, which already carries the service name. */ + readonly resource?: Resource; + /** + * Head sampling, and it is global: it decides whether a span is created at + * all, so it thins eve's own sinks and the `traceparent` eve propagates along + * with your exporters — a ratio below 1 in development thins `eve trace` and + * the TUI too. To thin one backend only, drop spans in a processor. + */ + readonly sampler?: Sampler; + /** Composed into one propagator. All inject; the first to extract wins. */ + readonly propagators?: readonly TextMapPropagator[]; + + // No `spanProcessors` and no `traceExporter`. Every destination is an + // `otelIntegration` in its own file, which is what makes this file's fields + // exactly the ones a process can only have one of. +} + +/** + * One destination in that pipeline — `instrumentation/sentry.ts`. + * + * It carries no singletons, so declaring a sampler in a second file does not + * compile. + */ +export function otelIntegration(options: OtelIntegrationOptions): OtelIntegration; + +export interface OtelIntegrationOptions { + /** Merged with every other file's, in directory order. */ + readonly spanProcessors?: readonly SpanProcessor[]; + /** + * Wrapped in a default `BatchSpanProcessor` and appended. `@vercel/otel` + * takes only one exporter, so wrapping is what lets each file carry its own. + */ + readonly traceExporter?: SpanExporter; +} + +// eve installs its two built-in integrations itself. Local traces are on in +// development; in production eve passes `"auto"` and `@vercel/otel` reports +// through the Vercel request context, which is what reaches Agent Runs. +// +// Two filenames are reserved to override them. Export `disableInstrumentation()` +// from `instrumentation/local.ts` or `instrumentation/agent-runs.ts` to turn +// one off; pass content options to keep it and narrow what it records. + +export function localTraces(options?: ContentOptions): OtelIntegration; +export function agentRuns(options?: ContentOptions): OtelIntegration; + +export interface ContentOptions { + /** Record model prompts and tool call inputs. Defaults to `true`. */ + readonly recordInputs?: boolean; + /** Record model responses and tool call outputs. Defaults to `true`. */ + readonly recordOutputs?: boolean; +} + +// Not offered: `contextManager`, which eve's span nesting depends on, and +// `instrumentations`, which cannot reach anything eve imported before `setup`. +``` + +Dropping `instrumentations` is a removal rather than a simplification, though +what it removes never worked: eve imports the model SDK before an authored +`setup` runs, so an auto-instrumentation registered there could not patch it. +`runtimeContext` goes with it, and that one did work. Today an author returns it +from a `step.started` handler and eve merges it onto the AI SDK's telemetry span +as `ai.settings.context.*`; on the new bus there is no return value and no +ambient span, so both halves are gone rather than one. Enrichment that has to +reach a span belongs on `otel()`, where eve builds the spans and can say which +one it means — the shape of that surface is the open question here. + +`recordInputs` and `recordOutputs` are per destination, so eve implements them as +a span processor in front of that destination's exporter. It cannot edit the span +it is handed: that object is shared with every other processor, and stripping an +attribute there strips it everywhere. So it copies the span, drops what the +destination declined, and exports the copy. That covers the destinations eve +builds the processor for — an `otelIntegration`, not a raw `spanProcessor` an +author supplies. + +Capture at construction is then the union of what the destinations ask for, which +inverts what the current environment variable implies: content is written onto +the span if anything wants it, and each destination that declined simply never +exports it. The union of nothing is still nothing, so an agent whose every +destination declines never materializes a prompt on a span at all — the hard +guarantee survives, as a property of the whole configuration rather than of one +switch. `EVE_TRACES_CONTENT` becomes the override for `localTraces()` alone and +intersects with its options, so off still wins where it applies. + +### Agent Runs reads the same spans + +The `agent.*` spans behind `eve trace` become the one format. Agent Runs renders +those rather than inferring a run from Workflow's telemetry, and your exporters +receive the same spans wherever you send them. + +eve gets them there by passing `"auto"` in production, which is what installs +`@vercel/otel`'s reporting through the Vercel request context. + +### All the options at once + +```typescript +// agent/instrumentation/otel.ts — only what a process can have one of. +import { B3Propagator } from "@opentelemetry/propagator-b3"; +import { resourceFromAttributes } from "@opentelemetry/resources"; +import { ParentBasedSampler, TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base"; +import { otel } from "eve/instrumentation/otel"; + +const production = process.env.VERCEL_ENV === "production"; + +export default otel({ + functionId: "support-agent", + + resource: resourceFromAttributes({ + "deployment.environment": process.env.VERCEL_ENV ?? "development", + "service.version": process.env.VERCEL_GIT_COMMIT_SHA ?? "local", + }), + + // A twentieth in production, and globally: every destination sees the same + // twentieth, because the spans they would have read do not exist. + sampler: new ParentBasedSampler({ + root: new TraceIdRatioBasedSampler(production ? 0.05 : 1), + }), + + // An upstream service sends B3 headers, so read them and keep the trace. + propagators: [new B3Propagator()], +}); +``` + +```typescript +// agent/instrumentation/honeycomb.ts +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; +import { otelIntegration } from "eve/instrumentation/otel"; + +export default otelIntegration({ + traceExporter: new OTLPTraceExporter({ + url: "https://api.honeycomb.io/v1/traces", + headers: { "x-honeycomb-team": process.env.HONEYCOMB_API_KEY! }, + }), +}); +``` + +```typescript +// agent/instrumentation/datadog.ts — a second backend, because on-call +// reads this one. Structure and timing only; the prompts stay put. +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; +import { otelIntegration } from "eve/instrumentation/otel"; + +export default otelIntegration({ + traceExporter: new OTLPTraceExporter({ + url: process.env.DATADOG_OTLP_TRACES_ENDPOINT!, + headers: { "dd-api-key": process.env.DD_API_KEY! }, + }), + recordInputs: false, + recordOutputs: false, +}); +``` + +```typescript +// agent/instrumentation/agent-runs.ts — a reserved name. eve ships to Agent +// Runs in production without this file; write it only to narrow or stop it. +import { agentRuns } from "eve/instrumentation/otel"; + +export default agentRuns({ recordInputs: false }); + +// agent/instrumentation/local.ts — the other reserved name, turned off. +import { disableInstrumentation } from "eve/instrumentation"; + +export default disableInstrumentation(); +``` + +## Backends that are not OpenTelemetry + +Supporting a backend that is not OpenTelemetry needs two things from eve, and +they are the same two `otel` needs. + +The first is durable state. Whatever handle you open when something starts cannot +be held until it ends, because an approval suspends, a retry runs again, and a +replaying worker re-executes work it has already observed. So you put JSON in +`ctx.state`, eve keeps it under your namespace, and the terminal handler reads it +wherever it runs. + +The second is `idempotencyKey`. Key what you write by it and a replaying worker +updates what it wrote before rather than recording the same work twice. + +Both are independent of OpenTelemetry, so a backend that wants neither singletons +nor a tracer provider still gets what it needs: one file beside the others, +returning one provider, reading the same as any of them. Backends like Braintrust +build both of these themselves today, working around what `defineInstrumentation` +exposes. Below is a strawman for what that integration looks like on the bus +instead. + +```typescript +// agent/instrumentation/braintrust.ts +import { braintrust } from "braintrust/eve"; + +export default braintrust({ metadata: { app: "support-agent" } }); +``` + +```typescript +// braintrust/eve — the vendor side, importing eve's types and nothing else. +import type { InstrumentationProvider } from "eve/instrumentation"; + +export function braintrust(options: BraintrustOptions = {}): InstrumentationProvider { + let logger: Logger; + + return { + setup({ agentName, environment, evaluation }) { + logger = initLogger({ + projectName: agentName, + apiKey: process.env.BRAINTRUST_API_KEY, + metadata: { ...options.metadata, environment, ...evaluation }, + }); + }, + + flush: () => logger.flush(), + + events: { + "step.attempt.started"({ idempotencyKey, step, model, input }, ctx) { + ctx.state.set( + logger + .startSpan({ + id: idempotencyKey, + name: "eve.step", + type: "llm", + input: input?.messages, + metadata: { model: model.id, provider: model.provider, ...step }, + }) + .export(), + ); + }, + "step.attempt.completed"({ output, usage }, ctx) { + const span = logger.resumeSpan(ctx.state.get() as string); + span.log({ output: output?.messages, metrics: metrics(usage) }); + span.end(); + }, + "step.attempt.failed"({ error }, ctx) { + const span = logger.resumeSpan(ctx.state.get() as string); + span.log({ error }); + span.end(); + }, + + "action.started": ({ idempotencyKey, kind, name, input }, ctx) => /* … */, + "action.completed": ({ output }, ctx) => /* … */, + "action.failed": ({ error }, ctx) => /* … */, + }, + }; +} +``` + +Redaction is a span processor. eve puts one in front of each OpenTelemetry +destination, and `recordInputs` and `recordOutputs` say what it strips. The bus +has no equivalent: a handler is handed the prompts and the tool payloads as they +are. A provider that records them offers its own switches. + +## Migration + +`agent/instrumentation.ts` is the break. eve fails the build when it finds one +and names the new layout in the error, because the fix is mechanical. The `setup` +callback in the **Today** example becomes the `otelIntegration` in the +**Proposed** one, moved to `instrumentation/.ts`; the `registerOTel` +call and the `serviceName` go away, because eve makes that call now. `functionId` +and `traceChannelRequests` move to `instrumentation/otel.ts`, a file you add only +if you had one of them. `recordInputs` and `recordOutputs` move onto the +destination, where they describe that destination rather than the agent. + +Every one of those could have been read at the old location and forwarded. The +`setup` callback could not, and it is the reason for the break. A `setup` that +calls `registerOTel` is the collision: eve registers too, and one of the two +silently loses. Accepting the old file keeps local tracing off for exactly the +agents meant to get it back. + +A shim would be worse. eve could intercept `registerOTel` and fold what it was +handed into its own pipeline, but only for an author who went through +`@vercel/otel`. Anyone who builds a tracer provider and sets it global goes +around it, and the failure returns as something silent that depends on which +import you reached for. + +`runtimeContext` is the one field the error cannot name a replacement for, +because there is not one yet. Everything else is a file move. The integrations +eve ships are one edit each and go out with the change, and an agent that wrote +its own `instrumentation.ts` gets a build error the first time it compiles +against the new version — which is where a break like this should land.