-
Notifications
You must be signed in to change notification settings - Fork 0
docs: map orchestration evidence boundary #94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
LaloLalo1999
wants to merge
1
commit into
main
Choose a base branch
from
cx/runtime-evidence-portfolio
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Portfolio integration contract | ||
|
|
||
| This document maps Code Colony into the delivery portfolio without treating the seeded simulation | ||
| as live orchestration evidence. It does not define the root contract, and no Ultimate Harness or | ||
| Telar adapter is shipped here. | ||
|
|
||
| ## Role and maturity | ||
|
|
||
| | Classification | Current status | | ||
| |---|---| | ||
| | Primary role | Orchestration experiment and operator-experience prototype | | ||
| | Real surfaces | Application shells, domain types, selected Convex schema/components, Mastra triage agent, infrastructure, and PostHog boundary | | ||
| | Simulated surfaces | OMP runtime execution, diffs/checks/cost ledger, artifact review, memory/skill promotion, presence/votes, and marketplace behavior | | ||
| | Current evidence status | Not applicable for production runtime or benchmark claims | | ||
| | Portfolio integration | Proposed; no Ultimate Harness or Telar adapter is shipped here | | ||
|
|
||
| ## Evidence boundary | ||
|
|
||
| | Category | Current system of record | Allowed claim | | ||
| |---|---|---| | ||
| | Product behavior | Source, tests, built app, and exercised rendered path | The verified prototype interaction only | | ||
| | Seeded orchestration | `src/domain/types.ts` and seed transitions | UI/domain simulation, never a completed external run | | ||
| | Convex state | Owning schema and wrapper functions | Implemented backend structure; not proof that every frontend flow is wired | | ||
| | Runtime/model receipt | None for the simulated OMP run | `unavailable` | | ||
| | Quality/cost/latency/tokens | Mock values where labeled; real telemetry only when a live path emits it | Never mix mock and observed fields in one receipt | | ||
| | Blind review | Not implemented | No independent/blind-quality claim | | ||
|
|
||
| PostHog is the planned/implemented telemetry seam for real product and runtime events, but the | ||
| existence of an instrumentation module does not prove an event was emitted or received. | ||
|
|
||
| ## Portfolio seam | ||
|
|
||
| 1. Code Colony owns the simulation and operator experience. It must keep mock data visibly labeled. | ||
| 2. A future live runtime integration consumes a small adapter interface and returns immutable run, | ||
| artifact, and verification references. It must not leak provider internals into UI state. | ||
| 3. Ultimate Harness may later remain the only Run Control for externally executed attempts. Code | ||
| Colony visualizes its receipts; it does not start a second controller for the same attempt. | ||
| 4. The proposed Telar plane may own planning/governance and normalized evidence references. No | ||
| current Code Colony screen is proof that Telar behavior exists. | ||
|
|
||
| ## Future plan | ||
|
|
||
| - Replace the simulated runtime card only after a real adapter produces attempt lineage, resolved | ||
| harness/model, repository/config hashes, artifact hashes, checks, cost, latency, tokens, and | ||
| explicit missing-field reasons. | ||
| - Keep independent review separate from implementation and retain judge/reviewer identity, | ||
| evidence view, revision, verdict, and limitations. | ||
| - Prove one end-to-end spec-to-diff path before exposing portfolio rollups or learning promotion. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When this evidence map is used to audit the simulated run,
src/domain/types.tsprovides only type definitions; the actual run, event, artifact, reviewer, and cost values are declared insrc/domain/demoData.ts, then replayed bysrc/domain/runtime.ts. Calling the types and unnamed “seed transitions” the system of record omits the data that substantiates the seeded orchestration claims, so this row should explicitly identifysrc/domain/demoData.tsand the runtime replay boundary.Useful? React with 👍 / 👎.