Skip to content

feat(orchestrator): add conversation_events table, entity, and storage#1167

Merged
geoffjay merged 12 commits intomainfrom
issue-1159
Apr 18, 2026
Merged

feat(orchestrator): add conversation_events table, entity, and storage#1167
geoffjay merged 12 commits intomainfrom
issue-1159

Conversation

@geoffjay
Copy link
Copy Markdown
Owner

Adds the persistence foundation for agent conversation history — the conversation_events SQLite table, SeaORM entity, domain types, and four CRUD methods on AgentStorage.

Changes

  • Migration Remove duplicate request types in notify crate #16 (m20260417_000016_add_conversation_events.rs) — creates conversation_events table with two composite indexes (agent_id+created_at, agent_id+event_type); down() correctly drops indexes before the table
  • Entity (entity/conversation_event.rs) — SeaORM Model with 7 fields matching the schema; #![allow(dead_code)] suppresses expected unused-derive warnings until Persist agent stream events in orchestrator WebSocket handler #1160 wires it up
  • Domain types (types.rs) — ConversationEventType enum (8 variants, Display/FromStr), ConversationEvent struct with new() helper, ConversationQuery for filtering/pagination
  • Storage methods (storage.rs) — insert_conversation_event, list_conversation_events (with type/time/pagination filters), count_conversation_events, delete_conversation_events_for_agent; plus model_to_conversation_event helper

Test plan

  • 7 new unit tests in storage::tests: insert+count, list all, filter by type, limit+offset, delete (cross-agent isolation), metadata JSON roundtrip, all 8 event-type variants
  • cargo test -p orchestrator — 413 tests pass (pre-existing linear_fetch_http network tests ignored)
  • cargo fmt --check -p orchestrator — clean
  • cargo clippy -p orchestrator — no errors

Dependencies

Closes #1159

🤖 Generated with Claude Code

@geoffjay geoffjay added the review-agent Used to invoke a review by an agent tracking this label label Apr 17, 2026
Copy link
Copy Markdown
Owner Author

@geoffjay geoffjay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: feat(orchestrator): add conversation_events table, entity, and storage

Reviewed against the acceptance criteria in #1159. The migration, entity, domain types, and storage structure are all correct. Two issues must be fixed before merge, plus one strong recommendation.


🔴 Fix required — silent metadata loss in insert_conversation_event

storage.rs, insert_conversation_event:

// Current
metadata: Set(event
    .metadata
    .as_ref()
    .map(|v| serde_json::to_string(v).unwrap_or_default())),

unwrap_or_default() silently drops the metadata field if serde_json::to_string fails (e.g. a Value containing f64::NAN). The read path in model_to_conversation_event correctly propagates errors with .transpose()?. The write path should match:

// Fix
let metadata_str = event
    .metadata
    .as_ref()
    .map(serde_json::to_string)
    .transpose()
    .map_err(|e| anyhow::anyhow!("Failed to serialize event metadata: {}", e))?;

// In the ActiveModel:
metadata: Set(metadata_str),

This is consistent with anyhow-based error propagation used everywhere else in AgentStorage.


🔴 Fix required — since/until filter path is untested

The PR adds since and until to ConversationQuery and wires them into list_conversation_events, but no test exercises that code path. The 7 tests cover insert/count, list-all, type filtering, limit+offset, delete isolation, metadata roundtrip, and all 8 event type variants — but not time-range filtering.

Add a test:

#[tokio::test]
async fn test_conv_list_time_range() {
    let (storage, _tmp) = create_test_storage().await;
    let agent_id = Uuid::new_v4();

    storage
        .insert_conversation_event(&make_event(agent_id, ConversationEventType::Output))
        .await
        .unwrap();

    let boundary = Utc::now();

    storage
        .insert_conversation_event(&make_event(agent_id, ConversationEventType::Output))
        .await
        .unwrap();
    storage
        .insert_conversation_event(&make_event(agent_id, ConversationEventType::Output))
        .await
        .unwrap();

    // Only events after boundary
    let opts = ConversationQuery { since: Some(boundary), ..Default::default() };
    let events = storage.list_conversation_events(agent_id, &opts).await.unwrap();
    assert_eq!(events.len(), 2);

    // Only events before boundary
    let opts = ConversationQuery { until: Some(boundary), ..Default::default() };
    let events = storage.list_conversation_events(agent_id, &opts).await.unwrap();
    assert_eq!(events.len(), 1);
}

🟡 Recommendation — add session_number filter to ConversationQuery

The REST API (#1161) and UI (#1162) will need to replay a single session (e.g. "all events from session 3"). ConversationQuery is the right place for this and the fix is trivial while it's still a new type:

// types.rs
pub session_number: Option<i64>,
// storage.rs list_conversation_events
if let Some(sn) = opts.session_number {
    condition = condition.add(conv_entity::Column::SessionNumber.eq(sn));
}

Adding this after #1161 lands would require a ConversationQuery change that forces updates in all callers. Cheap to do now.


✅ What's correct

  • Migration uses if_not_exists(), creates both composite indexes, and down() correctly drops indexes before the table — idempotent and reversible
  • #[allow(dead_code)] on the entity file with a clear follow-up comment is the right pattern
  • model_to_conversation_event uses ? throughout — correct
  • test_conv_event_type_roundtrip verifies all 8 variants survive a DB round-trip — thorough
  • test_conv_delete_for_agent correctly verifies cross-agent isolation
  • api.rs and client.rs changes are cargo-fmt reformats only, no logic changes — acceptable
  • session_number correctly typed as i64 / big_integer throughout

@geoffjay
Copy link
Copy Markdown
Owner Author

geoffjay commented Apr 18, 2026

This change is part of the following stack:

Change managed by git-spice.

@geoffjay geoffjay added the review-agent Used to invoke a review by an agent tracking this label label Apr 18, 2026
geoffjay and others added 12 commits April 17, 2026 22:44
- Fix silent metadata loss in insert_conversation_event: replace
  unwrap_or_default() with transpose()?.map_err(...) so serialization
  failures propagate as errors rather than writing empty strings
- Add test_conv_list_time_range covering since/until filter paths
- Add test_conv_list_filter_by_session covering session_number filter
- Add session_number: Option<i64> to ConversationQuery with DB-level
  filter in list_conversation_events (recommended while type is new)
- Add #[allow(dead_code)] to conversation types and storage methods
  consumed by stacked branches (#1160, #1161, #1163)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Address two blocking issues from PR review:

1. Session number off-by-one on reconnect - replace get_usage_stats().session_count
   (which is N+1 when active) with a new get_max_conversation_session_number()
   storage method that queries MAX(session_number) directly from
   conversation_events. This is always correct regardless of usage-session
   semantics.

2. Add persistence tests - 9 new tests covering: session counter init without
   storage, session counter removed on unregister, storage-backed session init
   reads MAX(session_number), persist_context_cleared updates counter and writes
   event, send_user_message persists prompt_sent + activity_changed,
   handle_incoming_message persists output/tool_use/thinking/result events.

3. Fix race condition in register(): async storage init now uses max(storage,
   current) to avoid overwriting a newer value set by persist_context_cleared
   between register() and the spawned lookup completing.

4. Add #[allow(dead_code)] to storage query helpers and ConversationQuery struct
   that are used only in test code (clippy -D warnings).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ation, cursor encoding

- Push session_number filter to DB in ConversationQuery/list_conversation_events
  so has_more/total reflect the filtered result set (removes in-memory retain)
- Replace get_conversation_summary full table scan with two SQL aggregation
  queries (GROUP BY event_type COUNT + COUNT DISTINCT session_number)
- URL-encode before/after cursor values in client.rs to handle RFC 3339
  timestamps with + offsets correctly
- Add #[allow(dead_code)] to methods used by stacked integration test branch
- Add urlencoding = "2.1" dependency to orchestrator Cargo.toml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… persistence

Adds tests/conversation_persistence.rs with 26 integration tests covering:
- Storage layer: since/until cursor filtering, combined time windows,
  ascending ordering, session isolation, cross-agent delete isolation,
  all-event-type round-trips, and metadata round-trips.
- REST API: GET /agents/{id}/conversation (events, limit/has_more,
  event_type filter, session filter, after/before cursors, 404, empty),
  GET summary, single-event retrieval with wrong-agent/unknown-id 404s,
  DELETE /agents/{id}/conversation (204, 404, cross-agent isolation,
  idempotent), and count accuracy.
Also adds DELETE /agents/{id}/conversation endpoint (204 No Content) to
api.rs and urlencoding dev-dependency to Cargo.toml.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eanup (address review feedback)

- Fix counter: .absolute(n) -> .increment(n) so the Prometheus counter
  accumulates across cleanup cycles instead of being reset each run
- Wire prune_excess_conversation_events into the periodic cleanup loop;
  now iterates all known agents and enforces max_events_per_agent each cycle
- Guard cleanup_interval_secs, retention_days, and max_events_per_agent
  against zero in RetentionConfig::from_env (zero interval panics
  tokio::interval; zero retention/max would delete all events silently)
- Fix Default impl: hardcode safe values instead of delegating to from_env(),
  so tests get predictable defaults regardless of environment variables
- Add DEFAULT_* constants to RetentionConfig for the hardcoded fallbacks
- Fix test_prune_excess_removes_oldest_first: use explicit timestamps
  spaced 1 s apart to prevent flakiness on fast machines
- Update doc comments: max_events_per_agent now documents periodic enforcement

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@geoffjay geoffjay merged commit f6f897b into main Apr 18, 2026
7 of 24 checks passed
@geoffjay geoffjay deleted the issue-1159 branch April 18, 2026 16:29
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 18, 2026

Codecov Report

❌ Patch coverage is 89.22784% with 113 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.33%. Comparing base (116a628) to head (b04ff00).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
crates/orchestrator/src/main.rs 0.00% 39 Missing ⚠️
...tes/orchestrator/tests/conversation_persistence.rs 94.29% 35 Missing ⚠️
crates/orchestrator/src/client.rs 0.00% 22 Missing ⚠️
crates/orchestrator/src/manager.rs 53.84% 6 Missing ⚠️
...ration/m20260417_000016_add_conversation_events.rs 89.47% 4 Missing ⚠️
crates/orchestrator/src/websocket.rs 96.62% 3 Missing ⚠️
crates/orchestrator/src/storage.rs 98.50% 2 Missing ⚠️
crates/orchestrator/src/types.rs 94.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1167      +/-   ##
==========================================
+ Coverage   58.94%   60.33%   +1.39%     
==========================================
  Files         226      228       +2     
  Lines       24467    25489    +1022     
==========================================
+ Hits        14422    15380     +958     
- Misses      10045    10109      +64     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rework PR has review feedback that must be addressed before merging review-agent Used to invoke a review by an agent tracking this label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add conversation_events table and SeaORM entity to orchestrator

1 participant