Skip to content

[MBA-3823] Coexistence without drift: single writer, legacy write denial, outbox relay, trigger neutralization, reconciliation - #96

Merged
devin-ai-integration[bot] merged 9 commits into
estate/legacy-baselinefrom
devin/mba-3823-single-writer-outbox-v2
Nov 11, 2026
Merged

devin-ai-integration[bot] merged 9 commits into
estate/legacy-baselinefrom
devin/mba-3823-single-writer-outbox-v2

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Summary

MBA-3823 (M4): scheduling-service becomes the only writer of bookings and seat counts. Every modern mutation writes an outbox_event row in the same transaction, and an idempotent relay (running as ad_sched_relay) projects it into AssessmentDelivery. Legacy logins can no longer write booking or seat tables. After the M1 mixed traffic goes through the routing gateway (:8000), reconciliation shows 0 mismatches and db/checks/seat-drift.sql returns 0 rows. On the legacy baseline the same report shows 2 mismatches and 2 drift rows.

Deliberate legacy behaviour changes (all cited in ADR 0003):

  • Legacy write denial, db/security/040_single_writer.sql. Covers HB-01, HB-02, HB-28, HB-29, MR-01, MR-02 and MR-07. Every legacy login gets DENY INSERT/UPDATE/DELETE on Booking/Waitlist, DENY UPDATE on ExamSession(Capacity, SeatsAvailable, AccommodationCapacity), and DENY EXECUTE on usp_BookSeat/CancelBooking/ChangeCapacity/RescheduleBooking. Only ad_sched_relay can write.
  • Per-component logins (036_component_logins.sql): ad_portal, ad_booking_api, ad_results, ad_capacity_sync, ad_sched_relay, plus the existing ad_sched_import. These replace the shared ad_app. The local-dev passwords follow the convention 035_sched_import_login.sql already uses. Azure/Release configs contain placeholders only.
  • Trigger neutralization for relayed writes only. Covers HB-01, HB-04, HB-05, HB-16, MR-01 and MR-14. 032_trg_Booking_SeatSync and 033_trg_ExamSession_Update return early when SESSION_CONTEXT(N'writer') = N'scheduling-relay'. 033 still stamps LastModified.
  • capacity-sync disabled (HB-03, MR-17): capacity.sync.enabled=false. runSync only writes its CAPACITY_SYNC AuditLog row, marked SKIPPED. Reconciliation takes over its role.
  • results-service no longer sets Booking.Status='ATTENDED' itself. It calls POST /api/v2/bookings/{id}/attendance on scheduling-service (:8090 directly), and the relay projects the result back to legacy.
  • booking-api keeps the v1 contract. The gateway sends v1 writes to scheduling-service for every centre (routing.modern-centers=*). A write that hits :8081 directly is rejected by SQL (permission error) and mapped to 403 LEGACY_WRITE_DENIED.

How the pieces work:

SchedulingService.book/cancel/reschedule/waitlist/recordAttendance/changeCapacity
  @Transactional { mutate booking/waitlist_entry/exam_session/capacity_change; outbox.append(type, aggregate, snapshot) }
RelayWorker (poll 250ms) -> LegacyRelay.apply(event) in one legacy tx as ad_sched_relay:
  sp_set_session_context 'writer','scheduling-relay'
  if exists RelayCheckpoint(EventId) -> no-op
  upsert ExamSession (Capacity/SeatsAvailable/AccommodationCapacity), Booking (SET IDENTITY_INSERT, legacy id = modern id >= 50000000),
         Waitlist, AuditLog; insert RelayCheckpoint; write booking.legacy_booking_id back
GET /internal/reconciliation | java -jar scheduling-service.jar reconcile -> reports/reconciliation-<ts>.md (+ seat-drift.sql)
GET /internal/relay -> {total,pending,lastSeq,lastRelayedSeq,oldestPendingAt}; gauges scheduling.outbox.pending / .lag
  • V2__outbox.sql adds outbox_event (seq BIGINT IDENTITY, unique event_id) and booking.legacy_booking_id.
  • Event types: BookingCreated, BookingCancelled, BookingRescheduled, BookingWaitlisted, AttendanceRecorded, CapacityChanged, plus InventoryBaselined. The importer emits InventoryBaselined once after a snapshot import, so the first relay pass aligns legacy counters with the modern derived values. Without it, pre-existing legacy drift would carry over.
  • Routing gateway: routing.coexistence-mode=relay and routing.modern-centers=*. The * wildcard is accepted only in relay mode; with none, the startup guard still refuses it. X-Route and the floor of 50000000 are unchanged.
  • Attendance needs a service credential. AttendanceTokenGuard compares header X-Service-Token with scheduling.attendance.service-token (env SCHEDULING_ATTENDANCE_SERVICE_TOKEN) using MessageDigest.isEqual, before recordAttendance. A missing or wrong token gets 401 {"error":"UNAUTHORIZED",...} and no outbox event. If the property is unset, every call is refused (fail closed). results-service sends scheduling.service-token (env SCHEDULING_SERVICE_TOKEN). run-scheduling-service.* and run-legacy.* set the same dev-only value; Azure takes the env var. The gateway answers 404 for /api/v2/bookings/*/attendance on :8000. The path is normalised first: decoded, ;params stripped, // collapsed, lower-cased. The other v2 writes stay unauthenticated, matching the legacy no-auth model (MR-16).
  • OpenAPI: attendance has a serviceToken apiKey security scheme and a 401 that uses a separate ServiceTokenError schema. Other attendance errors reuse NOT_CANCELLABLE. This keeps the shared ApiError enum unchanged, so OpenApiCompatibilityTest against the published baseline still passes. api/openapi.yaml was regenerated as described in api/README.md.
  • scripts/mixed-traffic.ps1: behind the gateway, the portal /Booking/Book form redirects to the modern UI. The script now finishes that booking (and its cancel) the way /app does, through /api/v2/bookings. Direct legacy runs behave as before.

Evidence

  • After (through :8000): reports/reconciliation-20261111-205430.md shows 462 sessions and 3,627 bookings with 0 mismatches; reports/seat-drift-m4-relay.txt has 0 rows.
  • Before (legacy-baseline, direct): reports/reconciliation-20261111-205122-legacy-baseline.md shows 2 session mismatches; reports/seat-drift-legacy-baseline.txt has 2 rows (session 155 drift 7, session 158 drift 4).
  • Mixed traffic via :8000: portal 20 ok / 0 failed, partner 20 ok / 0 failed, 3 and 3 cancels, seat-drift 0 rows. Outbox: 47 events, 0 pending.
  • Relay lag, 20 bookings via :8000, until the dbo.Booking row is visible (includes sqlcmd polling): p50 0.28 s, p95 0.29 s, max 0.30 s.
  • Results flow and attendance auth, live on Windows (reports/attendance-auth-live-check.txt):
    • POST :8000/api/v1/results returned 201. The booking became ATTENDED in Scheduling and in legacy, with AuditLog Booking|ATTENDED|scheduling-service.
    • Attendance on :8090 returned 401 with no token and 401 with a wrong token. On :8000 it returned 404 both without a token and with a valid one.
    • No AttendanceRecorded event was written before the result. Afterwards reconciliation shows 462 sessions, 0 mismatches, 0 seat drift, and the relay has 0 pending.
  • A direct legacy write, POST :8081/api/v1/bookings, returns 403 LEGACY_WRITE_DENIED.

Tests (Windows Server 2022 VM; Docker was available, so the Testcontainers ITs ran too)

module command result
scheduling-service mvn -B clean verify (JDK 21, Maven 3.9.9) 57 unit + 55 IT, 0 failures, BUILD SUCCESS (JaCoCo gate passes). Test contexts run with the relay off (src/test/resources/application.properties), so the suite never writes to a local AssessmentDelivery.
routing-gateway mvn -B verify (JDK 21) 28 tests, 0 failures, BUILD SUCCESS
booking-api, results-service, capacity-sync services/mvnw.cmd -B -pl booking-api,results-service,capacity-sync verify (JDK 8) 5 + 24 + 2 tests, 0 failures, BUILD SUCCESS

New coverage:

  • LegacySingleWriterIT (8): each legacy login gets a permission error on Booking INSERT and on SeatsAvailable UPDATE, procedure EXECUTE is denied, and the relay login can write. It runs against local SQL Express via LegacyEstateDb, or skips.
  • RelayIT (4): book, cancel, capacity increase and decrease, then reconciliation is exact. Every event is replayed twice with no duplicate Booking, Waitlist or AuditLog rows. InventoryBaselined brings drift to 0. Attendance is relayed.
  • AttendanceViaSchedulingTest, AttendanceRecorderTest, CapacitySyncServiceTest (skip path), ApiExceptionHandlerTest.
  • New routing wildcard and relay-guard tests.
  • Attendance auth: AttendanceTokenGuardTest (valid, missing, wrong, blank, unset). SchedulingApiIT checks 401 without a token and with a wrong token, the booking still BOOKED, no outbox event, then 200 ATTENDED with exactly one event. OpenApiSpecIT checks the security scheme. AttendanceRecorderTest checks the header is sent and a 401 maps to 502. The results-service ITs use a SchedulingStub that requires the token. InternalPathTest and GatewayIntegrationTest check the gateway 404.
  • ReconcileCommandTest: reconcile forces --scheduling.relay.enabled=false --scheduling.import.on-startup=false as command-line args, so it stays read-only.

UI/E2E (testing agent, browser through :8000, freshly reseeded estate): passed. Recording and screenshots are in the PR comment.

Open / notes

  • Jira is read-only for this run (requester's rule), so the PR link was not posted to the ticket.

  • booking-api is not modified to proxy through HTTP. Its v1 writes reach scheduling-service via the gateway, and its DAO writes are now denied at the SQL level (see ADR 0003).

  • The attendance endpoint previously had no authentication (Devin Review escalation). Per the coordinator's decision it now requires X-Service-Token, and the gateway does not expose it (see above).

  • Cutover and rollback order is in runbook §6: re-import before *, and hand attendance and modern-owned ids back before stopping scheduling-service.

Devin-Org: engineering

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/790bcbf347474e669d759bbef4b2fe6f
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/790bcbf347474e669d759bbef4b2fe6f?variant=devin
Requested by: @mbatchelor81


Note

Devin errored when opening this Pull Request as mbatchelor81.
As a fallback, Devin opened this PR as itself.


🟢 Devin Review found nothing to fix · View findings · Follow Devin's work

Devin Review

…ox relay, trigger neutralization, reconciliation

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration Bot and others added 3 commits November 11, 2026 21:01
Co-Authored-By: Mason <masonbatchelor81@gmail.com>
…step before commit

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
…y writes

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

devin-ai-integration Bot and others added 2 commits November 11, 2026 21:09
…r write a local AssessmentDelivery

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
…oking ids before stopping scheduling-service

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

UI/E2E test (testing agent, Windows VM): passed. Tested in the browser through the gateway :8000 against SQL Server Express, starting from a freshly reseeded estate (head 2187657; the commits since the runtime build change only tests and the runbook).

Modern booking (:8000/app/) Legacy projection (/Booking/MyBookings)
Modern booking Legacy booking
  • Booking 50000001 (NW00000901, session 154) appears in legacy MyBookings as BOOKED, on both :8000 and :8080. Seats went 11 → 10 exactly, with no extra trigger decrement.
  • ET25 booking 50000002: the accommodation code is kept in legacy, and the cancellation is projected as CANCELLED. Seats went 10 → 9 → 10.
  • Full session 153: booking 50000003 is WAITLISTED at position 4 and the legacy waitlist went 3 → 4.
  • Modern capacity change on session 153, 22 → 23: legacy SeatsAvailable went 0 → 1. Booked 22 and waitlist 4 are unchanged; capacity increases don't promote waitlisted rows (ADR 0002).
  • API checks:
    • The gateway's v1 candidate bookings call returns 50000001 (X-Route: legacy).
    • The legacy form POST returns 303 with X-Route: modern.
    • A direct :8081 POST returns 403 LEGACY_WRITE_DENIED.
  • Final GET /internal/reconciliation: 462 sessions and 3,590 bookings, with 0 session mismatches, 0 booking mismatches and empty seatDrift. Relay: 6 events relayed, 0 pending.

Final reconciliation

E2E recording (animated)

Not covered by the UI run (they are covered by ITs or the scripts in the description instead):

  • killing the relay mid-batch (RelayIT replays every event twice);
  • concurrent booking races;
  • an instrumented 2 s lag (the script sample gave p95 ≈ 0.29 s).

Pre-existing behaviour (M3): :8000/app without the trailing slash redirects to :8090/app/.

Note on an earlier failed run. That run showed 9 mismatches on sessions 154, 155, 163, 263 and 300. The cause was in the tests, not the runtime: the scheduling-service Spring ITs started with the relay on, and by default it pointed at localhost:1433/AssessmentDelivery. Commit 01344c2 makes the relay default to off in test contexts, so only RelayIT turns it on, against its own database. After that commit, mvnw verify left the local AssessmentDelivery untouched (same MAX(LastModified), Booking count and AuditId).

devin-ai-integration Bot and others added 3 commits November 11, 2026 21:30
…e it before returning to modern

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
scheduling-service checks X-Service-Token in constant time against
scheduling.attendance.service-token (fail closed when unset, 401 with the
usual error body, no outbox event). results-service sends
scheduling.service-token; run scripts set the same dev value. The gateway
answers 404 for /api/v2/bookings/*/attendance. OpenAPI: serviceToken
security scheme and a separate ServiceTokenError 401 schema so ApiError
stays compatible with the published baseline.

Co-Authored-By: Mason <masonbatchelor81@gmail.com>
Co-Authored-By: Mason <masonbatchelor81@gmail.com>
@devin-ai-integration
devin-ai-integration Bot merged commit b88cdaf into estate/legacy-baseline Nov 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants