Skip to content

feat: IPC for the Driver-/Unit framework (SHM, Event Bus, Message Queue) - #3

Merged
dkruempe merged 35 commits into
masterfrom
feat/ipc-driver-unit-framework
Oct 21, 2026
Merged

dkruempe merged 35 commits into
masterfrom
feat/ipc-driver-unit-framework

Conversation

@dkruempe

Copy link
Copy Markdown
Owner

Summary

Makes the IPC layer (Shared Memory, Event Bus, Message Queue) actually usable for the planned Driver-/Unit framework. Implements the full plan in docs/ipc-driver-unit-plan.md — all phases A, B and C.

Plan: docs/ipc-driver-unit-plan.md (tracking tables, all rows marked done)
Potentials source: docs/ipc-potencials.md

What changed

Shared Memory

  • SHM-1 Typed SharedMemoryRepository<T> (object store) + ArrayShm / RingBufferShm (lock-free Vyukov MPMC) / QueueShm containers in shared memory, plus SharedMemoryRepositoryRegistry.
  • SHM-2 ShmView / ShmSlice generation-aware, span-like zero-copy views with staleness checks.
  • SHM-3 Segment lifecycle hooks (BeforeGrow / AfterGrow / BeforeShrink / AfterShrink / OnRemap).
  • SHM-4 SharedMemorySegmentStats (allocated/free bytes, grow/shrink/recovery counts, generation).
  • SHM-5 SharedMemoryCleanupService — scheduler-driven orphan cleanup with dry-run, PID liveness guard, DB reconciliation, idempotent.

Event Bus

  • EB-1 Schema id/version fields + priority on Event, EventSchemaRegistry.
  • EB-2 Sticky / last-value events (topic cache with TTL, delivered on subscribe).
  • EB-3 Priorities (Critical→Low, separate rings per subscriber+priority, FIFO within a level).
  • EB-4 Drop policies (Fail / DropOldest / DropNewest / Block), default Fail (backward compatible).
  • EB-5 Wildcard topic filters via TopicMatcher (*, ?), exact-match fast path.
  • EB-6 Queue depth observability (busQueueDepth, subscriberQueueDepth, drop counts).

Message Queue

  • MQ-1 Owner/liveness tracking (MessageQueueOwner), heartbeat, reclaimOrphans, schema owner columns, bootstrap reconciliation.
  • MQ-2 Typed Request/Reply API with correlation id, sync call(timeout), handler loop.
  • MQ-3 Explicit overflow / dead-letter policy (Fail / DropNewest / DropOldest / BlockWithTimeout / DeadLetter).
  • MQ-5 Ops endpoints: CLI + HTTP stats/depth/lag/owner/DLQ and guarded purge.
  • MQ-6 Optional SHM-backed queue (MessageQueueShmCore, lock-free ring) with MessageQueueCore API parity and backend flag.

Compatibility

  • Defaults unchanged (e.g. EB drop policy Fail), only additive changes; SHM/Event layouts extended without breaking existing fields.
  • Cross-process behaviour preserved; SHM lock-recovery reused for MQ orphan handling.

Tests

  • New suites: tests/MessageQueueRpcTest.cpp, tests/features/MessageQueueShmCoreTest.cpp, tests/features/SharedMemoryRepositoryTest.cpp, tests/features/ShmViewTest.cpp, tests/features/SharedMemoryCleanupServiceTest.cpp; extended tests/events/EventBusTest.cpp, CLI/HTTP/repository/plugin tests.
  • Local: Release, Non-Unity and Debug all build with -Werror; 1194/1194 unit tests pass; check-format / check-namespace / check-secrets pass.

Notes for reviewers

  • docs/ipc-potencials.md is the source document added alongside the plan.
  • Plan rows use [x] Erledigt; progress footer shows 7/7 · 6/6 · 3/3.

… (EB-3/5/6, EB-1/2/4)

Add IPC potentials source document.
… MQ-6)

- RequestReply envelope with correlation id, sync call(timeout), handler loop
- CorrelationId generator, timed receive primitive
- MessageQueueShmCore lock-free Vyukov ring in shared memory with API parity
- MessageQueueBackend flag + factory, example and tests
…(SHM-1, SHM-2)

- SharedMemoryRepository<T> object store with find_or_construct and registry
- ArrayShm/RingBufferShm (Vyukov MPMC)/QueueShm containers in shared memory
- ShmView/ShmSlice span-like views with generation/staleness checks
Scheduler-driven scan/apply with dry-run, PID liveness guard, DB reconciliation, idempotent.
…nd ops endpoints (MQ-1, MQ-3, MQ-5)

- MessageQueueOwner model + schema owner columns, heartbeat and reclaimOrphans
- Bootstrap plugin reconciliation of stale queues
- Overflow/dead-letter policy API on MessageQueueCore
- CLI stats/purge and HTTP stats/purge endpoints with DTOs
Comment thread src/features/base/msg/MessageQueueShmCore.cpp Fixed
Comment thread src/features/base/services/MessageQueueService.cpp Fixed
Comment thread src/include/lintel/core/shm/SharedMemoryRepository.h Fixed
The clang-tidy gate re-enables bugprone-easily-swappable-parameters and
performance-unnecessary-value-param for changed files, so the NOLINT
comments added with the IPC work no longer satisfy it. Replace them with
real named aggregates and strong types and pull those through interface,
mock, controller, service, CLI, examples and tests:

- MessageQueueQuery.h (new): MessageQueueFilter,
  MessageQueueOwnerRegistration, MessageQueueOwnerStateUpdate
- MessageQueueEntry.h: MessageQueueOwnerColumns
- EventBusEntry.h: EventBusCapacities
- MessageQueueRequestReply.h: RequestReplyQueues
- ShmView.h: shm::ShmGeneration for the view constructors
- SharedMemoryRepository.h: ShmCell::Layout and ShmCell::InitArgument
- RingBufferShm.h: byte-pointer buffer base, drops the int-to-ptr casts
- DropPolicy/EventPriority/MessageQueueBackend narrowed to std::uint8_t
- empty catches now log via LOG_WARN
Comment thread src/features/base/msg/MessageQueueShmCore.cpp Fixed
Comment thread src/features/base/services/SharedMemoryRepository.cpp Fixed
Comment thread src/include/lintel/core/shm/SharedMemoryRepository.h Fixed
CodeQL flags a std::byte * base scaled by bytes (cpp/suspicious-pointer-
scaling) when it is reinterpret_cast'ed from a wider object pointer, while
clang-tidy rejects the integer-to-pointer cast that avoids it. Use the
typed form where it exists and the project's documented integer-domain
form otherwise:

- MessageQueueShmCore::payloadOf advances SlotHeader * by one element
  (sizeof(SlotHeader) == kSlotHeaderSize), so it needs no cast at all
- slotAt and ShmCellRecord::payload keep the byte offset but materialise
  the address in the integer domain, same pattern as EventBus::ShmRing
CodeQL reports the std::apply lambda's parameter pack as hiding the
repository's own constructor-argument parameter.
template<typename... Args> static void construct(void *payload, ShmCell::InitArgument initArgument)
{
const auto &arguments = *static_cast<const std::tuple<std::decay_t<Args>...> *>(initArgument.value);
std::apply([payload](const auto &...ctorArgs) { ::new (payload) T(ctorArgs...); }, arguments);
@dkruempe
dkruempe merged commit 1333386 into master Oct 21, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants