EngagePulse is one Go program. When it runs it:
- Accepts player events over HTTP
- Saves them in Postgres
- Puts them on a message stream (Kafka; locally we use Redpanda)
- Reads those messages and runs the three rules
- Lets you read the player result over HTTP or gRPC
If VIP / welcome bonus / velocity are new words, read concepts.md first.
The useful parts are:
- safe accept of events
- message stream
- rules
- bonus credits that do not double-pay
We keep that in one binary so the demo stays easy to run. Later you could split “API” and “worker” into two processes if needed.
loadgen / partner
|
v
POST /v1/events
- check JSON (`schema_version` omitted or 1; unknown versions 400)
- check brand secret (HMAC signature, optional X-Key-Id)
- save row in outbox table (response `X-Trace-Id` is the OpenTelemetry trace)
- reply 202 Accepted
|
v
Outbox publisher (background)
- take pending rows
- send to Kafka topic player.events
- mark row published
|
v
Kafka consumer (background)
- read message
- run worker
- if worker fails: try again up to 3 times
- if still failing: send copy to player.events.dlq, then move on
|
v
Worker (one DB transaction)
- remember event id (skip if seen)
- run welcome → VIP → velocity rules
- maybe add bonus credit
- commit mark + state + credit together
|
v
GET /v1/players/{id} or gRPC GetPlayer
- needs brand API key
- returns VIP, tags, flag, balance
Kafka is a log of messages.
- Writers append events.
- Readers process them in order (per group).
- If the worker restarts, it can continue where it left off.
Locally, Redpanda is a smaller tool that speaks the same Kafka style of API.
Problem: if we send to Kafka and the app dies, it is unclear whether the event was accepted.
What we do:
- Save the event in an
outboxtable first. - Only then reply
202. - A background loop sends pending rows to Kafka.
- Same
event_idcannot be inserted twice for the same brand.
If the app crashes while sending, rows stuck as “publishing” are put back to “pending” and tried again. A store test leaves a row in publishing, reclaims it, then flushes: Kafka sees the event once.
The gauge engagepulse_outbox_pending shows how many rows are still waiting (or mid-send).
The demo script waits until that gauge is 0 and processed events catch up to accepted ones before printing player JSON.
Code: internal/store/outbox.go, internal/outbox/publisher.go.
Problem: sometimes processing fails (DB blip, bad data, bug).
What we do:
- Try the same event up to 3 times.
- If it still fails, write a “dead letter” message to
player.events.dlqwith:- the original event
- the error text
- how many tries we used
- Only then mark the original Kafka message as done.
- If even the DLQ write fails, we do not mark it done, so Kafka can deliver it again.
Local demos can turn on FAIL_INJECT=true. Then an event id that starts with poison- fails after mark, retries three times, and lands on player.events.dlq. Default ingest is unchanged. Leave this off on the public API.
GET /v1/dlq (API key) lists the last N dead letters for that brand. Redrive is a new signed POST /v1/events of the same event_id — not an admin credit. Arena’s Architecture page shows the list; a human has to click Redrive. The ledger unique key still blocks a second bonus.
Code: internal/kafka/consumer.go, internal/kafka/dlq.go, internal/kafka/inspect.go.
The stream may deliver the same event more than once.
The bonus table has a unique key on (tenant_id, event_id).
A second credit for the same event is rejected, so the balance does not jump by another 100.
Problem: if we mark an event “processed” and then die before saving player state or the bonus credit, a retry sees the mark and skips the rest forever.
What we do:
- Open one Postgres transaction.
- Ensure player rows, mark processed, update state, and credit the ledger inside that transaction.
- Commit only when all of those steps succeed.
- If anything fails, roll back — including the processed mark — so Kafka retries can run the full path again.
Code: internal/worker/worker.go, internal/store *Tx helpers, internal/ledger.CreditTx.
Always:
- Welcome bonus (first deposit)
- VIP score (bets)
- Velocity flag (many bets quickly)
Details: concepts.md. Code: internal/rules.
| Call | How you prove who you are |
|---|---|
POST /v1/events |
Header X-Signature (HMAC of the raw body); optional X-Key-Id during key rotation |
POST /v1/hmac/rotate |
Header X-API-Key. Overlap window for that brand only. Response has key ids, not secrets. |
POST /v1/tools/{name} |
Header X-API-Key; ingest also X-Signature of the event body |
GET /v1/players/{id} |
Header X-API-Key |
GET /v1/dlq |
Header X-API-Key; last N dead letters for that brand |
gRPC GetPlayer |
metadata x-api-key |
Unknown brand and bad signature both return 401 (same answer on purpose). A player id that exists for another brand returns 404 for your API key (no cross-brand leak).
Demo brands and keys are only for local use (see .env.example).
Several test packages talk to the same Postgres.
Migrate takes a database lock so they do not fight while creating tables.
CI also runs tests one package at a time (go test ./... -p 1).
Useful coverage for this path:
- worker rollback if processing fails after the mark (
internal/worker) - bad HMAC / unknown tenant → 401, cross-tenant player read → 404 (
internal/api/httpapi) - last N DLQ inspect + signed redrive credit-once (
internal/kafka,internal/api/httpapi) - ledger credit-once under concurrent delivery (
internal/ledger)
- Game UI / website for players
- Real money payments
- Auto-redrive of DLQ messages (inspect is read; redrive is a human click through HMAC ingest)
- Machine-learning “personal offers”