feat: drop newest on full queue, add configurable MaxQueueSize - #258
Conversation
Enqueue now drops the newest event and returns ErrQueueFull (invoking Callback.Failure) when the in-memory queue is full, instead of blocking the caller. This matches posthog-python and posthog-rs. - add Config.MaxQueueSize (default 10000), clamped up to BatchSize, replacing the hardcoded BatchSize*10 msgs channel sizing - change default BatchSize from 250 to 100 for cross-SDK consistency - document the drop-vs-block tradeoff for bulk/backfill callers on Enqueue and MaxQueueSize - benchmarks report a drop% metric instead of failing on overflow and discard logs to keep ns/op clean
posthog-go Compliance ReportDate: 2026-07-15 01:33:30 UTC ✅ All Tests Passed!111/111 tests passed Capture_V1 Tests✅ 94/94 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
Regenerated api/public-api.txt via bin/api-diff --update to reflect the new Config.MaxQueueSize field, DefaultMaxQueueSize/ErrQueueFull additions, and the DefaultBatchSize 250->100 change.
|
Addresses PR #258 review: the Enqueue drop-on-full branch invoked Callback.Failure and Warnf synchronously on the caller's goroutine. - Callbacks are contractually run from the client's internal goroutines; firing one from Enqueue could re-enter or block the caller's request path. Drop the notifyFailure call; the caller learns via the returned ErrQueueFull. This also makes the closed-channel race drop (ErrClosed, no callback) and the full-queue drop consistent. - Remove the per-event Warnf: under sustained overload it re-created the caller latency the non-blocking drop path exists to avoid. Update the white-box test to assert the drop fires no callback and logs nothing on the caller goroutine, and refresh the now-stale benchmark log-silencing comment (the flood came from stage-2 sendBatch backpressure logging, not the drop path).
|
btw take a look at PostHog/posthog-python#146 |
|
i think that dropping the newest event makes sense, given we have a chance at call time to notify the caller that the queue is full. otherwise, it'd be a silent side-effect. |
|
regarding PostHog/posthog-python#146 see my response here PostHog/posthog-python#731 (review) |
💡 Motivation and Context
When the in-memory queue filled up, Go's
Enqueueblocked the caller until space freed up, whereas posthog-python and posthog-rs drop and report. Blocking a request path on a full analytics queue turns a slow uploader or burst into caller latency. This switches Go to non-blocking reject-newest and makes the queue size a real, configurable knob.Companion to the posthog-js PR that raised the Node
maxQueueSizedefault and drop-log severity.💚 How did you test it?
queue_overflow_test.go: drop-on-full is non-blocking, returnsErrQueueFull, buffers nothing, and fires no callback/log on the caller goroutine; happy path still buffers.MaxQueueSizedefault/override/clamp/negative-rejection;BatchSizedefault now 100.go test -race ./...green.📝 Checklist
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Enqueueoverflow path is now a non-blockingselect/defaultthat drops the newest message and returnsErrQueueFull. The drop is reported only via the returned error — no callback or log on the caller's goroutine (per PR review).Config.MaxQueueSize(default 10000), clamped up toBatchSize, replacing the hardcodedBatchSize * 10sizing. ChangedDefaultBatchSize250 → 100 for cross-SDK consistency.