Skip to content

feat: drop newest on full queue, add configurable MaxQueueSize - #258

Merged
eli-r-ph merged 3 commits into
mainfrom
fix/enqueue-drop-newest-on-full
Jul 15, 2026
Merged

feat: drop newest on full queue, add configurable MaxQueueSize#258
eli-r-ph merged 3 commits into
mainfrom
fix/enqueue-drop-newest-on-full

Conversation

@eli-r-ph

@eli-r-ph eli-r-ph commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

When the in-memory queue filled up, Go's Enqueue blocked the caller until space freed up, whereas posthog-python and posthog-rs drop and report. Blocking a request path on a full analytics queue turns a slow uploader or burst into caller latency. This switches Go to non-blocking reject-newest and makes the queue size a real, configurable knob.

Companion to the posthog-js PR that raised the Node maxQueueSize default and drop-log severity.

💚 How did you test it?

  • New queue_overflow_test.go: drop-on-full is non-blocking, returns ErrQueueFull, buffers nothing, and fires no callback/log on the caller goroutine; happy path still buffers.
  • New config tests: MaxQueueSize default/override/clamp/negative-rejection; BatchSize default now 100.
  • Stress tests back to strict zero-drop / all-delivered (queue default is now 10000); full go test -race ./... green.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • Ran changeset to generate a changeset file.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Enqueue overflow path is now a non-blocking select/default that drops the newest message and returns ErrQueueFull. The drop is reported only via the returned error — no callback or log on the caller's goroutine (per PR review).
  • Added Config.MaxQueueSize (default 10000), clamped up to BatchSize, replacing the hardcoded BatchSize * 10 sizing. Changed DefaultBatchSize 250 → 100 for cross-SDK consistency.
  • The drop-on-full unit test is white-box (client built without starting the consumer loop, channel pre-filled) because stage-1 overflow isn't reachable deterministically through the public API.

Enqueue now drops the newest event and returns ErrQueueFull (invoking
Callback.Failure) when the in-memory queue is full, instead of blocking
the caller. This matches posthog-python and posthog-rs.

- add Config.MaxQueueSize (default 10000), clamped up to BatchSize, replacing
  the hardcoded BatchSize*10 msgs channel sizing
- change default BatchSize from 250 to 100 for cross-SDK consistency
- document the drop-vs-block tradeoff for bulk/backfill callers on Enqueue
  and MaxQueueSize
- benchmarks report a drop% metric instead of failing on overflow and discard
  logs to keep ns/op clean
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

posthog-go Compliance Report

Date: 2026-07-15 01:33:30 UTC
Duration: 168966ms

✅ All Tests Passed!

111/111 tests passed


Capture_V1 Tests

94/94 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint 17ms
Endpoint And Method.Does Not Use Legacy Endpoints 5ms
Required Headers.Has Authorization Bearer Header 4ms
Required Headers.Has Content Type Json 5ms
Required Headers.Has Posthog Sdk Info Format 4ms
Required Headers.Has Posthog Attempt Header 5ms
Required Headers.Has Posthog Request Id 4ms
Required Headers.Has Posthog Request Timestamp 4ms
Required Headers.Has User Agent 5ms
Body Format.Body Has Created At And Batch 5ms
Body Format.No Api Key In Body 4ms
Body Format.No Sent At In Body 5ms
Event Format.Event Has Required Root Fields 4ms
Event Format.Event Uuid Is Valid 4ms
Event Format.Event Timestamp Is Rfc3339 5ms
Event Format.Distinct Id Is String 4ms
Event Format.Distinct Id At Root Not Properties 5ms
Event Format.Custom Properties Preserved 4ms
Event Format.Set Properties Preserved 5ms
Event Format.Set Once Properties Preserved 7ms
Event Format.Groups Properties Preserved 5ms
Event Format.Sdk Generates Uuid If Not Provided 4ms
Event Format.Event Has Required Root Fields Batch 7ms
Event Format.Event Uuid Is Valid Batch 6ms
Event Format.Event Timestamp Is Rfc3339 Batch 6ms
Event Format.Distinct Id Is String Batch 7ms
Event Format.Distinct Id At Root Not Properties Batch 16ms
Event Format.Custom Properties Preserved Batch 7ms
Event Format.Set Properties Preserved Batch 16ms
Event Format.Set Once Properties Preserved Batch 7ms
Event Format.Groups Properties Preserved Batch 6ms
Event Format.Sdk Generates Uuid If Not Provided Batch 7ms
Batch Behavior.Multiple Events In Single Batch 28ms
Batch Behavior.Batch Envelope Smoke 7ms
Batch Behavior.Flush With No Events Sends Nothing 3ms
Batch Behavior.Flush At Triggers Batch 1006ms
Batch Behavior.Created At Reflects Batch Creation Time 5ms
Deduplication.Generates Unique Uuids 11ms
Deduplication.Different Events Same Content Different Uuids 6ms
Deduplication.Preserves Uuid On Retry 5132ms
Deduplication.Preserves Timestamp On Retry 5135ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5135ms
Deduplication.No Duplicate Events In Batch 30ms
Header Behavior On Retry.Attempt Header Starts At One 5ms
Header Behavior On Retry.Attempt Header Increments On Retry 10130ms
Header Behavior On Retry.Request Id Preserved On Retry 5133ms
Header Behavior On Retry.Different Requests Have Different Request Ids 2011ms
Header Behavior On Retry.Request Timestamp Changes On Retry 5135ms
Response Format Validation.Success Response Has Uuid Keyed Results 6ms
Response Format Validation.Success Response Has Ok For Each Event 26ms
Response Format Validation.Success No Retry After When All Ok 26ms
Response Format Validation.Success Retry After Present When Retry Events 128ms
Response Format Validation.Success No Retry After When Drop Only 26ms
Response Format Validation.Response Echoes Request Id 5ms
Retry Behavior.Retries On 408 5131ms
Retry Behavior.Retries On 500 5133ms
Retry Behavior.Retries On 503 5135ms
Retry Behavior.Retries On 504 5133ms
Retry Behavior.Retryable Errors Have Retry After 2131ms
Retry Behavior.Respects Retry After On Retryable Error 8137ms
Retry Behavior.Does Not Retry On 400 2131ms
Retry Behavior.Does Not Retry On 401 2132ms
Retry Behavior.Does Not Retry On 402 2131ms
Retry Behavior.Does Not Retry On 413 2131ms
Retry Behavior.Does Not Retry On 415 2132ms
Retry Behavior.Non Retryable Errors Have No Retry After 2131ms
Retry Behavior.Implements Backoff 15144ms
Retry Behavior.Max Retries Respected 15143ms
Partial Batch Handling.Handles 200 Full Success 2008ms
Partial Batch Handling.Handles 200 With All Ok 3031ms
Partial Batch Handling.Does Not Retry Dropped Events 3029ms
Partial Batch Handling.Does Not Retry Limited Events 3030ms
Partial Batch Handling.Prunes Ok Events On Partial Retry 5133ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry 5132ms
Partial Batch Handling.Retries Only Retry Events From Partial 5130ms
Partial Batch Handling.Partial Retry Preserves Uuids 5133ms
Partial Batch Handling.Partial Retry Attempt Header Increments 5133ms
Partial Batch Handling.Partial Retry Request Id Preserved 5134ms
Partial Batch Handling.Respects Retry After On Partial 5135ms
Partial Batch Handling.Unknown Result Treated As Terminal 3029ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry 3032ms
Compression.Sends Gzip Content Encoding 15ms
Compression.No Content Encoding When Disabled 15ms
Compression.Compressed Body Is Decompressible 14ms
Error Handling.Does Not Retry On Unknown 4Xx 2130ms
Event Options.Cookieless Mode Override 6ms
Event Options.Disable Skew Correction Override 4ms
Event Options.Process Person Profile Override 5ms
Event Options.Product Tour Id Override 4ms
Event Options.Unset Options Omitted 5ms
Event Options.Options Override In Batch 6ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties 6ms
Geoip And Historical Migration.Historical Migration Set In Body 5ms
Geoip And Historical Migration.Historical Migration Absent By Default 6ms

Feature_Flags Tests

17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 20ms
Request Payload.Flags Request Uses V2 Query Param 20ms
Request Payload.Flags Request Hits Flags Path Not Decide 19ms
Request Payload.Flags Request Omits Authorization Header 14ms
Request Payload.Token In Flags Body Matches Init 14ms
Request Payload.Groups Round Trip 14ms
Request Payload.Groups Default To Empty Object 15ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 15ms
Request Payload.Disable Geoip Omitted Defaults To False 14ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 15ms
Request Lifecycle.No Flags Request On Init Alone 1ms
Request Lifecycle.No Flags Request On Normal Capture 5ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 27ms
Request Lifecycle.Mock Response Value Is Returned To Caller 14ms
Retry Behavior.Retries Flags On 502 15ms
Retry Behavior.Retries Flags On 504 16ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 15ms

@eli-r-ph eli-r-ph self-assigned this Jul 15, 2026
Regenerated api/public-api.txt via bin/api-diff --update to reflect the new
Config.MaxQueueSize field, DefaultMaxQueueSize/ErrQueueFull additions, and the
DefaultBatchSize 250->100 change.
@eli-r-ph
eli-r-ph marked this pull request as ready for review July 15, 2026 01:09
@eli-r-ph
eli-r-ph requested a review from a team as a code owner July 15, 2026 01:09
@greptile-apps

greptile-apps Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. posthog.go, line 574-576 (link)

    P2 Closed-Channel Drop Skips Callback

    If Close runs after the initial c.closed check but before sendPrepared writes to c.msgs, the closed-channel panic is recovered as ErrClosed and the prepared message is dropped without Callback.Failure. That differs from the new full-queue drop path and leaves shutdown-racing callers without the failure notification they now get for other dropped messages.

Reviews (1): Last reviewed commit: "chore: update public API snapshot for Ma..." | Re-trigger Greptile

Comment thread posthog.go Outdated
Comment thread posthog.go Outdated
Addresses PR #258 review: the Enqueue drop-on-full branch invoked
Callback.Failure and Warnf synchronously on the caller's goroutine.

- Callbacks are contractually run from the client's internal goroutines;
  firing one from Enqueue could re-enter or block the caller's request
  path. Drop the notifyFailure call; the caller learns via the returned
  ErrQueueFull. This also makes the closed-channel race drop (ErrClosed,
  no callback) and the full-queue drop consistent.
- Remove the per-event Warnf: under sustained overload it re-created the
  caller latency the non-blocking drop path exists to avoid.

Update the white-box test to assert the drop fires no callback and logs
nothing on the caller goroutine, and refresh the now-stale benchmark
log-silencing comment (the flood came from stage-2 sendBatch backpressure
logging, not the drop path).
@marandaneto

Copy link
Copy Markdown
Member

btw take a look at PostHog/posthog-python#146
i also think we should not block if the queue is full
iirc mobile works as a circular buffer, so we drop the oldest and not the newest from the queue, both have trade offs but worth discussing which ones makes more sense for frontend vs backend sdks cc @PostHog/team-client-libraries

@dustinbyrne

Copy link
Copy Markdown
Contributor

i think that dropping the newest event makes sense, given we have a chance at call time to notify the caller that the queue is full. otherwise, it'd be a silent side-effect.

@dustinbyrne

Copy link
Copy Markdown
Contributor

regarding PostHog/posthog-python#146 see my response here PostHog/posthog-python#731 (review)

@eli-r-ph
eli-r-ph merged commit 39c72dc into main Jul 15, 2026
26 checks passed
@eli-r-ph
eli-r-ph deleted the fix/enqueue-drop-newest-on-full branch July 15, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants