From d8bc5819aac8e13193fb4fd6090c315dfae4b5bd Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 3 Aug 2026 19:23:55 +0300 Subject: [PATCH 01/22] feat(server): opt-in uncaught exception capture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds captureUncaughtExceptions (default false) to the server PostHogConfig. When enabled, the core PostHogErrorTrackingAutoCaptureIntegration installs a Thread.defaultUncaughtExceptionHandler that captures the throwable as a fatal, unhandled $exception (mechanism UncaughtExceptionHandler), flushes, then delegates to the previously installed handler. Core changes (all additive; Android behavior and the released install(PostHogInterface) path unchanged): - Gate strategy seam on the integration so the server can install with a local-only gate (no remote config, which the server SDK never fetches); Android keeps the remote errorTracking.autocaptureExceptions gate. - Captures flow through an internal CaptureTarget seam so the server's stateless client can drive the integration. - Handler-install ownership is tracked per integration instance, so closing a second opted-in client (whose install was a process-wide no-op) does not tear down the handler a still-open first client owns. - New @PostHogInternal PostHogCapturedThrowables identity marker (weak, ReferenceQueue-pruned). The guard is directional: log mirrors consult it, the uncaught handler only marks — a crash is always captured as the authoritative fatal/unhandled record even if the same instance was logged first (logger.error(..., e); throw e), and marking keeps post-crash log mirrors from re-reporting it. - Repeated setup() cannot replace the owning integration with a non-owning one, which would leave the global handler installed after close(). - With no previous default handler to chain to, the handler reproduces the JVM's built-in stderr crash output so enabling capture never hides crashes from stderr log collection. - Server config KDoc documents the flushAt implication for the crash path. --- .changeset/core-captured-throwables.md | 5 + .changeset/core-uncaught-gate.md | 5 + .changeset/server-uncaught-exceptions.md | 5 + posthog-server/api/posthog-server.api | 3 + .../main/java/com/posthog/server/PostHog.kt | 44 +++ .../java/com/posthog/server/PostHogConfig.kt | 33 +++ .../server/PostHogUncaughtExceptionTest.kt | 278 ++++++++++++++++++ posthog/api/posthog.api | 12 + ...tHogErrorTrackingAutoCaptureIntegration.kt | 117 +++++++- .../PostHogCapturedThrowables.kt | 69 +++++ ...ErrorTrackingAutoCaptureIntegrationTest.kt | 137 +++++++++ .../PostHogCapturedThrowablesTest.kt | 37 +++ 12 files changed, 736 insertions(+), 9 deletions(-) create mode 100644 .changeset/core-captured-throwables.md create mode 100644 .changeset/core-uncaught-gate.md create mode 100644 .changeset/server-uncaught-exceptions.md create mode 100644 posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt create mode 100644 posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt create mode 100644 posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt diff --git a/.changeset/core-captured-throwables.md b/.changeset/core-captured-throwables.md new file mode 100644 index 000000000..5dd7eb2ed --- /dev/null +++ b/.changeset/core-captured-throwables.md @@ -0,0 +1,5 @@ +--- +'posthog': patch +--- + +Add an internal process-wide `PostHogCapturedThrowables` guard (marked `@PostHogInternal`, visible only because of the multi-module architecture) that lets independent error-capture paths avoid double-reporting the same `Throwable` instance. The guard is directional: log-mirror paths (e.g. the `posthog-server-logback` appender) consult it and skip instances already reported, while the uncaught-exception handler only marks — a crash is always captured as the authoritative fatal/unhandled record even if the same instance was logged first, and marking it keeps post-crash log mirrors from reporting it again. Membership is keyed on instance identity and held weakly, so the guard never keeps a throwable or its stack alive. diff --git a/.changeset/core-uncaught-gate.md b/.changeset/core-uncaught-gate.md new file mode 100644 index 000000000..17a4d68b5 --- /dev/null +++ b/.changeset/core-uncaught-gate.md @@ -0,0 +1,5 @@ +--- +'posthog': patch +--- + +`PostHogErrorTrackingAutoCaptureIntegration` can now be gated on a caller-supplied strategy instead of the built-in gate (local `errorTrackingConfig.autoCapture` with remote config as a kill-switch): a new `PostHogErrorTrackingAutoCaptureIntegration(config, enabledGate)` constructor lets SDK layers that never fetch remote config (e.g. the server SDK) decide autocapture purely from local config. The uncaught handler also delivers captures through an internal `CaptureTarget` seam (`installWith`) so it can drive clients that are not a core `PostHogInterface`, and when no previous default handler exists it now reproduces the JVM's own `Exception in thread ...` stderr output, so installing capture never hides a crash from log collection. Android behavior and the existing `install(PostHogInterface)` path are otherwise unchanged; the additions are internal (`@PostHogInternal`) and visible only because of the multi-module architecture. diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md new file mode 100644 index 000000000..d29e7056e --- /dev/null +++ b/.changeset/server-uncaught-exceptions.md @@ -0,0 +1,5 @@ +--- +'posthog-server': minor +--- + +Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event is enqueued and sent synchronously on the crashing thread, bypassing `flushAt`, so capturing the crash does not depend on the periodic flush; delivery is still best-effort under an immediate hard exit. diff --git a/posthog-server/api/posthog-server.api b/posthog-server/api/posthog-server.api index 4c50cfaaf..e10d6bcb3 100644 --- a/posthog-server/api/posthog-server.api +++ b/posthog-server/api/posthog-server.api @@ -134,6 +134,7 @@ public class com/posthog/server/PostHogConfig { public final fun addIntegration (Lcom/posthog/PostHogIntegration;)V public static final fun builder (Ljava/lang/String;)Lcom/posthog/server/PostHogConfig$Builder; public final fun getApiKey ()Ljava/lang/String; + public final fun getCaptureUncaughtExceptions ()Z public final fun getDebug ()Z public final fun getEncryption ()Lcom/posthog/PostHogEncryption; public final fun getEvaluationContexts ()Ljava/util/List; @@ -158,6 +159,7 @@ public class com/posthog/server/PostHogConfig { public final fun getRemoteConfig ()Z public final fun getSendFeatureFlagEvent ()Z public final fun removeBeforeSend (Lcom/posthog/PostHogBeforeSend;)V + public final fun setCaptureUncaughtExceptions (Z)V public final fun setDebug (Z)V public final fun setEncryption (Lcom/posthog/PostHogEncryption;)V public final fun setEvaluationContexts (Ljava/util/List;)V @@ -185,6 +187,7 @@ public class com/posthog/server/PostHogConfig { public final class com/posthog/server/PostHogConfig$Builder { public fun (Ljava/lang/String;)V public final fun build ()Lcom/posthog/server/PostHogConfig; + public final fun captureUncaughtExceptions (Z)Lcom/posthog/server/PostHogConfig$Builder; public final fun debug (Z)Lcom/posthog/server/PostHogConfig$Builder; public final fun encryption (Lcom/posthog/PostHogEncryption;)Lcom/posthog/server/PostHogConfig$Builder; public final fun evaluationContexts (Ljava/util/List;)Lcom/posthog/server/PostHogConfig$Builder; diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index 1cd9a97c7..12395e293 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -2,6 +2,7 @@ package com.posthog.server import com.posthog.FeatureFlagResult import com.posthog.PostHogStateless +import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration import com.posthog.internal.FeatureFlag import com.posthog.server.internal.EvaluationsHost import com.posthog.server.internal.PostHogFeatureFlags @@ -26,11 +27,54 @@ public class PostHog : PostHogStateless(), PostHogInterface { } } + /** + * Uncaught-exception integration installed when [PostHogConfig.captureUncaughtExceptions] is + * enabled, retained so it can be uninstalled on [close]. + */ + private var uncaughtExceptionIntegration: PostHogErrorTrackingAutoCaptureIntegration? = null + override fun setup(config: T) { + // The base keeps its original state when it rejects a setup (already set up, or an invalid + // config), so only wire anything on top when THIS call is the one that enabled the client — + // otherwise a second setup() could install a handler bound to a config the base discarded. + val alreadySetUp = isEnabled() super.setup(config.asCoreConfig()) + if (alreadySetUp || !isEnabled()) { + return + } + + // Core setup never installs integrations for the stateless base, so wire the uncaught + // handler explicitly. Gate purely on the local server flag — the server SDK never fetches + // remote config, so the remote-config gate the Android SDK uses can never fire here. + // Single-owner by design: the handler is process-wide, so only the first client that opts in + // installs it. With several live clients all opting in, closing the owner restores the + // previous handler and the remaining clients do not take over — capture stops until a client + // is set up again. Server apps use one client per process, so we don't ref-count here. + if (config.captureUncaughtExceptions) { + getConfig()?.let { coreConfig -> + val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } + // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; + // routing it through captureException preserves those via the shared coercer, and the + // queue sends fatal exception events synchronously on the crashing thread. + integration.installWith( + object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { + override fun capture(throwable: Throwable) { + captureException(throwable) + } + + override fun flush() { + this@PostHog.flush() + } + }, + ) + uncaughtExceptionIntegration = integration + } + } } override fun close() { + uncaughtExceptionIntegration?.uninstall() + uncaughtExceptionIntegration = null super.close() } diff --git a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt index 30a6bbef9..336268f82 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt @@ -200,6 +200,28 @@ public open class PostHogConfig constructor( */ public var inAppExcludes: List = DEFAULT_IN_APP_EXCLUDES + /** + * Opt in to capturing uncaught exceptions for the whole JVM as `$exception` events. + * + * When true, [PostHog] installs a [Thread.defaultUncaughtExceptionHandler] on setup that + * captures the crashing exception (marked fatal, `handled=false`, mechanism + * `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered + * handler. The handler is removed again on [PostHog.close]. + * + * Unlike the Android SDK, this is gated purely on this local flag — the server SDK never + * fetches remote config, so no remote toggle is involved. + * + * Delivery: the queue treats a fatal `$exception` event specially — it is enqueued and sent + * synchronously on the crashing thread, bypassing [flushAt] — so the crash itself does not depend + * on the periodic flush. The handler still calls `flush()` afterwards for anything else that was + * pending. Delivery remains best-effort under an immediate hard exit (the same guarantee the + * Android SDK provides). See [PostHog] for details. + * + * Docs https://posthog.com/docs/error-tracking + * Defaults to false + */ + public var captureUncaughtExceptions: Boolean = false + private val beforeSendCallbacks = mutableListOf() private val integrations = mutableListOf() @@ -382,6 +404,7 @@ public open class PostHogConfig constructor( private var releaseIdentifier: String? = null private var inAppIncludes: List = emptyList() private var inAppExcludes: List = DEFAULT_IN_APP_EXCLUDES + private var captureUncaughtExceptions: Boolean = false /** * Sets the PostHog ingestion host. @@ -594,6 +617,15 @@ public open class PostHogConfig constructor( */ public fun inAppExcludes(inAppExcludes: List): Builder = apply { this.inAppExcludes = inAppExcludes.toList() } + /** + * Opts in to capturing uncaught JVM exceptions as `$exception` events. + * + * @param captureUncaughtExceptions true to install a global uncaught-exception handler on setup. + * @return This builder. + */ + public fun captureUncaughtExceptions(captureUncaughtExceptions: Boolean): Builder = + apply { this.captureUncaughtExceptions = captureUncaughtExceptions } + /** * Builds a [PostHogConfig] from the accumulated values. * @@ -628,6 +660,7 @@ public open class PostHogConfig constructor( config.releaseIdentifier = releaseIdentifier config.inAppIncludes = inAppIncludes config.inAppExcludes = inAppExcludes + config.captureUncaughtExceptions = captureUncaughtExceptions return config } } diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt new file mode 100644 index 000000000..8bd8d2252 --- /dev/null +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt @@ -0,0 +1,278 @@ +package com.posthog.server + +import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration +import okhttp3.mockwebserver.MockResponse +import okhttp3.mockwebserver.MockWebServer +import java.util.concurrent.TimeUnit +import kotlin.test.AfterTest +import kotlin.test.BeforeTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertNotNull +import kotlin.test.assertSame +import kotlin.test.assertTrue + +/** + * Tests for the opt-in server uncaught-exception capture wired via + * [PostHogConfig.captureUncaughtExceptions]. + * + * The global [Thread.setDefaultUncaughtExceptionHandler] and the integration's process-global + * install flag are shared JVM state, so every test restores the original handler and closes the + * client to keep the suite isolated. + */ +internal class PostHogUncaughtExceptionTest { + private var originalHandler: Thread.UncaughtExceptionHandler? = null + + @BeforeTest + fun setUp() { + originalHandler = Thread.getDefaultUncaughtExceptionHandler() + } + + @AfterTest + fun tearDown() { + // Defensively clear the process-global install flag if a test threw before closing, so a + // leaked handler can't turn later tests' installs into no-ops. + (Thread.getDefaultUncaughtExceptionHandler() as? PostHogErrorTrackingAutoCaptureIntegration) + ?.uninstall() + Thread.setDefaultUncaughtExceptionHandler(originalHandler) + } + + private fun startServer(): MockWebServer = + MockWebServer().apply { + enqueue(MockResponse().setResponseCode(200)) + start() + } + + @Test + fun `disabled by default does not install an uncaught handler`() { + val sentinel = Thread.UncaughtExceptionHandler { _, _ -> } + Thread.setDefaultUncaughtExceptionHandler(sentinel) + + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host("https://example.com") + .build(), + ) + + // The default handler must be untouched when the option is off. + assertSame(sentinel, Thread.getDefaultUncaughtExceptionHandler()) + + postHog.close() + } + + @Test + fun `enabled installs handler and chains the previous one, restored on close`() { + var chainedThread: Thread? = null + var chainedThrowable: Throwable? = null + val sentinel = + Thread.UncaughtExceptionHandler { thread, throwable -> + chainedThread = thread + chainedThrowable = throwable + } + Thread.setDefaultUncaughtExceptionHandler(sentinel) + + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + // Our integration is now the default handler and it is not the sentinel. + val installed = Thread.getDefaultUncaughtExceptionHandler() + assertTrue( + installed is PostHogErrorTrackingAutoCaptureIntegration, + "Expected the PostHog integration to be installed as the default handler", + ) + + // Simulate an uncaught exception. + val thread = Thread.currentThread() + val boom = RuntimeException("boom") + installed.uncaughtException(thread, boom) + + // The previous handler is chained after capture. + assertSame(thread, chainedThread) + assertSame(boom, chainedThrowable) + + // Close removes our handler and restores the sentinel. + postHog.close() + assertSame(sentinel, Thread.getDefaultUncaughtExceptionHandler()) + + mockServer.shutdown() + } + + @Test + fun `uncaught exception is captured as a fatal, unhandled exception event`() { + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + val handler = Thread.getDefaultUncaughtExceptionHandler() + assertTrue(handler is PostHogErrorTrackingAutoCaptureIntegration) + + handler.uncaughtException(Thread.currentThread(), IllegalStateException("kaboom")) + + val request = mockServer.takeRequest(5, TimeUnit.SECONDS) + assertNotNull(request, "Expected a /batch request within 5 seconds") + + val batch = request.parseBatch() + val exceptionEvent = batch.findEvent("\$exception") + assertNotNull(exceptionEvent, "Expected an \$exception event") + + val props = batch.eventProperties("\$exception") + assertEquals("fatal", props["\$exception_level"], "Uncaught exceptions must be fatal") + + @Suppress("UNCHECKED_CAST") + val exceptionList = props["\$exception_list"] as? List> + assertNotNull(exceptionList, "Expected a \$exception_list") + assertTrue(exceptionList.isNotEmpty()) + + @Suppress("UNCHECKED_CAST") + val mechanism = exceptionList.first()["mechanism"] as? Map + assertNotNull(mechanism, "Expected a mechanism on the first exception item") + assertEquals(false, mechanism["handled"], "Uncaught exceptions must be marked handled=false") + assertEquals( + "UncaughtExceptionHandler", + mechanism["type"], + "Uncaught exceptions must carry the UncaughtExceptionHandler mechanism", + ) + + postHog.close() + mockServer.shutdown() + } + + @Test + fun `repeated setup keeps handler ownership so close still restores the previous handler`() { + val sentinel = Thread.UncaughtExceptionHandler { _, _ -> } + Thread.setDefaultUncaughtExceptionHandler(sentinel) + + val mockServer = startServer() + val config = + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(1) + .captureUncaughtExceptions(true) + .build() + val postHog = PostHog.with(config) + + assertTrue( + Thread.getDefaultUncaughtExceptionHandler() is PostHogErrorTrackingAutoCaptureIntegration, + ) + + // A second setup on the same instance is a no-op for the base client; it must not replace + // the owning integration with a non-owning one, or close() could no longer uninstall. + postHog.setup(config) + + postHog.close() + assertSame(sentinel, Thread.getDefaultUncaughtExceptionHandler()) + + mockServer.shutdown() + } + + @Test + fun `a rejected repeated setup does not install a handler`() { + val sentinel = Thread.UncaughtExceptionHandler { _, _ -> } + Thread.setDefaultUncaughtExceptionHandler(sentinel) + + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .build(), + ) + + assertSame(sentinel, Thread.getDefaultUncaughtExceptionHandler()) + + // The base client ignores a second setup and keeps its original config, so opting in through + // that rejected call must not install a handler either. + postHog.setup( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .captureUncaughtExceptions(true) + .build(), + ) + + assertSame( + sentinel, + Thread.getDefaultUncaughtExceptionHandler(), + "A rejected setup must not install the uncaught handler", + ) + + postHog.close() + mockServer.shutdown() + } + + @Test + fun `double setup does not install a second handler`() { + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + val firstHandler = Thread.getDefaultUncaughtExceptionHandler() + assertTrue(firstHandler is PostHogErrorTrackingAutoCaptureIntegration) + + // A second client that also opts in must not stack a second handler on top of ours (the + // process-global install guard makes the second install a no-op). + val second = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + assertSame( + firstHandler, + Thread.getDefaultUncaughtExceptionHandler(), + "The second setup must not replace the already-installed handler", + ) + + second.close() + postHog.close() + mockServer.shutdown() + } + + @Test + fun `enabled works with no remote config present`() { + // The server SDK never fetches remote config; the local-only gate must still install. + val sentinel = Thread.UncaughtExceptionHandler { _, _ -> } + Thread.setDefaultUncaughtExceptionHandler(sentinel) + + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .captureUncaughtExceptions(true) + .build(), + ) + + assertTrue( + Thread.getDefaultUncaughtExceptionHandler() is PostHogErrorTrackingAutoCaptureIntegration, + "Local-only gate should install even without any remote config", + ) + assertFalse(Thread.getDefaultUncaughtExceptionHandler() === sentinel) + + postHog.close() + mockServer.shutdown() + } +} diff --git a/posthog/api/posthog.api b/posthog/api/posthog.api index eeb326107..7d31d5332 100644 --- a/posthog/api/posthog.api +++ b/posthog/api/posthog.api @@ -533,12 +533,19 @@ public abstract interface annotation class com/posthog/PostHogVisibleForTesting public final class com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration : com/posthog/PostHogIntegration, java/lang/Thread$UncaughtExceptionHandler { public fun (Lcom/posthog/PostHogConfig;)V + public fun (Lcom/posthog/PostHogConfig;Lkotlin/jvm/functions/Function0;)V public fun install (Lcom/posthog/PostHogInterface;)V + public final fun installWith (Lcom/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration$CaptureTarget;)V public fun onRemoteConfig (Z)V public fun uncaughtException (Ljava/lang/Thread;Ljava/lang/Throwable;)V public fun uninstall ()V } +public abstract interface class com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration$CaptureTarget { + public abstract fun capture (Ljava/lang/Throwable;)V + public abstract fun flush ()V +} + public final class com/posthog/errortracking/PostHogErrorTrackingConfig { public fun ()V public fun (Z)V @@ -1170,6 +1177,11 @@ public final class com/posthog/internal/VariantDefinition { public final fun getRolloutPercentage ()D } +public final class com/posthog/internal/errortracking/PostHogCapturedThrowables { + public static final field INSTANCE Lcom/posthog/internal/errortracking/PostHogCapturedThrowables; + public final fun markAndCheck (Ljava/lang/Throwable;)Z +} + public final class com/posthog/internal/errortracking/ThrowableCoercer { public static final field EXCEPTION_LEVEL_ATTRIBUTE Ljava/lang/String; public static final field EXCEPTION_LEVEL_FATAL Ljava/lang/String; diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index 6d803852a..c16686654 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -3,6 +3,8 @@ package com.posthog.errortracking import com.posthog.PostHogConfig import com.posthog.PostHogIntegration import com.posthog.PostHogInterface +import com.posthog.PostHogInternal +import com.posthog.internal.errortracking.PostHogCapturedThrowables import com.posthog.internal.errortracking.PostHogThrowable import com.posthog.internal.errortracking.UncaughtExceptionHandlerAdapter import java.util.concurrent.atomic.AtomicBoolean @@ -11,12 +13,19 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th private val config: PostHogConfig private val adapterExceptionHandler: UncaughtExceptionHandlerAdapter + /** + * Decides whether the handler may capture. Defaults to the Android/core gate: local + * `errorTrackingConfig.autoCapture` with remote config acting only as a kill-switch. Layers that + * decide autocapture purely from local config (e.g. the server SDK, which never fetches remote + * config) supply their own gate. + */ + private val enabledGate: () -> Boolean + // @Volatile: read on the crashing thread in uncaughtException with no happens-before edge to // the install()/uninstall() writes; a pre-existing thread could otherwise see a stale null and // skip delegating to the app/system handler. @Volatile private var defaultExceptionHandler: Thread.UncaughtExceptionHandler? = null - private var postHog: PostHogInterface? = null private var ownsInstallation = false // Tracks whether we should capture, separate from whether we're linked into the handler chain. @@ -25,14 +34,60 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th @Volatile private var captureEnabled = false + /** + * Where captured uncaught exceptions are delivered. Set on install. The core [PostHogInterface] + * client and the stateless server client do not share a common capture supertype, so the handler + * targets this minimal seam instead of a concrete client type. + */ + private var captureTarget: CaptureTarget? = null + public constructor(config: PostHogConfig) { this.config = config this.adapterExceptionHandler = UncaughtExceptionHandlerAdapter.Adapter.getInstance() + this.enabledGate = { defaultGate() } + } + + /** + * Internal constructor allowing a custom [enabledGate]. Used by SDK layers (e.g. the server SDK) + * that decide autocapture purely from local config without any remote-config round trip. + * + * Not part of the public API; visible only because of the multi-module architecture. + */ + @PostHogInternal + public constructor(config: PostHogConfig, enabledGate: () -> Boolean) { + this.config = config + this.adapterExceptionHandler = UncaughtExceptionHandlerAdapter.Adapter.getInstance() + this.enabledGate = enabledGate } internal constructor(config: PostHogConfig, adapterExceptionHandler: UncaughtExceptionHandlerAdapter) { this.config = config this.adapterExceptionHandler = adapterExceptionHandler + this.enabledGate = { defaultGate() } + } + + internal constructor( + config: PostHogConfig, + adapterExceptionHandler: UncaughtExceptionHandlerAdapter, + enabledGate: () -> Boolean, + ) { + this.config = config + this.adapterExceptionHandler = adapterExceptionHandler + this.enabledGate = enabledGate + } + + /** + * Minimal capture surface the uncaught handler needs. Both the core client and the stateless + * server client can satisfy it, without sharing a public supertype. [capture] and [flush] are + * separate so the handler can ask for everything pending to be sent as its last act. + * + * Not part of the public API; visible only because of the multi-module architecture. + */ + @PostHogInternal + public interface CaptureTarget { + public fun capture(throwable: Throwable) + + public fun flush() } internal companion object { @@ -45,7 +100,29 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th @Synchronized override fun install(postHog: PostHogInterface) { - this.postHog = postHog + installWith( + object : CaptureTarget { + override fun capture(throwable: Throwable) { + postHog.captureException(throwable) + } + + override fun flush() { + postHog.flush() + } + }, + ) + } + + /** + * Installs the handler delivering captures to [target]. Used by SDK layers whose client is not a + * core [PostHogInterface] (e.g. the server SDK). + * + * Not part of the public API; visible only because of the multi-module architecture. + */ + @PostHogInternal + @Synchronized + public fun installWith(target: CaptureTarget) { + this.captureTarget = target // Already linked into the chain (possibly dormant below a handler installed after us): // resume capturing in place. Re-running the link logic while we're a mid-chain delegate @@ -84,8 +161,10 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th } } + private fun canCapture(): Boolean = enabledGate() + // Local config is the primary gate; remote config is only a kill-switch (below). - private fun canCapture(): Boolean = config.errorTrackingConfig.autoCapture && !remoteKillSwitchActive() + private fun defaultGate(): Boolean = config.errorTrackingConfig.autoCapture && !remoteKillSwitchActive() // Remote config is a kill-switch, not a gate: it blocks capture only when a config that // already exists — fetched this session or cached from a prior launch — explicitly disables @@ -122,7 +201,7 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th ownsInstallation = false integrationInstalled.set(false) // We're out of the chain now, so drop the delegate ref (a re-install re-reads it). - // postHog is kept: onRemoteConfig re-enable calls install(postHog) on this instance. + // captureTarget is kept: an onRemoteConfig re-enable re-installs with it. defaultExceptionHandler = null config.logger.log("Exception autocapture is disabled.") } else { @@ -147,7 +226,7 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th } val autocaptureExceptionsEnabled = config.remoteConfigHolder?.isAutocaptureExceptionsEnabled() ?: false if (autocaptureExceptionsEnabled) { - postHog?.let { install(it) } + captureTarget?.let { installWith(it) } } else { uninstall() } @@ -158,13 +237,33 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th throwable: Throwable, ) { if (captureEnabled) { - postHog?.let { postHog -> - postHog.captureException(PostHogThrowable(throwable, thread)) - postHog.flush() + captureTarget?.let { target -> + // Mark the throwable so post-crash log mirrors of this exact instance (e.g. a + // shutdown hook logging the crash) don't re-report it — but never skip the capture + // itself: this is the authoritative fatal/unhandled record for the crash and must not + // be downgraded by an earlier handled capture of the same instance + // (`logger.error(..., e); throw e`). + PostHogCapturedThrowables.markAndCheck(throwable) + target.capture(PostHogThrowable(throwable, thread)) + // The queue sends a fatal exception event synchronously on this (the crashing) + // thread, bypassing the flushAt threshold; this flush covers anything else still + // pending. Delivery is still best-effort under an immediate hard exit — same + // guarantee as the Android SDK. + target.flush() } } // Always delegate: we may still be mid-chain even while dormant. - defaultExceptionHandler?.uncaughtException(thread, throwable) + val previousHandler = defaultExceptionHandler + if (previousHandler != null) { + previousHandler.uncaughtException(thread, throwable) + } else if (throwable !is ThreadDeath) { + // No previous default handler: reproduce the JVM's built-in crash output that + // ThreadGroup would have printed had we not installed ourselves as the default handler, + // so opting into capture never hides crashes from stderr log collection. ThreadDeath is + // excluded because ThreadGroup stays silent for it. + System.err.print("Exception in thread \"${thread.name}\" ") + throwable.printStackTrace(System.err) + } } } diff --git a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt new file mode 100644 index 000000000..2c1aae9f0 --- /dev/null +++ b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt @@ -0,0 +1,69 @@ +package com.posthog.internal.errortracking + +import com.posthog.PostHogInternal +import java.lang.ref.ReferenceQueue +import java.lang.ref.WeakReference + +/** + * Process-wide guard that lets independent error-capture paths avoid double-reporting the very same + * [Throwable] instance. The guard is directional: log-mirror paths (the Logback appender) consult it + * and skip instances already reported, while the uncaught-exception handler only marks — a crash is + * always captured as the authoritative fatal/unhandled record, even if the same instance was logged + * first, and marking it prevents post-crash log mirrors from reporting it again. + * + * Membership is keyed strictly on **instance identity** (reference equality, not `equals`/`hashCode` + * — a `Throwable` subclass with value equality must not make two distinct instances collide) and + * held weakly, so entries disappear once the throwable is otherwise unreachable: the guard never + * keeps a throwable (or its stack) alive. + * + * Not part of the public API; visible only because of the multi-module architecture. + */ +@PostHogInternal +public object PostHogCapturedThrowables { + private val queue = ReferenceQueue() + + // Identity keys of throwables already captured. HashSet is not thread-safe, so all access is + // synchronized on the set. Cleared entries are pruned opportunistically via [queue]. + private val seen = HashSet() + + /** + * Records [throwable] as captured and reports whether the caller should capture it. + * + * @return true if this is the first time the instance has been marked (the caller should + * capture it), false if it was already marked (the caller should skip it). + */ + public fun markAndCheck(throwable: Throwable): Boolean = + synchronized(seen) { + pruneCleared() + seen.add(IdentityWeakKey(throwable, queue)) + } + + // Drop keys whose referent has been collected. Must be called while holding the lock. + private fun pruneCleared() { + while (true) { + val cleared = queue.poll() ?: break + seen.remove(cleared) + } + } + + // Weak reference whose identity is the referential identity of the referent, so distinct + // instances never collide even if their class overrides equals/hashCode. hashCode is captured + // eagerly (identity hash is stable) so a key still matches after its referent is cleared. + private class IdentityWeakKey( + referent: Throwable, + queue: ReferenceQueue, + ) : WeakReference(referent, queue) { + private val identityHash = System.identityHashCode(referent) + + override fun hashCode(): Int = identityHash + + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is IdentityWeakKey) return false + val self = get() + // Reference equality on the referents; a cleared referent only matches itself (handled + // by the identity check above), which is fine — such keys are pruned via the queue. + return self != null && self === other.get() + } + } +} diff --git a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt index 2aefb1799..ecdab1042 100644 --- a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt +++ b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt @@ -4,6 +4,7 @@ import com.posthog.PostHogConfig import com.posthog.PostHogInterface import com.posthog.internal.PostHogPrintLogger import com.posthog.internal.PostHogRemoteConfig +import com.posthog.internal.errortracking.PostHogCapturedThrowables import com.posthog.internal.errortracking.PostHogThrowable import com.posthog.internal.errortracking.UncaughtExceptionHandlerAdapter import org.mockito.kotlin.any @@ -16,6 +17,7 @@ import org.mockito.kotlin.whenever import kotlin.test.AfterTest import kotlin.test.BeforeTest import kotlin.test.Test +import kotlin.test.assertEquals internal class PostHogErrorTrackingAutoCaptureIntegrationTest { private val mockConfig = mock() @@ -261,6 +263,30 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { verify(mockAdapter, never()).setDefaultUncaughtExceptionHandler(any()) } + @Test + fun `local-only gate installs without any remote config present`() { + // No remoteConfigHolder and no errorTrackingConfig stubbed: the default gate reads both, a + // local-only gate bypasses them entirely. This is the server SDK's path. + currentHandler = null + + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + integration.install(mockPostHog) + + verify(mockAdapter).setDefaultUncaughtExceptionHandler(integration) + + integration.uninstall() + } + + @Test + fun `local-only gate that is false does not install`() { + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { false } + integration.install(mockPostHog) + + verify(mockAdapter, never()).setDefaultUncaughtExceptionHandler(any()) + + integration.uninstall() + } + @Test fun `onRemoteConfig can re-install after being disabled`() { whenever(mockConfig.remoteConfigHolder).thenReturn(mockRemoteConfig) @@ -556,4 +582,115 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { integration.uninstall() } + + private class RecordingTarget : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { + val captured = mutableListOf() + var flushCount = 0 + + override fun capture(throwable: Throwable) { + captured.add(throwable) + } + + override fun flush() { + flushCount++ + } + } + + @Test + fun `uncaughtException captures even when the throwable was already captured elsewhere`() { + val target = RecordingTarget() + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + integration.installWith(target) + + val alreadyLogged = RuntimeException("logged then crashed") + // Simulate the appender having captured this exact instance first. + assertEquals(true, PostHogCapturedThrowables.markAndCheck(alreadyLogged)) + + integration.uncaughtException(Thread.currentThread(), alreadyLogged) + + // The crash is the authoritative fatal/unhandled record: it must be captured even though + // the instance was already reported as a handled log capture, and the queue is flushed so + // both events leave before the process exits. + assertEquals(1, target.captured.size, "The crash capture must not be suppressed by dedup") + assertEquals(1, target.flushCount, "flush() must run on the crash path") + + integration.uninstall() + } + + @Test + fun `uncaughtException marks the throwable so later log mirrors dedup against it`() { + val target = RecordingTarget() + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + integration.installWith(target) + + val crash = RuntimeException("crashed then logged") + integration.uncaughtException(Thread.currentThread(), crash) + + assertEquals(1, target.captured.size) + // A post-crash log mirror consulting the guard must see the instance as already captured. + assertEquals(false, PostHogCapturedThrowables.markAndCheck(crash)) + + integration.uninstall() + } + + @Test + fun `uncaughtException reproduces the JVM default crash output when no previous handler exists`() { + currentHandler = null + + val target = RecordingTarget() + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + integration.installWith(target) + + val originalErr = System.err + val stderr = java.io.ByteArrayOutputStream() + System.setErr(java.io.PrintStream(stderr)) + try { + integration.uncaughtException(Thread.currentThread(), RuntimeException("printed crash")) + } finally { + System.setErr(originalErr) + } + + val output = stderr.toString() + // Installing capture must not hide the crash from stderr: with no previous handler to + // chain to, the integration prints what ThreadGroup's default behavior would have. + assertEquals(true, output.contains("Exception in thread"), "Expected the default crash banner, got: $output") + assertEquals(true, output.contains("printed crash"), "Expected the throwable in stderr, got: $output") + + integration.uninstall() + } + + @Test + fun `uncaughtException captures and flushes for a fresh throwable`() { + val target = RecordingTarget() + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + integration.installWith(target) + + integration.uncaughtException(Thread.currentThread(), RuntimeException("fresh")) + + assertEquals(1, target.captured.size, "A first-seen throwable must be captured") + assertEquals(1, target.flushCount) + + integration.uninstall() + } + + @Test + fun `uninstall by a non-installing instance does not tear down the installed handler`() { + // First instance installs and owns the global handler. + currentHandler = mockExceptionHandler + val first = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + first.installWith(RecordingTarget()) + verify(mockAdapter).setDefaultUncaughtExceptionHandler(first) + + // Second instance's install is a process-wide no-op (a handler is already installed). + val second = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + second.installWith(RecordingTarget()) + + // Closing the second must NOT restore/replace the handler — it never installed. + second.uninstall() + verify(mockAdapter, never()).setDefaultUncaughtExceptionHandler(mockExceptionHandler) + + // The first still owns it and can restore on its own uninstall. + first.uninstall() + verify(mockAdapter).setDefaultUncaughtExceptionHandler(mockExceptionHandler) + } } diff --git a/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt b/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt new file mode 100644 index 000000000..9d1d1dda7 --- /dev/null +++ b/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt @@ -0,0 +1,37 @@ +package com.posthog.internal.errortracking + +import kotlin.test.Test +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +internal class PostHogCapturedThrowablesTest { + @Test + fun `first mark returns true, repeat mark of same instance returns false`() { + val throwable = RuntimeException("boom") + + assertTrue(PostHogCapturedThrowables.markAndCheck(throwable), "first sighting should be captured") + assertFalse(PostHogCapturedThrowables.markAndCheck(throwable), "same instance should be deduped") + } + + @Test + fun `distinct instances that are equal by value are both captured`() { + // A Throwable subclass with value equality must NOT make two distinct instances collide: + // dedup is strictly instance-identity based. + val a = ValueEqualThrowable("same") + val b = ValueEqualThrowable("same") + + // Sanity: they are equal by value but distinct instances. + assertTrue(a == b) + assertFalse(a === b) + + assertTrue(PostHogCapturedThrowables.markAndCheck(a), "first instance captured") + assertTrue(PostHogCapturedThrowables.markAndCheck(b), "second, value-equal instance must still be captured") + assertFalse(PostHogCapturedThrowables.markAndCheck(a), "re-marking the first instance is still deduped") + } + + private class ValueEqualThrowable(private val token: String) : RuntimeException(token) { + override fun equals(other: Any?): Boolean = other is ValueEqualThrowable && other.token == token + + override fun hashCode(): Int = token.hashCode() + } +} From 06affefbb0f181ec2eb4d6d0cf9695b4d0c777d5 Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:56:54 +0000 Subject: [PATCH 02/22] fix(server): guard uncaught-handler lifecycle with setupLock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The server PostHog.setup/close overrides did their integration lifecycle work outside setupLock. Two concurrent setup() calls could both read alreadySetUp as false, letting a rejected call install an uncaught-exception handler bound to a config the base discarded; and a close() racing setup() could read uncaughtExceptionIntegration before it was assigned, leaking the process-wide handler after the client closed. Wrap both the setup and close bodies in synchronized(setupLock) — the same reentrant monitor the base uses and the core client installs its integrations under — so the enabled-transition check + handler install are atomic with the base's setup, and the field is only ever touched under the lock. No behavior change for the single-client-per-process path. Paths: posthog-server/src/main/java/com/posthog/server/PostHog.kt Generated-By: PostHog Desktop Task-Id: 468ded4c-ba2e-440c-8096-5d78489afcdc --- .../main/java/com/posthog/server/PostHog.kt | 84 +++++++++++-------- 1 file changed, 48 insertions(+), 36 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index 12395e293..ab3c01461 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -34,48 +34,60 @@ public class PostHog : PostHogStateless(), PostHogInterface { private var uncaughtExceptionIntegration: PostHogErrorTrackingAutoCaptureIntegration? = null override fun setup(config: T) { - // The base keeps its original state when it rejects a setup (already set up, or an invalid - // config), so only wire anything on top when THIS call is the one that enabled the client — - // otherwise a second setup() could install a handler bound to a config the base discarded. - val alreadySetUp = isEnabled() - super.setup(config.asCoreConfig()) - if (alreadySetUp || !isEnabled()) { - return - } + // Hold setupLock across the whole lifecycle so the enabled-transition check and the handler + // install stay atomic with the base's own setup. The monitor is reentrant, so super.setup + // re-acquires it harmlessly. Without this, two concurrent setup() calls could both observe + // alreadySetUp as false and the rejected one would install a handler bound to a config the + // base discarded; and a concurrent close() could read uncaughtExceptionIntegration before it + // was assigned and leak the process-wide handler after the client closed. + synchronized(setupLock) { + // The base keeps its original state when it rejects a setup (already set up, or an invalid + // config), so only wire anything on top when THIS call is the one that enabled the client — + // otherwise a second setup() could install a handler bound to a config the base discarded. + val alreadySetUp = isEnabled() + super.setup(config.asCoreConfig()) + if (alreadySetUp || !isEnabled()) { + return + } - // Core setup never installs integrations for the stateless base, so wire the uncaught - // handler explicitly. Gate purely on the local server flag — the server SDK never fetches - // remote config, so the remote-config gate the Android SDK uses can never fire here. - // Single-owner by design: the handler is process-wide, so only the first client that opts in - // installs it. With several live clients all opting in, closing the owner restores the - // previous handler and the remaining clients do not take over — capture stops until a client - // is set up again. Server apps use one client per process, so we don't ref-count here. - if (config.captureUncaughtExceptions) { - getConfig()?.let { coreConfig -> - val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } - // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; - // routing it through captureException preserves those via the shared coercer, and the - // queue sends fatal exception events synchronously on the crashing thread. - integration.installWith( - object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { - override fun capture(throwable: Throwable) { - captureException(throwable) - } - - override fun flush() { - this@PostHog.flush() - } - }, - ) - uncaughtExceptionIntegration = integration + // Core setup never installs integrations for the stateless base, so wire the uncaught + // handler explicitly. Gate purely on the local server flag — the server SDK never fetches + // remote config, so the remote-config gate the Android SDK uses can never fire here. + // Single-owner by design: the handler is process-wide, so only the first client that opts in + // installs it. With several live clients all opting in, closing the owner restores the + // previous handler and the remaining clients do not take over — capture stops until a client + // is set up again. Server apps use one client per process, so we don't ref-count here. + if (config.captureUncaughtExceptions) { + getConfig()?.let { coreConfig -> + val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } + // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; + // routing it through captureException preserves those via the shared coercer, and the + // queue sends fatal exception events synchronously on the crashing thread. + integration.installWith( + object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { + override fun capture(throwable: Throwable) { + captureException(throwable) + } + + override fun flush() { + this@PostHog.flush() + } + }, + ) + uncaughtExceptionIntegration = integration + } } } } override fun close() { - uncaughtExceptionIntegration?.uninstall() - uncaughtExceptionIntegration = null - super.close() + // Same lock as setup so the uninstall + field clear cannot race a concurrent setup() that is + // still assigning uncaughtExceptionIntegration; super.close re-acquires the reentrant lock. + synchronized(setupLock) { + uncaughtExceptionIntegration?.uninstall() + uncaughtExceptionIntegration = null + super.close() + } } override fun identify( From d2ced6cdff27cc2531a384675d359b095487ea1c Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Thu, 20 Aug 2026 20:56:27 +0300 Subject: [PATCH 03/22] fix(server): flush the crash capture behind an executor barrier --- .changeset/server-uncaught-exceptions.md | 2 +- .../main/java/com/posthog/server/PostHog.kt | 26 ++++- .../java/com/posthog/server/PostHogConfig.kt | 12 +- .../server/internal/PostHogMemoryQueue.kt | 80 +++++++++++++ .../server/PostHogUncaughtExceptionTest.kt | 109 +++++++++++++++++- .../server/internal/PostHogMemoryQueueTest.kt | 76 ++++++++++++ ...tHogErrorTrackingAutoCaptureIntegration.kt | 8 +- 7 files changed, 296 insertions(+), 17 deletions(-) diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index d29e7056e..def367b7b 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -2,4 +2,4 @@ 'posthog-server': minor --- -Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event is enqueued and sent synchronously on the crashing thread, bypassing `flushAt`, so capturing the crash does not depend on the periodic flush; delivery is still best-effort under an immediate hard exit. +Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. The crash capture is enqueued asynchronously like any other event, and the crash path then performs a bounded blocking flush that is ordered behind that pending enqueue and ignores `flushAt`, so the event gets a network attempt before the JVM exits without ever blocking the crashing thread indefinitely. Delivery stays best-effort, the same guarantee class as the Android SDK: the flush can hit its timeout, the HTTP attempt can fail, and an immediate hard exit can cut it short. Sending the fatal event itself synchronously and independently of the queue threshold is a known follow-up. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index ab3c01461..4a92f6c5d 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -6,6 +6,7 @@ import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration import com.posthog.internal.FeatureFlag import com.posthog.server.internal.EvaluationsHost import com.posthog.server.internal.PostHogFeatureFlags +import com.posthog.server.internal.PostHogMemoryQueue @Suppress("DEPRECATION") public class PostHog : PostHogStateless(), PostHogInterface { @@ -61,8 +62,7 @@ public class PostHog : PostHogStateless(), PostHogInterface { getConfig()?.let { coreConfig -> val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; - // routing it through captureException preserves those via the shared coercer, and the - // queue sends fatal exception events synchronously on the crashing thread. + // routing it through captureException preserves those via the shared coercer. integration.installWith( object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { override fun capture(throwable: Throwable) { @@ -70,7 +70,21 @@ public class PostHog : PostHogStateless(), PostHogInterface { } override fun flush() { - this@PostHog.flush() + // Crash path only: capture() above enqueues on the queue executor, and the + // regular flush() runs inline on this (the crashing) thread — it would + // usually read the queue before that enqueue landed, send nothing, and let + // the event die with the JVM. flushBlocking orders the drain behind the + // pending enqueue and blocks the crashing thread for at most + // CRASH_FLUSH_TIMEOUT_MS, like the Rust SDK's bounded panic-hook flush. + // The server client always runs a PostHogMemoryQueue + // (PostHogConfig.asCoreConfig), so the fallback is unreachable in practice + // and only keeps the crash flush from silently becoming a no-op. + val memoryQueue = this@PostHog.queue as? PostHogMemoryQueue + if (memoryQueue != null) { + memoryQueue.flushBlocking(CRASH_FLUSH_TIMEOUT_MS) + } else { + this@PostHog.flush() + } } }, ) @@ -461,6 +475,12 @@ public class PostHog : PostHogStateless(), PostHogInterface { } public companion object { + /** + * How long the crashing thread waits for the crash event's flush to reach the network attempt + * before it delegates to the next handler and lets the JVM go down. + */ + private const val CRASH_FLUSH_TIMEOUT_MS = 2_000L + /** * Sets up the SDK and returns an instance that you can hold and pass around. * diff --git a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt index 336268f82..c1e2ed3f0 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt @@ -211,11 +211,13 @@ public open class PostHogConfig constructor( * Unlike the Android SDK, this is gated purely on this local flag — the server SDK never * fetches remote config, so no remote toggle is involved. * - * Delivery: the queue treats a fatal `$exception` event specially — it is enqueued and sent - * synchronously on the crashing thread, bypassing [flushAt] — so the crash itself does not depend - * on the periodic flush. The handler still calls `flush()` afterwards for anything else that was - * pending. Delivery remains best-effort under an immediate hard exit (the same guarantee the - * Android SDK provides). See [PostHog] for details. + * Delivery: the crash capture is enqueued asynchronously like any other event, and the crash path + * then performs a bounded blocking flush — ordered behind that pending enqueue and ignoring + * [flushAt] — so the event gets a network attempt before the JVM exits instead of waiting for the + * periodic flush, without ever blocking the crashing thread indefinitely. Delivery stays + * best-effort, the same guarantee class the Android SDK provides: the flush can hit its timeout, + * the HTTP attempt can fail, and an immediate hard exit can cut it short. See [PostHog] for + * details. * * Docs https://posthog.com/docs/error-tracking * Defaults to false diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 7d894d3b3..81027caeb 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -12,7 +12,9 @@ import java.io.IOException import java.util.Date import java.util.Timer import java.util.TimerTask +import java.util.concurrent.CountDownLatch import java.util.concurrent.ExecutorService +import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicBoolean import kotlin.concurrent.schedule import kotlin.math.min @@ -76,6 +78,84 @@ internal class PostHogMemoryQueue( return } + flushIgnoringThreshold() + } + + /** + * Bounded blocking flush for the crash path: submits a barrier task onto the queue executor and + * waits up to [timeoutMs] for that task to send the pending batch. + * + * [add] enqueues on the executor, so a caller that flushes inline (see [flush]) can read the + * deque before its own event landed and send nothing — fatal on a crashing thread whose process + * is about to exit, taking the daemon queue thread with it. The executor is single-threaded + * (owned by `PostHogStateless`), so a barrier submitted after [add] runs strictly after that + * enqueue instead of racing it. + * + * The send runs on the executor thread and ignores `flushAt` (a crash must not wait for the + * threshold), so the caller blocks on the barrier only, never longer than [timeoutMs]. + * + * @return true when the barrier ran within the timeout. Delivery itself stays best-effort: the + * batch is capped at `maxBatchSize`, another flush already in progress (the periodic timer runs + * on its own thread) makes the barrier a no-op, and the HTTP attempt can still fail or be cut + * short by process exit. False means the barrier could not be scheduled (executor shut down) or + * did not run in time. + */ + fun flushBlocking(timeoutMs: Long): Boolean { + val drained = CountDownLatch(1) + + try { + executor.execute { + try { + flushIgnoringThreshold() + } finally { + drained.countDown() + } + } + } catch (e: Throwable) { + // RejectedExecutionException and friends: the executor is gone, there is nothing to await + config.logger.log("Blocking flush could not be scheduled: $e.") + return false + } + + return awaitUninterruptibly(drained, timeoutMs) + } + + /** + * Waits out the whole [timeoutMs] even when the calling thread is already interrupted or gets + * interrupted while waiting — a crash on a thread some shutdown just interrupted must still get + * its flush attempt instead of returning on the spot with the budget unspent. The interrupt flag + * is restored before returning so the caller's own handling still sees it. + */ + private fun awaitUninterruptibly( + latch: CountDownLatch, + timeoutMs: Long, + ): Boolean { + val deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeoutMs) + var interrupted = false + try { + while (true) { + val remaining = deadline - System.nanoTime() + if (remaining <= 0) { + config.logger.log("Blocking flush timed out after $timeoutMs ms.") + return false + } + try { + if (latch.await(remaining, TimeUnit.NANOSECONDS)) { + return true + } + } catch (e: InterruptedException) { + // await clears the interrupt status when it throws, so the next wait really waits. + interrupted = true + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt() + } + } + } + + private fun flushIgnoringThreshold() { if (isFlushing.getAndSet(true)) { config.logger.log("Queue is flushing.") return diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt index 8bd8d2252..afd588db7 100644 --- a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt @@ -1,9 +1,14 @@ package com.posthog.server import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration +import okhttp3.mockwebserver.Dispatcher import okhttp3.mockwebserver.MockResponse import okhttp3.mockwebserver.MockWebServer +import okhttp3.mockwebserver.RecordedRequest +import java.util.concurrent.CopyOnWriteArrayList +import java.util.concurrent.CountDownLatch import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger import kotlin.test.AfterTest import kotlin.test.BeforeTest import kotlin.test.Test @@ -78,7 +83,6 @@ internal class PostHogUncaughtExceptionTest { PostHog.with( PostHogConfig.builder(TEST_API_KEY) .host(mockServer.url("/").toString()) - .flushAt(1) .captureUncaughtExceptions(true) .build(), ) @@ -109,11 +113,12 @@ internal class PostHogUncaughtExceptionTest { @Test fun `uncaught exception is captured as a fatal, unhandled exception event`() { val mockServer = startServer() + // Default flushAt (100) on purpose: the crash path's blocking flush is what delivers the + // event, so a low threshold would mask whether that flush works at all. val postHog = PostHog.with( PostHogConfig.builder(TEST_API_KEY) .host(mockServer.url("/").toString()) - .flushAt(1) .captureUncaughtExceptions(true) .build(), ) @@ -123,8 +128,9 @@ internal class PostHogUncaughtExceptionTest { handler.uncaughtException(Thread.currentThread(), IllegalStateException("kaboom")) - val request = mockServer.takeRequest(5, TimeUnit.SECONDS) - assertNotNull(request, "Expected a /batch request within 5 seconds") + // Already sent when the handler returned, so this should not have to wait. + val request = mockServer.takeRequest(1, TimeUnit.SECONDS) + assertNotNull(request, "Expected the crash event to be flushed before the handler returned") val batch = request.parseBatch() val exceptionEvent = batch.findEvent("\$exception") @@ -251,6 +257,101 @@ internal class PostHogUncaughtExceptionTest { mockServer.shutdown() } + @Test + fun `crash event is flushed even while its enqueue is still pending`() { + // Makes the crash-path race deterministic instead of hoping for a scheduling order. The queue + // thread is parked inside the gated warmup batch, so the crash capture's enqueue is provably + // still pending when the handler flushes, and with flushAt above one and a 600s flush interval + // nothing else can deliver the crash event. The handler runs on its own thread so the test can + // assert it is still blocked inside the flush while the enqueue is pending — the inline flush + // it used to do returned immediately, having read an empty queue, and the event died with the + // JVM. + val batches = CopyOnWriteArrayList() + val warmupReceived = CountDownLatch(1) + val gate = CountDownLatch(1) + val mockServer = MockWebServer() + mockServer.dispatcher = + object : Dispatcher() { + private val batchRequests = AtomicInteger(0) + + override fun dispatch(request: RecordedRequest): MockResponse { + if (request.path?.contains("/batch") == true) { + batches.add(request.parseBatch()) + if (batchRequests.getAndIncrement() == 0) { + warmupReceived.countDown() + // Holds the queue thread until the test opens the gate. + gate.await(5, TimeUnit.SECONDS) + } + } + return MockResponse().setResponseCode(200) + } + } + mockServer.start() + + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + .flushAt(2) + // Long enough that only the crash path can deliver the event within the test. + .flushIntervalSeconds(600) + .captureUncaughtExceptions(true) + .build(), + ) + + // Two events reach flushAt(2), so the queue thread flushes and parks in the gated request. + postHog.capture(DISTINCT_ID, "warmup_1") + postHog.capture(DISTINCT_ID, "warmup_2") + assertTrue( + warmupReceived.await(5, TimeUnit.SECONDS), + "Expected the warmup batch to reach the server and park the queue thread", + ) + + val handler = Thread.getDefaultUncaughtExceptionHandler() + assertTrue(handler is PostHogErrorTrackingAutoCaptureIntegration) + val crashThread = + Thread { + handler.uncaughtException(Thread.currentThread(), IllegalStateException("kaboom")) + }.apply { + isDaemon = true + start() + } + + // The only timed wait on this thread's crash path is the blocking flush's bounded await, so + // TIMED_WAITING while the queue thread is parked means the barrier is queued behind the + // pending enqueue and the handler is waiting for it. The old inline flush never blocked: the + // thread just finishes, and this assertion fails. + val parkedBy = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) + while (crashThread.isAlive && + crashThread.state != Thread.State.TIMED_WAITING && + System.nanoTime() < parkedBy + ) { + Thread.sleep(1) + } + assertTrue( + crashThread.isAlive && crashThread.state == Thread.State.TIMED_WAITING, + "Expected the handler to still be blocked inside the crash flush", + ) + assertEquals(1, batches.size, "The crash event cannot have been sent yet") + + gate.countDown() + crashThread.join(TimeUnit.SECONDS.toMillis(5)) + assertFalse( + crashThread.isAlive, + "Expected the handler to return once the crash flush completed", + ) + + // The handler only returns after its flush ran, so the crash batch is already on the wire. + assertEquals(2, batches.size, "Expected the crash event to be flushed before returning") + assertNotNull( + batches[1].findEvent("\$exception"), + "Expected an \$exception event in the crash batch", + ) + + postHog.close() + mockServer.shutdown() + } + @Test fun `enabled works with no remote config present`() { // The server SDK never fetches remote config; the local-only gate must still install. diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index 1a6893d9c..e143aba2d 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -16,6 +16,7 @@ import okhttp3.mockwebserver.MockResponse import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse import org.junit.Assert.assertTrue +import java.util.concurrent.CountDownLatch import java.util.concurrent.Executors import kotlin.test.Test @@ -248,4 +249,79 @@ internal class PostHogMemoryQueueTest { http.shutdown() executor.shutdownAndAwaitTermination() } + + @Test + fun `flushBlocking sends an event whose enqueue has not run yet`() { + val http = createMockHttp(MockResponse().setBody("{}")) + val sut = getSut(http.url("/").toString(), flushAt = 100) + + // Park the single queue thread so the enqueue below is provably still pending while the + // caller flushes inline: the crash-path race, made deterministic without any sleeping. + val blocked = CountDownLatch(1) + executor.execute { blocked.await() } + + sut.add(generateEvent()) + + // The inline flush cannot see the pending enqueue, so it sends nothing. + sut.flush() + assertEquals(0, http.requestCount) + + blocked.countDown() + + // The barrier is queued behind the enqueue on the same single thread, so the send sees it. + assertTrue("Expected the blocking flush to complete", sut.flushBlocking(2_000)) + assertEquals(1, http.requestCount) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `flushBlocking still flushes when the calling thread is already interrupted`() { + val http = createMockHttp(MockResponse().setBody("{}")) + val sut = getSut(http.url("/").toString(), flushAt = 100) + + sut.add(generateEvent()) + + // A crash on a thread some shutdown just interrupted must not lose its flush. + Thread.currentThread().interrupt() + try { + assertTrue("Expected the blocking flush to complete", sut.flushBlocking(2_000)) + assertEquals(1, http.requestCount) + assertTrue( + "Expected the interrupt flag to be restored for the caller", + Thread.currentThread().isInterrupted, + ) + } finally { + // Never leak the interrupt into whatever runs next on this thread. + Thread.interrupted() + } + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `flushBlocking gives up after the timeout instead of hanging`() { + val http = createMockHttp() + val sut = getSut(http.url("/").toString(), flushAt = 100) + + // Occupy the single queue thread so the barrier can never run. + val blocked = CountDownLatch(1) + executor.execute { blocked.await() } + + sut.add(generateEvent()) + + val startedAt = System.nanoTime() + assertFalse("Expected the blocking flush to time out", sut.flushBlocking(200)) + val elapsedMs = (System.nanoTime() - startedAt) / 1_000_000 + + assertTrue("Expected to wait for the timeout, waited $elapsedMs ms", elapsedMs >= 200) + assertTrue("Expected to return right after the timeout, waited $elapsedMs ms", elapsedMs < 2_000) + assertEquals(0, http.requestCount) + + blocked.countDown() + http.shutdown() + executor.shutdownAndAwaitTermination() + } } diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index c16686654..b40cec34c 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -245,10 +245,10 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th // (`logger.error(..., e); throw e`). PostHogCapturedThrowables.markAndCheck(throwable) target.capture(PostHogThrowable(throwable, thread)) - // The queue sends a fatal exception event synchronously on this (the crashing) - // thread, bypassing the flushAt threshold; this flush covers anything else still - // pending. Delivery is still best-effort under an immediate hard exit — same - // guarantee as the Android SDK. + // Depending on the target's queue the capture above may only enqueue the event, so + // this flush is its last chance to reach the network before the process goes down. + // Targets whose enqueue is asynchronous make this a bounded blocking flush (see the + // server SDK's target). Delivery stays best-effort under an immediate hard exit. target.flush() } } From 59a37c1a31391724f04480fc6e80c3ca359e6af3 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 03:47:02 +0300 Subject: [PATCH 04/22] fix(core): keep the ThreadDeath check compiling under newer JDKs ThreadDeath is deprecated for removal since JDK 20 and the project compiles with warnings as errors, so toolchains on JDK 20+ fail the build on the bare reference. Move the check behind a narrowly scoped suppression; the behavior (reproducing ThreadGroup's silence for ThreadDeath) is unchanged. --- .../PostHogErrorTrackingAutoCaptureIntegration.kt | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index b40cec34c..33129036e 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -257,7 +257,7 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th val previousHandler = defaultExceptionHandler if (previousHandler != null) { previousHandler.uncaughtException(thread, throwable) - } else if (throwable !is ThreadDeath) { + } else if (!isThreadDeath(throwable)) { // No previous default handler: reproduce the JVM's built-in crash output that // ThreadGroup would have printed had we not installed ourselves as the default handler, // so opting into capture never hides crashes from stderr log collection. ThreadDeath is @@ -266,4 +266,11 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th throwable.printStackTrace(System.err) } } + + // ThreadDeath is deprecated (for removal) since JDK 20 and this project compiles with warnings + // as errors, so the check lives behind a narrowly scoped suppression (both names, since Kotlin + // reports for-removal deprecations as DEPRECATION_ERROR). It still matters on older JDKs, where + // Thread.stop can throw it and ThreadGroup stays silent for it. + @Suppress("DEPRECATION", "DEPRECATION_ERROR") + private fun isThreadDeath(throwable: Throwable): Boolean = throwable is ThreadDeath } From 02b3de285afd83d7c5cebd571a8ed005310117d7 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 03:50:36 +0300 Subject: [PATCH 05/22] fix(core): always delegate even when the uncaught capture fails MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit If capture or flush throws inside uncaughtException, the failure used to escape the handler, so the previous default handler (the app's own crash handling) and the stderr fallback never ran — enabling PostHog could replace the application's crash handling. Catch capture-path failures, log them, and always fall through to delegation. --- ...tHogErrorTrackingAutoCaptureIntegration.kt | 30 +++++++++++-------- ...ErrorTrackingAutoCaptureIntegrationTest.kt | 22 ++++++++++++++ 2 files changed, 40 insertions(+), 12 deletions(-) diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index 33129036e..8f4a48c5b 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -238,18 +238,24 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th ) { if (captureEnabled) { captureTarget?.let { target -> - // Mark the throwable so post-crash log mirrors of this exact instance (e.g. a - // shutdown hook logging the crash) don't re-report it — but never skip the capture - // itself: this is the authoritative fatal/unhandled record for the crash and must not - // be downgraded by an earlier handled capture of the same instance - // (`logger.error(..., e); throw e`). - PostHogCapturedThrowables.markAndCheck(throwable) - target.capture(PostHogThrowable(throwable, thread)) - // Depending on the target's queue the capture above may only enqueue the event, so - // this flush is its last chance to reach the network before the process goes down. - // Targets whose enqueue is asynchronous make this a bounded blocking flush (see the - // server SDK's target). Delivery stays best-effort under an immediate hard exit. - target.flush() + // Telemetry must never replace the application's own crash handling: if capture or + // flush throws, log and fall through to the delegation below instead of letting the + // failure escape this handler. + try { + // Mark the throwable so post-crash log mirrors of this exact instance (e.g. a + // shutdown hook logging the crash) don't re-report it — but never skip the capture + // itself: this is the authoritative fatal/unhandled record for the crash and must not + // be downgraded by an earlier handled capture of the same instance + // (`logger.error(..., e); throw e`). + PostHogCapturedThrowables.markAndCheck(throwable) + target.capture(PostHogThrowable(throwable, thread)) + // Depending on the target's queue the capture above may only enqueue the event, so + // this flush is its last chance to reach the network before the process goes down. + // Delivery stays best-effort under an immediate hard exit. + target.flush() + } catch (e: Throwable) { + config.logger.log("Capturing the uncaught exception failed: $e.") + } } } diff --git a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt index ecdab1042..2b53855be 100644 --- a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt +++ b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt @@ -188,6 +188,28 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { integration.uninstall() } + @Test + fun `uncaughtException still delegates to the previous handler when capture itself throws`() { + whenever(mockConfig.remoteConfigHolder).thenReturn(mockRemoteConfig) + whenever(mockRemoteConfig.isAutocaptureExceptionsEnabled()).thenReturn(true) + currentHandler = mockExceptionHandler + whenever(mockPostHog.captureException(any(), anyOrNull())).thenThrow(IllegalStateException("telemetry broke")) + + val thread = Thread.currentThread() + val throwable = RuntimeException("Test exception") + + val integration = getSut() + integration.install(mockPostHog) + + // Regression: a throwing capture/flush used to escape uncaughtException, so the previous + // handler (the app's own crash handling) never ran. + integration.uncaughtException(thread, throwable) + + verify(mockExceptionHandler).uncaughtException(thread, throwable) + + integration.uninstall() + } + @Test fun `onRemoteConfig does nothing when remoteConfigHolder is null`() { whenever(mockConfig.remoteConfigHolder).thenReturn(mockRemoteConfig) From 737f697de8fa86121c0a7682e126aedd2d1a56c9 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 03:58:51 +0300 Subject: [PATCH 06/22] refactor(server): route fatal exception events through a queue fatal path Key the crash path off the same fatal-record marker the core PostHogQueue uses (PostHogEvent.isFatalExceptionEvent) instead of a client-side downcast to flushBlocking: a fatal $exception event's enqueue and send now run as one ordered task on the queue executor, draining batch by batch (ignoring flushAt) until the queue is empty or the bounded timeout is spent, while the crashing thread waits at most FATAL_FLUSH_TIMEOUT_MS. This moves the crash-delivery logic out of PostHog.kt (the capture target now just flushes), narrows the scheduling catch to RejectedExecutionException, and fixes the bounded-batch gap where a backlog of maxBatchSize or more older events left the crash event stranded behind a single capped batch. --- .changeset/server-uncaught-exceptions.md | 2 +- .../main/java/com/posthog/server/PostHog.kt | 29 +---- .../server/internal/PostHogMemoryQueue.kt | 106 ++++++++++++------ .../server/PostHogUncaughtExceptionTest.kt | 10 +- .../server/internal/PostHogMemoryQueueTest.kt | 93 +++++++++++---- 5 files changed, 156 insertions(+), 84 deletions(-) diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index def367b7b..102bda66a 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -2,4 +2,4 @@ 'posthog-server': minor --- -Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. The crash capture is enqueued asynchronously like any other event, and the crash path then performs a bounded blocking flush that is ordered behind that pending enqueue and ignores `flushAt`, so the event gets a network attempt before the JVM exits without ever blocking the crashing thread indefinitely. Delivery stays best-effort, the same guarantee class as the Android SDK: the flush can hit its timeout, the HTTP attempt can fail, and an immediate hard exit can cut it short. Sending the fatal event itself synchronously and independently of the queue threshold is a known follow-up. +Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index 4a92f6c5d..f0b39bcb3 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -6,7 +6,6 @@ import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration import com.posthog.internal.FeatureFlag import com.posthog.server.internal.EvaluationsHost import com.posthog.server.internal.PostHogFeatureFlags -import com.posthog.server.internal.PostHogMemoryQueue @Suppress("DEPRECATION") public class PostHog : PostHogStateless(), PostHogInterface { @@ -62,7 +61,11 @@ public class PostHog : PostHogStateless(), PostHogInterface { getConfig()?.let { coreConfig -> val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; - // routing it through captureException preserves those via the shared coercer. + // routing it through captureException preserves those via the shared coercer. A + // fatal-level event takes PostHogMemoryQueue's bounded blocking fatal path inside + // add() (same fatal-record marker the core queue keys on), so capture() itself + // delivers the crash before returning; the flush below is just a best-effort sweep + // of whatever else is still queued. integration.installWith( object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { override fun capture(throwable: Throwable) { @@ -70,21 +73,7 @@ public class PostHog : PostHogStateless(), PostHogInterface { } override fun flush() { - // Crash path only: capture() above enqueues on the queue executor, and the - // regular flush() runs inline on this (the crashing) thread — it would - // usually read the queue before that enqueue landed, send nothing, and let - // the event die with the JVM. flushBlocking orders the drain behind the - // pending enqueue and blocks the crashing thread for at most - // CRASH_FLUSH_TIMEOUT_MS, like the Rust SDK's bounded panic-hook flush. - // The server client always runs a PostHogMemoryQueue - // (PostHogConfig.asCoreConfig), so the fallback is unreachable in practice - // and only keeps the crash flush from silently becoming a no-op. - val memoryQueue = this@PostHog.queue as? PostHogMemoryQueue - if (memoryQueue != null) { - memoryQueue.flushBlocking(CRASH_FLUSH_TIMEOUT_MS) - } else { - this@PostHog.flush() - } + this@PostHog.flush() } }, ) @@ -475,12 +464,6 @@ public class PostHog : PostHogStateless(), PostHogInterface { } public companion object { - /** - * How long the crashing thread waits for the crash event's flush to reach the network attempt - * before it delegates to the next handler and lets the JVM go down. - */ - private const val CRASH_FLUSH_TIMEOUT_MS = 2_000L - /** * Sets up the SDK and returns an instance that you can hold and pass around. * diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 81027caeb..36b03e116 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -14,6 +14,7 @@ import java.util.Timer import java.util.TimerTask import java.util.concurrent.CountDownLatch import java.util.concurrent.ExecutorService +import java.util.concurrent.RejectedExecutionException import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicBoolean import kotlin.concurrent.schedule @@ -33,6 +34,7 @@ internal class PostHogMemoryQueue( private val executor: ExecutorService, private val retryDelaySeconds: Int = DEFAULT_RETRY_DELAY_SECONDS, private val maxRetryDelaySeconds: Int = DEFAULT_MAX_RETRY_DELAY_SECONDS, + private val fatalFlushTimeoutMs: Long = FATAL_FLUSH_TIMEOUT_MS, ) : PostHogQueueInterface { private val events: ArrayDeque = ArrayDeque() private val eventsLock = Any() @@ -51,25 +53,35 @@ internal class PostHogMemoryQueue( private val delay: Long get() = (config.flushIntervalSeconds * 1000).toLong() override fun add(record: PostHogEvent) { - executor.executeSafely { - var removedEvent: PostHogEvent? = null + // Same fatal-record marker the core PostHogQueue keys on: a fatal $exception event is about + // to take the process down with it, so it must not take the regular async path. + if (record.isFatalExceptionEvent()) { + addFatalBlocking(record) + return + } - synchronized(eventsLock) { - if (events.size >= config.maxQueueSize) { - removedEvent = events.removeFirstOrNull() - } + executor.executeSafely { + enqueue(record) + flushIfOverThreshold() + } + } - events.addLast(record) - } + private fun enqueue(record: PostHogEvent) { + var removedEvent: PostHogEvent? = null - if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + synchronized(eventsLock) { + if (events.size >= config.maxQueueSize) { + removedEvent = events.removeFirstOrNull() } - config.logger.log("Event: ${record.event} was added to the queue.") + events.addLast(record) + } - flushIfOverThreshold() + if (removedEvent != null) { + config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") } + + config.logger.log("Event: ${record.event} was added to the queue.") } override fun flush() { @@ -82,42 +94,60 @@ internal class PostHogMemoryQueue( } /** - * Bounded blocking flush for the crash path: submits a barrier task onto the queue executor and - * waits up to [timeoutMs] for that task to send the pending batch. - * - * [add] enqueues on the executor, so a caller that flushes inline (see [flush]) can read the - * deque before its own event landed and send nothing — fatal on a crashing thread whose process - * is about to exit, taking the daemon queue thread with it. The executor is single-threaded - * (owned by `PostHogStateless`), so a barrier submitted after [add] runs strictly after that - * enqueue instead of racing it. + * Crash path for fatal `$exception` events: enqueue and drain in one ordered task on the queue + * executor, blocking the calling (crashing) thread for at most [fatalFlushTimeoutMs]. * - * The send runs on the executor thread and ignores `flushAt` (a crash must not wait for the - * threshold), so the caller blocks on the barrier only, never longer than [timeoutMs]. + * The regular [add] path submits the enqueue asynchronously, so a crashing thread that flushed + * inline could read the deque before its own event landed, send nothing, and let the event die + * with the JVM. Running enqueue + drain as one task on the single-threaded executor makes the + * send happen-after the enqueue, and draining batch by batch (ignoring `flushAt`) until the + * queue is empty keeps a backlog of `maxBatchSize` or more from stranding the fatal event, + * which FIFO puts last. * - * @return true when the barrier ran within the timeout. Delivery itself stays best-effort: the - * batch is capped at `maxBatchSize`, another flush already in progress (the periodic timer runs - * on its own thread) makes the barrier a no-op, and the HTTP attempt can still fail or be cut - * short by process exit. False means the barrier could not be scheduled (executor shut down) or - * did not run in time. + * Delivery stays best-effort: the caller stops waiting at the timeout (the drain keeps going on + * the executor thread for whatever process lifetime remains), and each HTTP attempt can fail or + * be cut short by process exit. */ - fun flushBlocking(timeoutMs: Long): Boolean { - val drained = CountDownLatch(1) + private fun addFatalBlocking(record: PostHogEvent) { + val done = CountDownLatch(1) + val deadlineNanos = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(fatalFlushTimeoutMs) try { executor.execute { try { - flushIgnoringThreshold() + enqueue(record) + drainUntilDeadline(deadlineNanos) } finally { - drained.countDown() + done.countDown() } } - } catch (e: Throwable) { - // RejectedExecutionException and friends: the executor is gone, there is nothing to await - config.logger.log("Blocking flush could not be scheduled: $e.") - return false + } catch (e: RejectedExecutionException) { + // the executor is gone (client closed mid-crash); there is nothing to await + config.logger.log("The fatal event flush could not be scheduled: $e.") + return } - return awaitUninterruptibly(drained, timeoutMs) + awaitUninterruptibly(done, fatalFlushTimeoutMs) + } + + /** + * Sends batch after batch until the queue is empty, the [deadlineNanos] budget is spent, or a + * pass makes no progress (a failed send requeues its batch, and a flush already in progress on + * another thread makes the pass a no-op) — a crashing process gets one straight-line attempt + * per batch, never a retry loop. + */ + private fun drainUntilDeadline(deadlineNanos: Long) { + while (System.nanoTime() < deadlineNanos) { + val before = synchronized(eventsLock) { events.size } + if (before == 0) { + return + } + flushIgnoringThreshold() + val after = synchronized(eventsLock) { events.size } + if (after >= before) { + return + } + } } /** @@ -334,5 +364,9 @@ internal class PostHogMemoryQueue( public companion object { private const val DEFAULT_RETRY_DELAY_SECONDS = 5 private const val DEFAULT_MAX_RETRY_DELAY_SECONDS = 60 + + // How long a crashing thread waits for the fatal enqueue + drain, mirroring the Rust SDK's + // bounded panic-hook flush. Bounded so telemetry can never hang a dying process. + internal const val FATAL_FLUSH_TIMEOUT_MS = 2_000L } } diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt index afd588db7..96f3c0674 100644 --- a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt @@ -260,10 +260,10 @@ internal class PostHogUncaughtExceptionTest { @Test fun `crash event is flushed even while its enqueue is still pending`() { // Makes the crash-path race deterministic instead of hoping for a scheduling order. The queue - // thread is parked inside the gated warmup batch, so the crash capture's enqueue is provably + // thread is parked inside the gated warmup batch, so the crash capture's fatal task is provably // still pending when the handler flushes, and with flushAt above one and a 600s flush interval // nothing else can deliver the crash event. The handler runs on its own thread so the test can - // assert it is still blocked inside the flush while the enqueue is pending — the inline flush + // assert it is still blocked inside the capture while the task is pending — the inline flush // it used to do returned immediately, having read an empty queue, and the event died with the // JVM. val batches = CopyOnWriteArrayList() @@ -317,9 +317,9 @@ internal class PostHogUncaughtExceptionTest { start() } - // The only timed wait on this thread's crash path is the blocking flush's bounded await, so - // TIMED_WAITING while the queue thread is parked means the barrier is queued behind the - // pending enqueue and the handler is waiting for it. The old inline flush never blocked: the + // The only timed wait on this thread's crash path is the fatal add's bounded await, so + // TIMED_WAITING while the queue thread is parked means the enqueue-and-drain task is queued + // behind the parked warmup flush. The old inline flush never blocked: the // thread just finishes, and this assertion fails. val parkedBy = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) while (crashThread.isAlive && diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index e143aba2d..7e4eb4762 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -1,6 +1,7 @@ package com.posthog.server.internal import com.posthog.PostHogConfig +import com.posthog.PostHogEvent import com.posthog.internal.PostHogApi import com.posthog.internal.PostHogApiEndpoint import com.posthog.internal.PostHogDateProvider @@ -32,6 +33,7 @@ internal class PostHogMemoryQueueTest { maxBatchSize: Int = 50, networkStatus: PostHogNetworkStatus? = null, retryDelaySeconds: Int = 5, + fatalFlushTimeoutMs: Long = 2_000L, ): PostHogMemoryQueue { val config = PostHogConfig("some_api_key", host).apply { @@ -49,9 +51,18 @@ internal class PostHogMemoryQueueTest { PostHogApiEndpoint.BATCH, executor = executor, retryDelaySeconds = retryDelaySeconds, + fatalFlushTimeoutMs = fatalFlushTimeoutMs, ) } + // A fatal $exception event, i.e. one PostHogEvent.isFatalExceptionEvent() marks for the + // blocking crash path in add(). + private fun generateFatalEvent(): PostHogEvent { + val event = generateEvent("\$exception") + event.properties?.put("\$exception_level", "fatal") + return event + } + @Test fun `adds a single event`() { val http = createMockHttp() @@ -251,42 +262,88 @@ internal class PostHogMemoryQueueTest { } @Test - fun `flushBlocking sends an event whose enqueue has not run yet`() { + fun `a fatal exception event is sent before add returns`() { val http = createMockHttp(MockResponse().setBody("{}")) val sut = getSut(http.url("/").toString(), flushAt = 100) - // Park the single queue thread so the enqueue below is provably still pending while the - // caller flushes inline: the crash-path race, made deterministic without any sleeping. + // add() blocks on the fatal path, so the request must already be there when it returns — + // no executor await, no waiting takeRequest. + sut.add(generateFatalEvent()) + + assertEquals(1, http.requestCount) + val body = http.takeRequest().body.unGzip() + assertTrue("Body should contain the fatal exception event", body.contains("\$exception")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `the fatal path is ordered behind a pending enqueue instead of racing it`() { + val http = createMockHttp(MockResponse().setBody("{}")) + val sut = getSut(http.url("/").toString(), flushAt = 100) + + // Park the single queue thread so the enqueue below is provably still pending: the + // crash-path race, made deterministic without any sleeping. val blocked = CountDownLatch(1) executor.execute { blocked.await() } - sut.add(generateEvent()) + sut.add(generateEvent("earlier_event")) - // The inline flush cannot see the pending enqueue, so it sends nothing. - sut.flush() + // The fatal add has to wait for the parked thread, so nothing can have been sent yet. + val sender = Thread { sut.add(generateFatalEvent()) } + sender.start() assertEquals(0, http.requestCount) blocked.countDown() + sender.join(5_000) + assertFalse("Expected the fatal add to return once the queue drained", sender.isAlive) - // The barrier is queued behind the enqueue on the same single thread, so the send sees it. - assertTrue("Expected the blocking flush to complete", sut.flushBlocking(2_000)) - assertEquals(1, http.requestCount) + // Both the pending earlier event and the fatal event went out. + val body = http.takeRequest().body.unGzip() + assertTrue("Body should contain the earlier pending event", body.contains("earlier_event")) + assertTrue("Body should contain the fatal exception event", body.contains("\$exception")) http.shutdown() executor.shutdownAndAwaitTermination() } @Test - fun `flushBlocking still flushes when the calling thread is already interrupted`() { + fun `the fatal path drains a backlog larger than maxBatchSize so the crash event is not stranded`() { + // 5 older events + the fatal one at maxBatchSize=2 need 3 sequential batches; a single + // bounded batch would send only the oldest 2 and strand the crash event (FIFO puts it last). + val http = + createMockHttp( + MockResponse().setBody("{}"), + MockResponse().setBody("{}"), + MockResponse().setBody("{}"), + ) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxBatchSize = 2) + + repeat(5) { sut.add(generateEvent("backlog_event_$it")) } + executor.awaitExecution() + assertEquals(0, http.requestCount) + + sut.add(generateFatalEvent()) + + assertEquals(3, http.requestCount) + val lastBody = + (1..3).joinToString("\n") { http.takeRequest().body.unGzip() } + assertTrue("The crash event must be part of the drained batches", lastBody.contains("\$exception")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `the fatal path still flushes when the calling thread is already interrupted`() { val http = createMockHttp(MockResponse().setBody("{}")) val sut = getSut(http.url("/").toString(), flushAt = 100) - sut.add(generateEvent()) - // A crash on a thread some shutdown just interrupted must not lose its flush. Thread.currentThread().interrupt() try { - assertTrue("Expected the blocking flush to complete", sut.flushBlocking(2_000)) + sut.add(generateFatalEvent()) assertEquals(1, http.requestCount) assertTrue( "Expected the interrupt flag to be restored for the caller", @@ -302,18 +359,16 @@ internal class PostHogMemoryQueueTest { } @Test - fun `flushBlocking gives up after the timeout instead of hanging`() { + fun `the fatal path gives up after the timeout instead of hanging the crashing thread`() { val http = createMockHttp() - val sut = getSut(http.url("/").toString(), flushAt = 100) + val sut = getSut(http.url("/").toString(), flushAt = 100, fatalFlushTimeoutMs = 200) - // Occupy the single queue thread so the barrier can never run. + // Occupy the single queue thread so the fatal enqueue-and-drain task can never run. val blocked = CountDownLatch(1) executor.execute { blocked.await() } - sut.add(generateEvent()) - val startedAt = System.nanoTime() - assertFalse("Expected the blocking flush to time out", sut.flushBlocking(200)) + sut.add(generateFatalEvent()) val elapsedMs = (System.nanoTime() - startedAt) / 1_000_000 assertTrue("Expected to wait for the timeout, waited $elapsedMs ms", elapsedMs >= 200) From 277db0d0d2fd5ed3819433994b8376ee009851f4 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:00:28 +0300 Subject: [PATCH 07/22] refactor(core): move the captured-throwables guard to the logback PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PostHogCapturedThrowables has no consumer in this PR — the consulting side is the posthog-server-logback appender's log-mirror dedup. Move the guard, its marking call, its tests and its changeset to that stacked PR so the guard lands together with its consumer. --- .changeset/core-captured-throwables.md | 5 -- posthog/api/posthog.api | 5 -- ...tHogErrorTrackingAutoCaptureIntegration.kt | 7 -- .../PostHogCapturedThrowables.kt | 69 ------------------- ...ErrorTrackingAutoCaptureIntegrationTest.kt | 38 ---------- .../PostHogCapturedThrowablesTest.kt | 37 ---------- 6 files changed, 161 deletions(-) delete mode 100644 .changeset/core-captured-throwables.md delete mode 100644 posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt delete mode 100644 posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt diff --git a/.changeset/core-captured-throwables.md b/.changeset/core-captured-throwables.md deleted file mode 100644 index 5dd7eb2ed..000000000 --- a/.changeset/core-captured-throwables.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'posthog': patch ---- - -Add an internal process-wide `PostHogCapturedThrowables` guard (marked `@PostHogInternal`, visible only because of the multi-module architecture) that lets independent error-capture paths avoid double-reporting the same `Throwable` instance. The guard is directional: log-mirror paths (e.g. the `posthog-server-logback` appender) consult it and skip instances already reported, while the uncaught-exception handler only marks — a crash is always captured as the authoritative fatal/unhandled record even if the same instance was logged first, and marking it keeps post-crash log mirrors from reporting it again. Membership is keyed on instance identity and held weakly, so the guard never keeps a throwable or its stack alive. diff --git a/posthog/api/posthog.api b/posthog/api/posthog.api index 7d31d5332..9d60d3cf4 100644 --- a/posthog/api/posthog.api +++ b/posthog/api/posthog.api @@ -1177,11 +1177,6 @@ public final class com/posthog/internal/VariantDefinition { public final fun getRolloutPercentage ()D } -public final class com/posthog/internal/errortracking/PostHogCapturedThrowables { - public static final field INSTANCE Lcom/posthog/internal/errortracking/PostHogCapturedThrowables; - public final fun markAndCheck (Ljava/lang/Throwable;)Z -} - public final class com/posthog/internal/errortracking/ThrowableCoercer { public static final field EXCEPTION_LEVEL_ATTRIBUTE Ljava/lang/String; public static final field EXCEPTION_LEVEL_FATAL Ljava/lang/String; diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index 8f4a48c5b..6e9fe3ffa 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -4,7 +4,6 @@ import com.posthog.PostHogConfig import com.posthog.PostHogIntegration import com.posthog.PostHogInterface import com.posthog.PostHogInternal -import com.posthog.internal.errortracking.PostHogCapturedThrowables import com.posthog.internal.errortracking.PostHogThrowable import com.posthog.internal.errortracking.UncaughtExceptionHandlerAdapter import java.util.concurrent.atomic.AtomicBoolean @@ -242,12 +241,6 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th // flush throws, log and fall through to the delegation below instead of letting the // failure escape this handler. try { - // Mark the throwable so post-crash log mirrors of this exact instance (e.g. a - // shutdown hook logging the crash) don't re-report it — but never skip the capture - // itself: this is the authoritative fatal/unhandled record for the crash and must not - // be downgraded by an earlier handled capture of the same instance - // (`logger.error(..., e); throw e`). - PostHogCapturedThrowables.markAndCheck(throwable) target.capture(PostHogThrowable(throwable, thread)) // Depending on the target's queue the capture above may only enqueue the event, so // this flush is its last chance to reach the network before the process goes down. diff --git a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt deleted file mode 100644 index 2c1aae9f0..000000000 --- a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogCapturedThrowables.kt +++ /dev/null @@ -1,69 +0,0 @@ -package com.posthog.internal.errortracking - -import com.posthog.PostHogInternal -import java.lang.ref.ReferenceQueue -import java.lang.ref.WeakReference - -/** - * Process-wide guard that lets independent error-capture paths avoid double-reporting the very same - * [Throwable] instance. The guard is directional: log-mirror paths (the Logback appender) consult it - * and skip instances already reported, while the uncaught-exception handler only marks — a crash is - * always captured as the authoritative fatal/unhandled record, even if the same instance was logged - * first, and marking it prevents post-crash log mirrors from reporting it again. - * - * Membership is keyed strictly on **instance identity** (reference equality, not `equals`/`hashCode` - * — a `Throwable` subclass with value equality must not make two distinct instances collide) and - * held weakly, so entries disappear once the throwable is otherwise unreachable: the guard never - * keeps a throwable (or its stack) alive. - * - * Not part of the public API; visible only because of the multi-module architecture. - */ -@PostHogInternal -public object PostHogCapturedThrowables { - private val queue = ReferenceQueue() - - // Identity keys of throwables already captured. HashSet is not thread-safe, so all access is - // synchronized on the set. Cleared entries are pruned opportunistically via [queue]. - private val seen = HashSet() - - /** - * Records [throwable] as captured and reports whether the caller should capture it. - * - * @return true if this is the first time the instance has been marked (the caller should - * capture it), false if it was already marked (the caller should skip it). - */ - public fun markAndCheck(throwable: Throwable): Boolean = - synchronized(seen) { - pruneCleared() - seen.add(IdentityWeakKey(throwable, queue)) - } - - // Drop keys whose referent has been collected. Must be called while holding the lock. - private fun pruneCleared() { - while (true) { - val cleared = queue.poll() ?: break - seen.remove(cleared) - } - } - - // Weak reference whose identity is the referential identity of the referent, so distinct - // instances never collide even if their class overrides equals/hashCode. hashCode is captured - // eagerly (identity hash is stable) so a key still matches after its referent is cleared. - private class IdentityWeakKey( - referent: Throwable, - queue: ReferenceQueue, - ) : WeakReference(referent, queue) { - private val identityHash = System.identityHashCode(referent) - - override fun hashCode(): Int = identityHash - - override fun equals(other: Any?): Boolean { - if (this === other) return true - if (other !is IdentityWeakKey) return false - val self = get() - // Reference equality on the referents; a cleared referent only matches itself (handled - // by the identity check above), which is fine — such keys are pruned via the queue. - return self != null && self === other.get() - } - } -} diff --git a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt index 2b53855be..7a5bc3057 100644 --- a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt +++ b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt @@ -4,7 +4,6 @@ import com.posthog.PostHogConfig import com.posthog.PostHogInterface import com.posthog.internal.PostHogPrintLogger import com.posthog.internal.PostHogRemoteConfig -import com.posthog.internal.errortracking.PostHogCapturedThrowables import com.posthog.internal.errortracking.PostHogThrowable import com.posthog.internal.errortracking.UncaughtExceptionHandlerAdapter import org.mockito.kotlin.any @@ -618,43 +617,6 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { } } - @Test - fun `uncaughtException captures even when the throwable was already captured elsewhere`() { - val target = RecordingTarget() - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } - integration.installWith(target) - - val alreadyLogged = RuntimeException("logged then crashed") - // Simulate the appender having captured this exact instance first. - assertEquals(true, PostHogCapturedThrowables.markAndCheck(alreadyLogged)) - - integration.uncaughtException(Thread.currentThread(), alreadyLogged) - - // The crash is the authoritative fatal/unhandled record: it must be captured even though - // the instance was already reported as a handled log capture, and the queue is flushed so - // both events leave before the process exits. - assertEquals(1, target.captured.size, "The crash capture must not be suppressed by dedup") - assertEquals(1, target.flushCount, "flush() must run on the crash path") - - integration.uninstall() - } - - @Test - fun `uncaughtException marks the throwable so later log mirrors dedup against it`() { - val target = RecordingTarget() - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } - integration.installWith(target) - - val crash = RuntimeException("crashed then logged") - integration.uncaughtException(Thread.currentThread(), crash) - - assertEquals(1, target.captured.size) - // A post-crash log mirror consulting the guard must see the instance as already captured. - assertEquals(false, PostHogCapturedThrowables.markAndCheck(crash)) - - integration.uninstall() - } - @Test fun `uncaughtException reproduces the JVM default crash output when no previous handler exists`() { currentHandler = null diff --git a/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt b/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt deleted file mode 100644 index 9d1d1dda7..000000000 --- a/posthog/src/test/java/com/posthog/internal/errortracking/PostHogCapturedThrowablesTest.kt +++ /dev/null @@ -1,37 +0,0 @@ -package com.posthog.internal.errortracking - -import kotlin.test.Test -import kotlin.test.assertFalse -import kotlin.test.assertTrue - -internal class PostHogCapturedThrowablesTest { - @Test - fun `first mark returns true, repeat mark of same instance returns false`() { - val throwable = RuntimeException("boom") - - assertTrue(PostHogCapturedThrowables.markAndCheck(throwable), "first sighting should be captured") - assertFalse(PostHogCapturedThrowables.markAndCheck(throwable), "same instance should be deduped") - } - - @Test - fun `distinct instances that are equal by value are both captured`() { - // A Throwable subclass with value equality must NOT make two distinct instances collide: - // dedup is strictly instance-identity based. - val a = ValueEqualThrowable("same") - val b = ValueEqualThrowable("same") - - // Sanity: they are equal by value but distinct instances. - assertTrue(a == b) - assertFalse(a === b) - - assertTrue(PostHogCapturedThrowables.markAndCheck(a), "first instance captured") - assertTrue(PostHogCapturedThrowables.markAndCheck(b), "second, value-equal instance must still be captured") - assertFalse(PostHogCapturedThrowables.markAndCheck(a), "re-marking the first instance is still deduped") - } - - private class ValueEqualThrowable(private val token: String) : RuntimeException(token) { - override fun equals(other: Any?): Boolean = other is ValueEqualThrowable && other.token == token - - override fun hashCode(): Int = token.hashCode() - } -} From c50cd288c9a5e13babe94689faa34d9b2903e822 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:05:41 +0300 Subject: [PATCH 08/22] fix(core): use the canonical onuncaughtexception mechanism and name the capture hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per the sdk-specs exception-event-metadata spec, mechanism.type names the semantic capture-boundary category — the canonical one for an uncaught handler is onuncaughtexception, not the noncanonical UncaughtExceptionHandler — and the concrete integration hook belongs in event-level $exception_source. The server SDK now stamps jvm.uncaught_exception_handler on its uncaught captures, following the lowercase . convention. --- .changeset/core-uncaught-gate.md | 2 +- .changeset/server-uncaught-exceptions.md | 2 +- .../src/main/java/com/posthog/server/PostHog.kt | 14 +++++++++++++- .../posthog/server/PostHogUncaughtExceptionTest.kt | 9 +++++++-- .../internal/errortracking/PostHogThrowable.kt | 4 +++- posthog/src/test/java/com/posthog/PostHogTest.kt | 2 +- 6 files changed, 26 insertions(+), 7 deletions(-) diff --git a/.changeset/core-uncaught-gate.md b/.changeset/core-uncaught-gate.md index 17a4d68b5..c9c8a9f8f 100644 --- a/.changeset/core-uncaught-gate.md +++ b/.changeset/core-uncaught-gate.md @@ -2,4 +2,4 @@ 'posthog': patch --- -`PostHogErrorTrackingAutoCaptureIntegration` can now be gated on a caller-supplied strategy instead of the built-in gate (local `errorTrackingConfig.autoCapture` with remote config as a kill-switch): a new `PostHogErrorTrackingAutoCaptureIntegration(config, enabledGate)` constructor lets SDK layers that never fetch remote config (e.g. the server SDK) decide autocapture purely from local config. The uncaught handler also delivers captures through an internal `CaptureTarget` seam (`installWith`) so it can drive clients that are not a core `PostHogInterface`, and when no previous default handler exists it now reproduces the JVM's own `Exception in thread ...` stderr output, so installing capture never hides a crash from log collection. Android behavior and the existing `install(PostHogInterface)` path are otherwise unchanged; the additions are internal (`@PostHogInternal`) and visible only because of the multi-module architecture. +`PostHogErrorTrackingAutoCaptureIntegration` can now be gated on a caller-supplied strategy instead of the built-in gate (local `errorTrackingConfig.autoCapture` with remote config as a kill-switch): a new `PostHogErrorTrackingAutoCaptureIntegration(config, enabledGate)` constructor lets SDK layers that never fetch remote config (e.g. the server SDK) decide autocapture purely from local config. The uncaught handler also delivers captures through an internal `CaptureTarget` seam (`installWith`) so it can drive clients that are not a core `PostHogInterface`, and when no previous default handler exists it now reproduces the JVM's own `Exception in thread ...` stderr output, so installing capture never hides a crash from log collection. Uncaught-exception events now carry the canonical `mechanism.type` `onuncaughtexception` (previously the noncanonical `UncaughtExceptionHandler`), per the sdk-specs exception-event-metadata spec. Android behavior and the existing `install(PostHogInterface)` path are otherwise unchanged; the additions are internal (`@PostHogInternal`) and visible only because of the multi-module architecture. diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index 102bda66a..9010aa86f 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -2,4 +2,4 @@ 'posthog-server': minor --- -Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as a fatal, unhandled `$exception` event (mechanism `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. +Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as an unhandled `$exception` event (mechanism `onuncaughtexception`, `$exception_source: jvm.uncaught_exception_handler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index f0b39bcb3..e094a6be0 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -69,7 +69,14 @@ public class PostHog : PostHogStateless(), PostHogInterface { integration.installWith( object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { override fun capture(throwable: Throwable) { - captureException(throwable) + // $exception_source names the concrete runtime hook per the sdk-specs + // convention (.); the mechanism category + // (onuncaughtexception) rides on the PostHogThrowable. + captureException( + throwable, + null, + mapOf(EXCEPTION_SOURCE_ATTRIBUTE to EXCEPTION_SOURCE_UNCAUGHT_HANDLER), + ) } override fun flush() { @@ -464,6 +471,11 @@ public class PostHog : PostHogStateless(), PostHogInterface { } public companion object { + // Event-level capture-integration identity for the uncaught handler, following the + // sdk-specs lowercase . convention. + private const val EXCEPTION_SOURCE_ATTRIBUTE = "\$exception_source" + private const val EXCEPTION_SOURCE_UNCAUGHT_HANDLER = "jvm.uncaught_exception_handler" + /** * Sets up the SDK and returns an instance that you can hold and pass around. * diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt index 96f3c0674..198f7ed02 100644 --- a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt @@ -149,9 +149,14 @@ internal class PostHogUncaughtExceptionTest { assertNotNull(mechanism, "Expected a mechanism on the first exception item") assertEquals(false, mechanism["handled"], "Uncaught exceptions must be marked handled=false") assertEquals( - "UncaughtExceptionHandler", + "onuncaughtexception", mechanism["type"], - "Uncaught exceptions must carry the UncaughtExceptionHandler mechanism", + "Uncaught exceptions must carry the canonical onuncaughtexception mechanism", + ) + assertEquals( + "jvm.uncaught_exception_handler", + props["\$exception_source"], + "Uncaught exceptions must name the concrete runtime hook in \$exception_source", ) postHog.close() diff --git a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt index 8223502b0..2db54d36e 100644 --- a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt +++ b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt @@ -3,5 +3,7 @@ package com.posthog.internal.errortracking internal class PostHogThrowable(throwable: Throwable, val thread: Thread = Thread.currentThread()) : Throwable(throwable) { val handled: Boolean = false val isFatal: Boolean = true - val mechanism: String = "UncaughtExceptionHandler" + // Canonical capture-boundary category from the sdk-specs exception-event-metadata spec; the + // concrete hook goes into the event-level $exception_source, not in here. + val mechanism: String = "onuncaughtexception" } diff --git a/posthog/src/test/java/com/posthog/PostHogTest.kt b/posthog/src/test/java/com/posthog/PostHogTest.kt index b3d391da0..659bab4e7 100644 --- a/posthog/src/test/java/com/posthog/PostHogTest.kt +++ b/posthog/src/test/java/com/posthog/PostHogTest.kt @@ -2912,7 +2912,7 @@ internal class PostHogTest { val mechanism = mainException["mechanism"] as Map<*, *> assertEquals(false, mechanism["handled"]) assertEquals(false, mechanism["synthetic"]) - assertEquals("UncaughtExceptionHandler", mechanism["type"]) + assertEquals("onuncaughtexception", mechanism["type"]) // A single-item list carries no chain ids at all. assertFalse(mechanism.containsKey("exception_id")) assertFalse(mechanism.containsKey("parent_id")) From e308994125f93276dbab37f670d3254b8e285ff8 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:14:16 +0300 Subject: [PATCH 09/22] fix(server): capture worker-thread uncaught exceptions as error, not fatal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the JVM an uncaught exception kills only its thread; the process continues unless the crashing thread is main. Per the sdk-specs boundary table, fatal is reserved for boundaries expected to terminate the process, so the integration now takes a layer-supplied per-thread fatal policy: the server SDK marks a main-thread crash fatal (blocking bounded delivery) and any other thread's error (regular async delivery, handled=false either way). Android keeps the always-fatal default, which is correct there — any uncaught exception kills the app. --- .changeset/server-uncaught-exceptions.md | 2 +- .../main/java/com/posthog/server/PostHog.kt | 11 +++- .../java/com/posthog/server/PostHogConfig.kt | 24 ++++---- .../server/PostHogUncaughtExceptionTest.kt | 58 +++++++++++++++++-- posthog/api/posthog.api | 2 +- ...tHogErrorTrackingAutoCaptureIntegration.kt | 23 ++++++-- .../errortracking/PostHogThrowable.kt | 11 +++- ...ErrorTrackingAutoCaptureIntegrationTest.kt | 33 +++++++++-- 8 files changed, 134 insertions(+), 30 deletions(-) diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index 9010aa86f..3b25dee9c 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -2,4 +2,4 @@ 'posthog-server': minor --- -Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as an unhandled `$exception` event (mechanism `onuncaughtexception`, `$exception_source: jvm.uncaught_exception_handler`), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. +Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as an unhandled `$exception` event (mechanism `onuncaughtexception`, `$exception_source: jvm.uncaught_exception_handler`; `$exception_level` is `fatal` for a main-thread crash and `error` for a worker-thread one, whose death the process survives), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index e094a6be0..c01fbf1b5 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -59,7 +59,8 @@ public class PostHog : PostHogStateless(), PostHogInterface { // is set up again. Server apps use one client per process, so we don't ref-count here. if (config.captureUncaughtExceptions) { getConfig()?.let { coreConfig -> - val integration = PostHogErrorTrackingAutoCaptureIntegration(coreConfig) { true } + val integration = + PostHogErrorTrackingAutoCaptureIntegration(coreConfig, { true }, ::isProcessFatal) // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; // routing it through captureException preserves those via the shared coercer. A // fatal-level event takes PostHogMemoryQueue's bounded blocking fatal path inside @@ -90,6 +91,14 @@ public class PostHog : PostHogStateless(), PostHogInterface { } } + // The JVM cannot tell whether a thread's death will end the process, so this approximates the + // spec's "expected to terminate" boundary with the main thread: an uncaught exception there is + // fatal, while a worker thread's kills only that thread (level error) and the process lives on. + // Id 1 is the initial thread on mainstream JVMs and "main" its conventional name; either match + // counts, since a missed main thread would silently downgrade a real crash. + @Suppress("DEPRECATION") // Thread.getId is deprecated on JDK 19+ but stable while a thread lives + private fun isProcessFatal(thread: Thread): Boolean = thread.id == 1L || thread.name == "main" + override fun close() { // Same lock as setup so the uninstall + field clear cannot race a concurrent setup() that is // still assigning uncaughtExceptionIntegration; super.close re-acquires the reentrant lock. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt index c1e2ed3f0..2c1262e2d 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt @@ -204,20 +204,24 @@ public open class PostHogConfig constructor( * Opt in to capturing uncaught exceptions for the whole JVM as `$exception` events. * * When true, [PostHog] installs a [Thread.defaultUncaughtExceptionHandler] on setup that - * captures the crashing exception (marked fatal, `handled=false`, mechanism - * `UncaughtExceptionHandler`), flushes, and then delegates to the previously registered - * handler. The handler is removed again on [PostHog.close]. + * captures the crashing exception (`handled=false`, mechanism `onuncaughtexception`, + * `$exception_source: jvm.uncaught_exception_handler`), flushes, and then delegates to the + * previously registered handler. The handler is removed again on [PostHog.close]. + * + * A main-thread crash is captured with `$exception_level` `fatal` (the process is expected to + * terminate); an uncaught exception on any other thread kills only that thread, so it is + * captured with level `error` and delivered like a regular event. * * Unlike the Android SDK, this is gated purely on this local flag — the server SDK never * fetches remote config, so no remote toggle is involved. * - * Delivery: the crash capture is enqueued asynchronously like any other event, and the crash path - * then performs a bounded blocking flush — ordered behind that pending enqueue and ignoring - * [flushAt] — so the event gets a network attempt before the JVM exits instead of waiting for the - * periodic flush, without ever blocking the crashing thread indefinitely. Delivery stays - * best-effort, the same guarantee class the Android SDK provides: the flush can hit its timeout, - * the HTTP attempt can fail, and an immediate hard exit can cut it short. See [PostHog] for - * details. + * Delivery for a fatal (main-thread) crash: the event takes a dedicated queue path that + * enqueues and sends in one ordered task, draining the queue batch by batch and ignoring + * [flushAt], while the crashing thread waits on it for a bounded timeout — so the crash gets a + * network attempt before the JVM exits without ever blocking the crashing thread indefinitely. + * Delivery stays best-effort, the same guarantee class the Android SDK provides: the drain can + * hit its timeout, the HTTP attempt can fail, and an immediate hard exit can cut it short. See + * [PostHog] for details. * * Docs https://posthog.com/docs/error-tracking * Defaults to false diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt index 198f7ed02..0891252ab 100644 --- a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionTest.kt @@ -111,10 +111,10 @@ internal class PostHogUncaughtExceptionTest { } @Test - fun `uncaught exception is captured as a fatal, unhandled exception event`() { + fun `uncaught main-thread exception is captured as a fatal, unhandled exception event`() { val mockServer = startServer() - // Default flushAt (100) on purpose: the crash path's blocking flush is what delivers the - // event, so a low threshold would mask whether that flush works at all. + // Default flushAt (100) on purpose: the crash path's blocking fatal delivery is what sends + // the event, so a low threshold would mask whether that path works at all. val postHog = PostHog.with( PostHogConfig.builder(TEST_API_KEY) @@ -126,7 +126,9 @@ internal class PostHogUncaughtExceptionTest { val handler = Thread.getDefaultUncaughtExceptionHandler() assertTrue(handler is PostHogErrorTrackingAutoCaptureIntegration) - handler.uncaughtException(Thread.currentThread(), IllegalStateException("kaboom")) + // The thread argument is data to the handler, so a main-named stand-in exercises the + // process-fatal policy without crashing the suite's real main thread. + handler.uncaughtException(Thread("main"), IllegalStateException("kaboom")) // Already sent when the handler returned, so this should not have to wait. val request = mockServer.takeRequest(1, TimeUnit.SECONDS) @@ -163,6 +165,50 @@ internal class PostHogUncaughtExceptionTest { mockServer.shutdown() } + @Test + fun `uncaught worker-thread exception is captured as an error, since the process survives it`() { + val mockServer = startServer() + val postHog = + PostHog.with( + PostHogConfig.builder(TEST_API_KEY) + .host(mockServer.url("/").toString()) + // A worker-thread capture takes the regular async path, so a threshold of one + // is what delivers it within the test. + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + val handler = Thread.getDefaultUncaughtExceptionHandler() + assertTrue(handler is PostHogErrorTrackingAutoCaptureIntegration) + + handler.uncaughtException(Thread("worker-7"), IllegalStateException("worker kaboom")) + + val request = mockServer.takeRequest(5, TimeUnit.SECONDS) + assertNotNull(request, "Expected the worker crash event to be flushed") + + val batch = request.parseBatch() + val props = batch.eventProperties("\$exception") + assertEquals( + "error", + props["\$exception_level"], + "A worker-thread uncaught exception only kills that thread, so it must not be fatal", + ) + + @Suppress("UNCHECKED_CAST") + val exceptionList = props["\$exception_list"] as? List> + assertNotNull(exceptionList) + + @Suppress("UNCHECKED_CAST") + val mechanism = exceptionList.first()["mechanism"] as? Map + assertNotNull(mechanism) + assertEquals(false, mechanism["handled"], "Escaping the thread still means handled=false") + assertEquals("onuncaughtexception", mechanism["type"]) + + postHog.close() + mockServer.shutdown() + } + @Test fun `repeated setup keeps handler ownership so close still restores the previous handler`() { val sentinel = Thread.UncaughtExceptionHandler { _, _ -> } @@ -316,7 +362,9 @@ internal class PostHogUncaughtExceptionTest { assertTrue(handler is PostHogErrorTrackingAutoCaptureIntegration) val crashThread = Thread { - handler.uncaughtException(Thread.currentThread(), IllegalStateException("kaboom")) + // A main-named stand-in: only a process-fatal crash takes the blocking fatal path + // whose ordering this test pins down. + handler.uncaughtException(Thread("main"), IllegalStateException("kaboom")) }.apply { isDaemon = true start() diff --git a/posthog/api/posthog.api b/posthog/api/posthog.api index 9d60d3cf4..bdbeec0e6 100644 --- a/posthog/api/posthog.api +++ b/posthog/api/posthog.api @@ -533,7 +533,7 @@ public abstract interface annotation class com/posthog/PostHogVisibleForTesting public final class com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration : com/posthog/PostHogIntegration, java/lang/Thread$UncaughtExceptionHandler { public fun (Lcom/posthog/PostHogConfig;)V - public fun (Lcom/posthog/PostHogConfig;Lkotlin/jvm/functions/Function0;)V + public fun (Lcom/posthog/PostHogConfig;Lkotlin/jvm/functions/Function0;Lkotlin/jvm/functions/Function1;)V public fun install (Lcom/posthog/PostHogInterface;)V public final fun installWith (Lcom/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration$CaptureTarget;)V public fun onRemoteConfig (Z)V diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index 6e9fe3ffa..42386831c 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -20,6 +20,14 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th */ private val enabledGate: () -> Boolean + /** + * Whether an uncaught exception on [Thread] is expected to terminate the process (level + * `fatal`) or only that thread (level `error`). Defaults to always-fatal, which is correct on + * Android where any uncaught exception kills the app; layers where the process survives a + * worker-thread crash (e.g. the server SDK) supply their own policy. + */ + private val fatalPolicy: (Thread) -> Boolean + // @Volatile: read on the crashing thread in uncaughtException with no happens-before edge to // the install()/uninstall() writes; a pre-existing thread could otherwise see a stale null and // skip delegating to the app/system handler. @@ -44,35 +52,42 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th this.config = config this.adapterExceptionHandler = UncaughtExceptionHandlerAdapter.Adapter.getInstance() this.enabledGate = { defaultGate() } + this.fatalPolicy = { true } } /** - * Internal constructor allowing a custom [enabledGate]. Used by SDK layers (e.g. the server SDK) - * that decide autocapture purely from local config without any remote-config round trip. + * Internal constructor allowing a custom [enabledGate] and [fatalPolicy]. Used by SDK layers + * (e.g. the server SDK) that decide autocapture purely from local config without any + * remote-config round trip, and where an uncaught exception does not always terminate the + * process. * * Not part of the public API; visible only because of the multi-module architecture. */ @PostHogInternal - public constructor(config: PostHogConfig, enabledGate: () -> Boolean) { + public constructor(config: PostHogConfig, enabledGate: () -> Boolean, fatalPolicy: (Thread) -> Boolean) { this.config = config this.adapterExceptionHandler = UncaughtExceptionHandlerAdapter.Adapter.getInstance() this.enabledGate = enabledGate + this.fatalPolicy = fatalPolicy } internal constructor(config: PostHogConfig, adapterExceptionHandler: UncaughtExceptionHandlerAdapter) { this.config = config this.adapterExceptionHandler = adapterExceptionHandler this.enabledGate = { defaultGate() } + this.fatalPolicy = { true } } internal constructor( config: PostHogConfig, adapterExceptionHandler: UncaughtExceptionHandlerAdapter, enabledGate: () -> Boolean, + fatalPolicy: (Thread) -> Boolean = { true }, ) { this.config = config this.adapterExceptionHandler = adapterExceptionHandler this.enabledGate = enabledGate + this.fatalPolicy = fatalPolicy } /** @@ -241,7 +256,7 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th // flush throws, log and fall through to the delegation below instead of letting the // failure escape this handler. try { - target.capture(PostHogThrowable(throwable, thread)) + target.capture(PostHogThrowable(throwable, thread, isFatal = fatalPolicy(thread))) // Depending on the target's queue the capture above may only enqueue the event, so // this flush is its last chance to reach the network before the process goes down. // Delivery stays best-effort under an immediate hard exit. diff --git a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt index 2db54d36e..1a46232e7 100644 --- a/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt +++ b/posthog/src/main/java/com/posthog/internal/errortracking/PostHogThrowable.kt @@ -1,8 +1,15 @@ package com.posthog.internal.errortracking -internal class PostHogThrowable(throwable: Throwable, val thread: Thread = Thread.currentThread()) : Throwable(throwable) { +internal class PostHogThrowable( + throwable: Throwable, + val thread: Thread = Thread.currentThread(), + // Whether the uncaught boundary is expected to terminate the process ($exception_level fatal) + // or only the crashing thread ($exception_level error). Defaults to fatal: on Android any + // uncaught exception kills the app; the server SDK supplies a per-thread policy. + val isFatal: Boolean = true, +) : Throwable(throwable) { val handled: Boolean = false - val isFatal: Boolean = true + // Canonical capture-boundary category from the sdk-specs exception-event-metadata spec; the // concrete hook goes into the event-level $exception_source, not in here. val mechanism: String = "onuncaughtexception" diff --git a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt index 7a5bc3057..cab4e715b 100644 --- a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt +++ b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt @@ -290,7 +290,7 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { // local-only gate bypasses them entirely. This is the server SDK's path. currentHandler = null - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) integration.install(mockPostHog) verify(mockAdapter).setDefaultUncaughtExceptionHandler(integration) @@ -300,7 +300,7 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { @Test fun `local-only gate that is false does not install`() { - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { false } + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { false }) integration.install(mockPostHog) verify(mockAdapter, never()).setDefaultUncaughtExceptionHandler(any()) @@ -622,7 +622,7 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { currentHandler = null val target = RecordingTarget() - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) integration.installWith(target) val originalErr = System.err @@ -646,7 +646,7 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { @Test fun `uncaughtException captures and flushes for a fresh throwable`() { val target = RecordingTarget() - val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + val integration = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) integration.installWith(target) integration.uncaughtException(Thread.currentThread(), RuntimeException("fresh")) @@ -657,16 +657,37 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { integration.uninstall() } + @Test + fun `uncaughtException applies the layer-supplied fatal policy per thread`() { + val target = RecordingTarget() + val integration = + PostHogErrorTrackingAutoCaptureIntegration( + mockConfig, + mockAdapter, + { true }, + { thread -> thread.name == "process-owner" }, + ) + integration.installWith(target) + + integration.uncaughtException(Thread("process-owner"), RuntimeException("boom")) + integration.uncaughtException(Thread("worker"), RuntimeException("boom")) + + assertEquals(true, (target.captured[0] as PostHogThrowable).isFatal) + assertEquals(false, (target.captured[1] as PostHogThrowable).isFatal) + + integration.uninstall() + } + @Test fun `uninstall by a non-installing instance does not tear down the installed handler`() { // First instance installs and owns the global handler. currentHandler = mockExceptionHandler - val first = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + val first = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) first.installWith(RecordingTarget()) verify(mockAdapter).setDefaultUncaughtExceptionHandler(first) // Second instance's install is a process-wide no-op (a handler is already installed). - val second = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter) { true } + val second = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) second.installWith(RecordingTarget()) // Closing the second must NOT restore/replace the handler — it never installed. From 3975eedac0f6f71658e9f4e722e23a070d6fa835 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:14:39 +0300 Subject: [PATCH 10/22] docs(server): document single-owner uncaught capture in the config KDoc The process-wide handler is first-owner-wins and does not transfer when the owning client closes; state that in the public captureUncaughtExceptions documentation instead of only in an implementation comment. --- .../src/main/java/com/posthog/server/PostHogConfig.kt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt index 2c1262e2d..6416dc87c 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHogConfig.kt @@ -215,6 +215,12 @@ public open class PostHogConfig constructor( * Unlike the Android SDK, this is gated purely on this local flag — the server SDK never * fetches remote config, so no remote toggle is involved. * + * The JVM has a single process-wide default handler, so capture is single-owner: the first + * client that opts in installs the handler, later opted-in clients do not. Closing the owner + * restores the previous handler — other still-open clients do not take over; capture resumes + * with the next client that is set up with this flag enabled. Server apps normally run one + * client per process, where none of this matters. + * * Delivery for a fatal (main-thread) crash: the event takes a dedicated queue path that * enqueues and sends in one ordered task, draining the queue batch by batch and ignoring * [flushAt], while the crashing thread waits on it for a bounded timeout — so the crash gets a From 259e11e9f4a9f741a9658f392ee76aa8847dd01a Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:27:03 +0300 Subject: [PATCH 11/22] fix(server): make the periodic flush timer a daemon thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The memory queue created its Timer non-daemon, unlike the core PostHogQueue's Timer(true) and the SDK's daemon executor threads, so any set-up client kept the JVM alive after main finished — or crashed — until close() was called. Found by the new forked-JVM crash tests: the main-crash fixture never exited. --- .changeset/server-uncaught-exceptions.md | 2 ++ .../java/com/posthog/server/internal/PostHogMemoryQueue.kt | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index 3b25dee9c..41b837bac 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -3,3 +3,5 @@ --- Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as an unhandled `$exception` event (mechanism `onuncaughtexception`, `$exception_source: jvm.uncaught_exception_handler`; `$exception_level` is `fatal` for a main-thread crash and `error` for a worker-thread one, whose death the process survives), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. + +The queue's periodic flush timer is now a daemon thread (matching the core SDK's queue), so a set-up client no longer keeps a finished — or crashed — JVM alive until `close()` is called. diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 36b03e116..77056b200 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -198,7 +198,10 @@ internal class PostHogMemoryQueue( executor.executeSafely { synchronized(timerLock) { if (timer == null) { - timer = Timer() + // Daemon, matching the core PostHogQueue's Timer(true) and the executor's + // daemon threads: a background flush timer must never keep a finished (or + // crashed) JVM alive until close() is called. + timer = Timer(true) startTimer(delay) config.logger.log("Queue timer started.") } From 2174b3be12e182841935218ba857c5541cfe1452 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:27:36 +0300 Subject: [PATCH 12/22] test(server): prove crash semantics on a real forked JVM The in-process suite invokes the handler directly, which cannot prove real process exit, worker-thread survival, or the stderr the JVM actually emits. Add a CrashFixture main class launched via ProcessBuilder: a main-thread crash exits 1 with the default crash banner on stderr and delivers a fatal event first; a worker-thread crash leaves the process running (exit 0) and delivers an error-level event. --- .../java/com/posthog/server/CrashFixture.kt | 49 ++++++++ .../PostHogUncaughtExceptionSubprocessTest.kt | 110 ++++++++++++++++++ 2 files changed, 159 insertions(+) create mode 100644 posthog-server/src/test/java/com/posthog/server/CrashFixture.kt create mode 100644 posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt diff --git a/posthog-server/src/test/java/com/posthog/server/CrashFixture.kt b/posthog-server/src/test/java/com/posthog/server/CrashFixture.kt new file mode 100644 index 000000000..bb85adcb6 --- /dev/null +++ b/posthog-server/src/test/java/com/posthog/server/CrashFixture.kt @@ -0,0 +1,49 @@ +package com.posthog.server + +/** + * Forked-JVM fixture for [PostHogUncaughtExceptionSubprocessTest]: a real client with uncaught + * capture enabled crashing for real, so process exit, worker-thread survival and stderr output can + * be asserted on an actual JVM instead of a directly invoked handler. + * + * Args: ` ` where scenario is `main-crash` or `worker-crash`. + */ +public object CrashFixture { + @JvmStatic + public fun main(args: Array) { + val host = args[0] + val scenario = args[1] + + val postHog = + PostHog.with( + PostHogConfig.builder("fixture_api_key") + .host(host) + // A worker-thread capture takes the regular async path; a threshold of one lets + // the fixture deliver it without waiting for the periodic flush. + .flushAt(1) + .captureUncaughtExceptions(true) + .build(), + ) + + when (scenario) { + // The real main thread dies: the handler must deliver the fatal event before the JVM + // exits (code 1) and reproduce the default crash banner on stderr. + "main-crash" -> throw IllegalStateException("fixture main crash") + + // Only the worker dies: the process must survive, the capture is level error, and the + // fixture exits 0 after flushing. + "worker-crash" -> { + val worker = Thread { throw IllegalStateException("fixture worker crash") } + worker.name = "fixture-worker" + worker.start() + worker.join() + // The capture is enqueued asynchronously; give the queue executor a moment before + // the inline flush so the event is visible to it. + Thread.sleep(500) + postHog.flush() + println("WORKER_SURVIVED") + } + + else -> error("unknown scenario: $scenario") + } + } +} diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt new file mode 100644 index 000000000..96bf9d389 --- /dev/null +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt @@ -0,0 +1,110 @@ +package com.posthog.server + +import okhttp3.mockwebserver.Dispatcher +import okhttp3.mockwebserver.MockResponse +import okhttp3.mockwebserver.MockWebServer +import okhttp3.mockwebserver.RecordedRequest +import java.io.File +import java.util.concurrent.CopyOnWriteArrayList +import java.util.concurrent.TimeUnit +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +/** + * Forked-JVM tests via [CrashFixture]: the in-process suite invokes the handler directly, which + * cannot prove real process exit, worker-thread survival, or the stderr the JVM actually emits. + */ +internal class PostHogUncaughtExceptionSubprocessTest { + private class FixtureRun( + val exitCode: Int, + val stdout: String, + val stderr: String, + val batches: List, + ) + + private fun runFixture(scenario: String): FixtureRun { + val batches = CopyOnWriteArrayList() + val server = MockWebServer() + server.dispatcher = + object : Dispatcher() { + override fun dispatch(request: RecordedRequest): MockResponse { + if (request.path?.contains("/batch") == true) { + batches.add(request.parseBatch()) + } + return MockResponse().setResponseCode(200) + } + } + server.start() + + try { + val javaBin = File(File(System.getProperty("java.home"), "bin"), "java").absolutePath + val process = + ProcessBuilder( + javaBin, + "-cp", + System.getProperty("java.class.path"), + CrashFixture::class.java.name, + server.url("/").toString(), + scenario, + ).start() + + // Both streams stay tiny (a banner line, one stack trace), far below the pipe buffer, + // so sequential reads cannot deadlock. + val stdout = process.inputStream.bufferedReader().readText() + val stderr = process.errorStream.bufferedReader().readText() + assertTrue(process.waitFor(60, TimeUnit.SECONDS), "Fixture JVM did not exit in time") + + return FixtureRun(process.exitValue(), stdout, stderr, batches.toList()) + } finally { + server.shutdown() + } + } + + private fun exceptionProperties(run: FixtureRun): Map { + val batch = run.batches.firstOrNull { it.findEvent("\$exception") != null } + assertNotNull(batch, "Expected an \$exception event to reach the server, got: ${run.batches}") + return batch.eventProperties("\$exception") + } + + @Test + fun `a main-thread crash terminates the process and delivers a fatal event first`() { + val run = runFixture("main-crash") + + // The JVM's exit code for an uncaught exception on main. + assertEquals(1, run.exitCode, "stderr was: ${run.stderr}") + + // Installing capture must not eat the default crash output. + assertTrue( + run.stderr.contains("Exception in thread \"main\""), + "Expected the default crash banner on stderr, got: ${run.stderr}", + ) + assertTrue( + run.stderr.contains("fixture main crash"), + "Expected the throwable on stderr, got: ${run.stderr}", + ) + + val props = exceptionProperties(run) + assertEquals("fatal", props["\$exception_level"]) + assertEquals("jvm.uncaught_exception_handler", props["\$exception_source"]) + } + + @Test + fun `a worker-thread crash leaves the process running and delivers an error event`() { + val run = runFixture("worker-crash") + + assertEquals(0, run.exitCode, "stderr was: ${run.stderr}") + assertTrue( + run.stdout.contains("WORKER_SURVIVED"), + "Expected the process to keep running after the worker died, got: ${run.stdout}", + ) + assertTrue( + run.stderr.contains("fixture worker crash"), + "Expected the worker's throwable on stderr, got: ${run.stderr}", + ) + + val props = exceptionProperties(run) + assertEquals("error", props["\$exception_level"]) + } +} From c074bd97836c87bc88427d9b232e3f31b2364d02 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:32:31 +0300 Subject: [PATCH 13/22] fix(server): wait out a concurrent flush in the fatal drain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A periodic-timer flush racing the crash made the fatal drain a silent no-op: the drain saw the isFlushing flag held, made no progress, and returned with the crash event still queued. Poll the flag within the bounded budget instead of bailing; a failed send (batch requeued, no progress) still exits — one straight-line attempt per batch, never a retry loop. --- .../server/internal/PostHogMemoryQueue.kt | 24 ++++++-- .../server/internal/PostHogMemoryQueueTest.kt | 61 +++++++++++++++++++ 2 files changed, 81 insertions(+), 4 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 77056b200..6dce0b943 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -132,9 +132,10 @@ internal class PostHogMemoryQueue( /** * Sends batch after batch until the queue is empty, the [deadlineNanos] budget is spent, or a - * pass makes no progress (a failed send requeues its batch, and a flush already in progress on - * another thread makes the pass a no-op) — a crashing process gets one straight-line attempt - * per batch, never a retry loop. + * send fails (a failed send requeues its batch; a crashing process gets one straight-line + * attempt per batch, never a retry loop). A flush already in progress on another thread (the + * periodic timer) is waited out within the budget instead of bailing, so a timer firing at + * crash time cannot make the fatal drain a silent no-op. */ private fun drainUntilDeadline(deadlineNanos: Long) { while (System.nanoTime() < deadlineNanos) { @@ -142,7 +143,19 @@ internal class PostHogMemoryQueue( if (before == 0) { return } - flushIgnoringThreshold() + if (isFlushing.getAndSet(true)) { + config.logger.log("Queue is flushing.") + try { + Thread.sleep(FATAL_DRAIN_POLL_MS) + } catch (e: InterruptedException) { + // the executor is being shut down; an escaping exception on this thread would + // land in the default uncaught handler — us + Thread.currentThread().interrupt() + return + } + continue + } + executeBatch() val after = synchronized(eventsLock) { events.size } if (after >= before) { return @@ -371,5 +384,8 @@ internal class PostHogMemoryQueue( // How long a crashing thread waits for the fatal enqueue + drain, mirroring the Rust SDK's // bounded panic-hook flush. Bounded so telemetry can never hang a dying process. internal const val FATAL_FLUSH_TIMEOUT_MS = 2_000L + + // How often the fatal drain re-checks a flush held by another thread. + private const val FATAL_DRAIN_POLL_MS = 10L } } diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index 7e4eb4762..ca9595cb4 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -13,12 +13,18 @@ import com.posthog.server.createMockHttp import com.posthog.server.generateEvent import com.posthog.server.shutdownAndAwaitTermination import com.posthog.server.unGzip +import okhttp3.mockwebserver.Dispatcher import okhttp3.mockwebserver.MockResponse +import okhttp3.mockwebserver.MockWebServer +import okhttp3.mockwebserver.RecordedRequest import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse import org.junit.Assert.assertTrue +import java.util.concurrent.CopyOnWriteArrayList import java.util.concurrent.CountDownLatch import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger import kotlin.test.Test internal class PostHogMemoryQueueTest { @@ -335,6 +341,61 @@ internal class PostHogMemoryQueueTest { executor.shutdownAndAwaitTermination() } + @Test + fun `the fatal path waits out a flush already in progress instead of giving up`() { + // A periodic-timer flush racing the crash used to make the fatal drain a silent no-op: the + // drain saw no progress and returned with the crash event still queued. + val gate = CountDownLatch(1) + val firstRequestReceived = CountDownLatch(1) + val requestBodies = CopyOnWriteArrayList() + val http = MockWebServer() + http.dispatcher = + object : Dispatcher() { + private val requests = AtomicInteger(0) + + override fun dispatch(request: RecordedRequest): MockResponse { + requestBodies.add(request.body.unGzip()) + if (requests.getAndIncrement() == 0) { + firstRequestReceived.countDown() + gate.await(5, TimeUnit.SECONDS) + } + return MockResponse().setBody("{}") + } + } + http.start() + val sut = getSut(http.url("/").toString(), flushAt = 100) + + sut.add(generateEvent("pre_event")) + executor.awaitExecution() + + // A stand-in for the periodic timer: flush() runs inline on this thread and parks in the + // gated request while holding the isFlushing flag. + val flusher = Thread { sut.flush() } + flusher.start() + assertTrue( + "Expected the concurrent flush to reach the server and hold the flag", + firstRequestReceived.await(5, TimeUnit.SECONDS), + ) + + val sender = Thread { sut.add(generateFatalEvent()) } + sender.start() + // Let the drain observe the held flag before the gate opens. + Thread.sleep(100) + gate.countDown() + + sender.join(5_000) + flusher.join(5_000) + assertFalse("Expected the fatal add to return", sender.isAlive) + + assertTrue( + "The crash event must be sent once the concurrent flush finished, got: $requestBodies", + requestBodies.any { it.contains("\$exception") }, + ) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + @Test fun `the fatal path still flushes when the calling thread is already interrupted`() { val http = createMockHttp(MockResponse().setBody("{}")) From 529b089f269305e1ab330df828faa18d35f96448 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:49:59 +0300 Subject: [PATCH 14/22] fix(server): send the fatal event in the first drained batch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A backlog batch failing with a retriable error is requeued and stops the drain (one straight-line attempt per batch), which consumed the whole crash budget before the appended fatal event ever reached the wire. Front-insert the fatal event instead, so the first — possibly only — wire attempt carries it; batch order does not matter to ingestion. Found by codex review. --- .../server/internal/PostHogMemoryQueue.kt | 31 ++++++++++++++++--- .../server/internal/PostHogMemoryQueueTest.kt | 28 +++++++++++++++-- 2 files changed, 52 insertions(+), 7 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 6dce0b943..c42148b21 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -84,6 +84,28 @@ internal class PostHogMemoryQueue( config.logger.log("Event: ${record.event} was added to the queue.") } + // Front-inserts the fatal event so the FIRST drained batch carries it: a backlog batch that + // fails with a retriable error is requeued and stops the drain (no retry loops on a crashing + // process), which would otherwise consume the whole budget before the crash event ever reached + // the wire. Batch order does not matter to ingestion — events carry their own timestamps. + private fun enqueueFatalFirst(record: PostHogEvent) { + var removedEvent: PostHogEvent? = null + + synchronized(eventsLock) { + if (events.size >= config.maxQueueSize) { + removedEvent = events.removeFirstOrNull() + } + + events.addFirst(record) + } + + if (removedEvent != null) { + config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + } + + config.logger.log("Event: ${record.event} was added to the front of the queue.") + } + override fun flush() { // only flushes if the queue has events if (!isAboveThreshold(1)) { @@ -100,9 +122,10 @@ internal class PostHogMemoryQueue( * The regular [add] path submits the enqueue asynchronously, so a crashing thread that flushed * inline could read the deque before its own event landed, send nothing, and let the event die * with the JVM. Running enqueue + drain as one task on the single-threaded executor makes the - * send happen-after the enqueue, and draining batch by batch (ignoring `flushAt`) until the - * queue is empty keeps a backlog of `maxBatchSize` or more from stranding the fatal event, - * which FIFO puts last. + * send happen-after the enqueue. The fatal event is front-inserted so the first drained batch + * carries it, and the drain then continues batch by batch (ignoring `flushAt`) until the queue + * is empty — a backlog of `maxBatchSize` or more can neither strand the fatal event nor eat the + * budget with a failing batch before the crash ever reaches the wire. * * Delivery stays best-effort: the caller stops waiting at the timeout (the drain keeps going on * the executor thread for whatever process lifetime remains), and each HTTP attempt can fail or @@ -115,7 +138,7 @@ internal class PostHogMemoryQueue( try { executor.execute { try { - enqueue(record) + enqueueFatalFirst(record) drainUntilDeadline(deadlineNanos) } finally { done.countDown() diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index ca9595cb4..0fb68afe1 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -333,9 +333,31 @@ internal class PostHogMemoryQueueTest { sut.add(generateFatalEvent()) assertEquals(3, http.requestCount) - val lastBody = - (1..3).joinToString("\n") { http.takeRequest().body.unGzip() } - assertTrue("The crash event must be part of the drained batches", lastBody.contains("\$exception")) + // Front-inserted, so the crash event rides the very first batch instead of trailing the + // backlog. + val firstBody = http.takeRequest().body.unGzip() + assertTrue("The crash event must be in the first drained batch", firstBody.contains("\$exception")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `the fatal event gets its wire attempt even when the first batch fails`() { + // A retriable failure requeues the batch and stops the drain (no retry loops on a crashing + // process). With the fatal event appended last it would never have reached the wire; front + // insertion puts it in that first, only attempt. + val http = createMockHttp(MockResponse().setResponseCode(500)) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxBatchSize = 2) + + repeat(5) { sut.add(generateEvent("backlog_event_$it")) } + executor.awaitExecution() + + sut.add(generateFatalEvent()) + + assertEquals(1, http.requestCount) + val body = http.takeRequest().body.unGzip() + assertTrue("The failed attempt must have carried the crash event", body.contains("\$exception")) http.shutdown() executor.shutdownAndAwaitTermination() From b7dc5a502350585165d7f915d9a8d089ccd0ec0c Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:49:59 +0300 Subject: [PATCH 15/22] fix(server): keep the crash path bounded after the fatal drain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The capture target's post-capture sweep called the public flush(), which runs an HTTP batch inline on the crashing thread with no timeout — e.g. retrying a batch a 5xx had just requeued — stalling crash delegation past the advertised bound. The fatal path already drains the queue within its budget, so the sweep is now a documented no-op; worker-thread (non-fatal) captures leave the process alive for the periodic flush. Also hardened the subprocess harness (stream drains on their own threads + destroyForcibly on timeout) and documented the fatal-policy approximation limits. Found by codex review. --- .../main/java/com/posthog/server/PostHog.kt | 16 +++++++--- .../PostHogUncaughtExceptionSubprocessTest.kt | 32 +++++++++++++++---- 2 files changed, 38 insertions(+), 10 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index c01fbf1b5..e677917d2 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -65,8 +65,7 @@ public class PostHog : PostHogStateless(), PostHogInterface { // routing it through captureException preserves those via the shared coercer. A // fatal-level event takes PostHogMemoryQueue's bounded blocking fatal path inside // add() (same fatal-record marker the core queue keys on), so capture() itself - // delivers the crash before returning; the flush below is just a best-effort sweep - // of whatever else is still queued. + // delivers the crash — and everything queued ahead of it — before returning. integration.installWith( object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { override fun capture(throwable: Throwable) { @@ -81,7 +80,12 @@ public class PostHog : PostHogStateless(), PostHogInterface { } override fun flush() { - this@PostHog.flush() + // Deliberately empty. The fatal path above already drains the queue + // within its bounded budget; the public flush() would run another HTTP + // batch inline on the crashing thread with no timeout (e.g. retrying a + // batch a 5xx just requeued), stalling crash delegation past the + // advertised bound. A worker-thread (non-fatal) capture leaves the + // process alive, so the periodic flush delivers it. } }, ) @@ -95,7 +99,11 @@ public class PostHog : PostHogStateless(), PostHogInterface { // spec's "expected to terminate" boundary with the main thread: an uncaught exception there is // fatal, while a worker thread's kills only that thread (level error) and the process lives on. // Id 1 is the initial thread on mainstream JVMs and "main" its conventional name; either match - // counts, since a missed main thread would silently downgrade a real crash. + // counts, since a missed main thread would silently downgrade a real crash. Known approximation + // limits, accepted rather than censusing live threads inside a crash handler: a worker that + // happens to be the last non-daemon thread does end the process (its crash is still level + // error, async delivery), and a main-thread exception need not end it while other non-daemon + // threads keep running. @Suppress("DEPRECATION") // Thread.getId is deprecated on JDK 19+ but stable while a thread lives private fun isProcessFatal(thread: Thread): Boolean = thread.id == 1L || thread.name == "main" diff --git a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt index 96bf9d389..e02e786c4 100644 --- a/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/PostHogUncaughtExceptionSubprocessTest.kt @@ -5,8 +5,10 @@ import okhttp3.mockwebserver.MockResponse import okhttp3.mockwebserver.MockWebServer import okhttp3.mockwebserver.RecordedRequest import java.io.File +import java.io.InputStream import java.util.concurrent.CopyOnWriteArrayList import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicReference import kotlin.test.Test import kotlin.test.assertEquals import kotlin.test.assertNotNull @@ -24,6 +26,16 @@ internal class PostHogUncaughtExceptionSubprocessTest { val batches: List, ) + private fun drainAsync(stream: InputStream): Pair, Thread> { + val content = AtomicReference("") + val reader = + Thread { content.set(stream.bufferedReader().readText()) }.apply { + isDaemon = true + start() + } + return content to reader + } + private fun runFixture(scenario: String): FixtureRun { val batches = CopyOnWriteArrayList() val server = MockWebServer() @@ -50,13 +62,21 @@ internal class PostHogUncaughtExceptionSubprocessTest { scenario, ).start() - // Both streams stay tiny (a banner line, one stack trace), far below the pipe buffer, - // so sequential reads cannot deadlock. - val stdout = process.inputStream.bufferedReader().readText() - val stderr = process.errorStream.bufferedReader().readText() - assertTrue(process.waitFor(60, TimeUnit.SECONDS), "Fixture JVM did not exit in time") + // Drain both streams on their own threads so a fixture that fails to exit — the exact + // regression this suite exists to catch — hits the timed wait below instead of hanging + // this thread in readText(). + val stdout = drainAsync(process.inputStream) + val stderr = drainAsync(process.errorStream) + val exited = process.waitFor(60, TimeUnit.SECONDS) + if (!exited) { + process.destroyForcibly() + process.waitFor(10, TimeUnit.SECONDS) + } + stdout.second.join(5_000) + stderr.second.join(5_000) + assertTrue(exited, "Fixture JVM did not exit in time; stderr was: ${stderr.first.get()}") - return FixtureRun(process.exitValue(), stdout, stderr, batches.toList()) + return FixtureRun(process.exitValue(), stdout.first.get(), stderr.first.get(), batches.toList()) } finally { server.shutdown() } From 942bd71186245c0c864c61bd16bede9832bcdd88 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 04:58:52 +0300 Subject: [PATCH 16/22] fix(server): never evict a fatal event by capacity trimming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Front-inserting the fatal event broke the head-is-oldest eviction invariant: in a process that survived a failed crash-send, ordinary captures reaching maxQueueSize evicted the deque head — the crash event awaiting retry. Capacity trimming now evicts the oldest non-fatal event (O(1) in the common case). Found by codex review. --- .../server/internal/PostHogMemoryQueue.kt | 46 +++++++++++-------- .../server/internal/PostHogMemoryQueueTest.kt | 26 +++++++++++ 2 files changed, 54 insertions(+), 18 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index c42148b21..da557a965 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -67,18 +67,15 @@ internal class PostHogMemoryQueue( } private fun enqueue(record: PostHogEvent) { - var removedEvent: PostHogEvent? = null - - synchronized(eventsLock) { - if (events.size >= config.maxQueueSize) { - removedEvent = events.removeFirstOrNull() + val removedEvent = + synchronized(eventsLock) { + val removed = evictIfFullLocked() + events.addLast(record) + removed } - events.addLast(record) - } - if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the queue.") @@ -89,23 +86,36 @@ internal class PostHogMemoryQueue( // process), which would otherwise consume the whole budget before the crash event ever reached // the wire. Batch order does not matter to ingestion — events carry their own timestamps. private fun enqueueFatalFirst(record: PostHogEvent) { - var removedEvent: PostHogEvent? = null - - synchronized(eventsLock) { - if (events.size >= config.maxQueueSize) { - removedEvent = events.removeFirstOrNull() + val removedEvent = + synchronized(eventsLock) { + val removed = evictIfFullLocked() + events.addFirst(record) + removed } - events.addFirst(record) - } - if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the front of the queue.") } + // Must be called under eventsLock. Evicts the oldest NON-fatal event: front-inserting a fatal + // record breaks the head-is-oldest invariant, and a fatal event parked at the head after a + // failed attempt (in a process that survived) must not be displaced by ordinary traffic. In the + // common case the head is non-fatal, so this stays O(1). If every queued event is fatal, nothing + // is evicted — fatal adds drain synchronously, so they cannot accumulate at capacity. + private fun evictIfFullLocked(): PostHogEvent? { + if (events.size < config.maxQueueSize) { + return null + } + val victimIndex = events.indexOfFirst { !it.isFatalExceptionEvent() } + if (victimIndex < 0) { + return null + } + return events.removeAt(victimIndex) + } + override fun flush() { // only flushes if the queue has events if (!isAboveThreshold(1)) { diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index 0fb68afe1..1c3f6a566 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -418,6 +418,32 @@ internal class PostHogMemoryQueueTest { executor.shutdownAndAwaitTermination() } + @Test + fun `a fatal event parked at the head after a failed attempt is not evicted by capacity trimming`() { + // First attempt fails, requeueing the fatal event at the deque head; in a process that + // survived, later ordinary captures reaching maxQueueSize used to evict the head as the + // "oldest" event — discarding the crash before any retry could send it. + val http = createMockHttp(MockResponse().setResponseCode(500), MockResponse().setBody("{}")) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxQueueSize = 2) + + sut.add(generateFatalEvent()) + assertEquals(1, http.requestCount) + + // Two ordinary events on a full queue: eviction must take the non-fatal one. + sut.add(generateEvent("ordinary_1")) + sut.add(generateEvent("ordinary_2")) + executor.awaitExecution() + + sut.flush() + assertEquals(2, http.requestCount) + http.takeRequest() // the failed first attempt + val retryBody = http.takeRequest().body.unGzip() + assertTrue("The retried batch must still contain the crash event", retryBody.contains("\$exception")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + @Test fun `the fatal path still flushes when the calling thread is already interrupted`() { val http = createMockHttp(MockResponse().setBody("{}")) From 195b37c4a5279cfe19833af38cf1e47355470aeb Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 05:07:08 +0300 Subject: [PATCH 17/22] fix(server): keep maxQueueSize a hard bound under fatal-only contents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fatal-eviction preference returned no victim when every queued event was fatal, letting repeated failed fatal submissions in a surviving process grow the deque past the cap. Evict the oldest event in that case — the cap wins over fatal priority. Found by codex review. --- .../server/internal/PostHogMemoryQueue.kt | 12 +++---- .../server/internal/PostHogMemoryQueueTest.kt | 33 +++++++++++++++++++ 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index da557a965..57bd46136 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -100,18 +100,18 @@ internal class PostHogMemoryQueue( config.logger.log("Event: ${record.event} was added to the front of the queue.") } - // Must be called under eventsLock. Evicts the oldest NON-fatal event: front-inserting a fatal - // record breaks the head-is-oldest invariant, and a fatal event parked at the head after a - // failed attempt (in a process that survived) must not be displaced by ordinary traffic. In the - // common case the head is non-fatal, so this stays O(1). If every queued event is fatal, nothing - // is evicted — fatal adds drain synchronously, so they cannot accumulate at capacity. + // Must be called under eventsLock. Prefers the oldest NON-fatal victim: front-inserting a + // fatal record breaks the head-is-oldest invariant, and a fatal event parked at the head after + // a failed attempt (in a process that survived) must not be displaced by ordinary traffic. In + // the common case the head is non-fatal, so this stays O(1). If every queued event is fatal, + // the oldest is evicted anyway — maxQueueSize is a hard bound. private fun evictIfFullLocked(): PostHogEvent? { if (events.size < config.maxQueueSize) { return null } val victimIndex = events.indexOfFirst { !it.isFatalExceptionEvent() } if (victimIndex < 0) { - return null + return events.removeFirstOrNull() } return events.removeAt(victimIndex) } diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index 1c3f6a566..a5e7661d2 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -444,6 +444,39 @@ internal class PostHogMemoryQueueTest { executor.shutdownAndAwaitTermination() } + @Test + fun `maxQueueSize stays a hard bound even when every queued event is fatal`() { + // Persistent send failures in a surviving process can park fatal events; the fatal-eviction + // preference must not turn the cap into unbounded growth — with only fatal events queued, + // the oldest one is evicted anyway. + val http = + createMockHttp( + MockResponse().setResponseCode(500), + MockResponse().setResponseCode(500), + MockResponse().setBody("{}"), + ) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxQueueSize = 1) + + val fatalA = generateFatalEvent().also { it.properties?.put("marker", "fatal_a") } + val fatalB = generateFatalEvent().also { it.properties?.put("marker", "fatal_b") } + + sut.add(fatalA) + sut.add(fatalB) + assertEquals(2, http.requestCount) + http.takeRequest() + http.takeRequest() + + // Only the newer fatal event survived the cap; the retry sends it alone. + sut.flush() + assertEquals(3, http.requestCount) + val retryBody = http.takeRequest().body.unGzip() + assertTrue("The newest fatal event must survive", retryBody.contains("fatal_b")) + assertFalse("The evicted fatal event must be gone", retryBody.contains("fatal_a")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + @Test fun `the fatal path still flushes when the calling thread is already interrupted`() { val http = createMockHttp(MockResponse().setBody("{}")) From c2a2ac2f39901a8c549fe594be4c3b1d4036baac Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 05:12:25 +0300 Subject: [PATCH 18/22] fix(server): refine fatal-priority eviction at the capacity bound MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corner cases in the all-fatal-queue fallback: ordinary traffic could displace a parked crash report (the incoming ordinary event is now dropped instead), and fatal-on-fatal trimming removed the head — the NEWEST record, since fatal events are front-inserted — instead of the oldest at the tail. maxQueueSize stays a hard bound throughout. Found by codex review. --- .../server/internal/PostHogMemoryQueue.kt | 68 ++++++++++++------- .../server/internal/PostHogMemoryQueueTest.kt | 54 +++++++++++++++ 2 files changed, 96 insertions(+), 26 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 57bd46136..ada289780 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -67,15 +67,31 @@ internal class PostHogMemoryQueue( } private fun enqueue(record: PostHogEvent) { - val removedEvent = - synchronized(eventsLock) { - val removed = evictIfFullLocked() + var removedEvent: PostHogEvent? = null + var dropped = false + + synchronized(eventsLock) { + if (events.size >= config.maxQueueSize) { + val victimIndex = nonFatalVictimIndexLocked() + if (victimIndex < 0) { + // Every queued event is a fatal record awaiting retry; ordinary traffic must + // not displace a crash report, so the incoming event is dropped instead. + dropped = true + } else { + removedEvent = events.removeAt(victimIndex) + } + } + if (!dropped) { events.addLast(record) - removed } + } + if (dropped) { + config.logger.log("Queue is full of fatal events awaiting retry, ${record.event} was dropped.") + return + } if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the queue.") @@ -86,35 +102,35 @@ internal class PostHogMemoryQueue( // process), which would otherwise consume the whole budget before the crash event ever reached // the wire. Batch order does not matter to ingestion — events carry their own timestamps. private fun enqueueFatalFirst(record: PostHogEvent) { - val removedEvent = - synchronized(eventsLock) { - val removed = evictIfFullLocked() - events.addFirst(record) - removed + var removedEvent: PostHogEvent? = null + + synchronized(eventsLock) { + if (events.size >= config.maxQueueSize) { + val victimIndex = nonFatalVictimIndexLocked() + removedEvent = + if (victimIndex >= 0) { + events.removeAt(victimIndex) + } else { + // All-fatal contents were all front-inserted, so the tail is the oldest; + // maxQueueSize stays a hard bound and the newest crash reports win. + events.removeLastOrNull() + } } + events.addFirst(record) + } if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the front of the queue.") } - // Must be called under eventsLock. Prefers the oldest NON-fatal victim: front-inserting a - // fatal record breaks the head-is-oldest invariant, and a fatal event parked at the head after - // a failed attempt (in a process that survived) must not be displaced by ordinary traffic. In - // the common case the head is non-fatal, so this stays O(1). If every queued event is fatal, - // the oldest is evicted anyway — maxQueueSize is a hard bound. - private fun evictIfFullLocked(): PostHogEvent? { - if (events.size < config.maxQueueSize) { - return null - } - val victimIndex = events.indexOfFirst { !it.isFatalExceptionEvent() } - if (victimIndex < 0) { - return events.removeFirstOrNull() - } - return events.removeAt(victimIndex) - } + // Must be called under eventsLock. The preferred eviction victim is the oldest NON-fatal event: + // front-inserting fatal records breaks the head-is-oldest invariant, and a fatal event parked + // after a failed attempt (in a process that survived) must not be displaced by ordinary + // traffic. In the common case the head is non-fatal, so callers stay O(1). + private fun nonFatalVictimIndexLocked(): Int = events.indexOfFirst { !it.isFatalExceptionEvent() } override fun flush() { // only flushes if the queue has events diff --git a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt index a5e7661d2..f8bb2b408 100644 --- a/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt +++ b/posthog-server/src/test/java/com/posthog/server/internal/PostHogMemoryQueueTest.kt @@ -477,6 +477,60 @@ internal class PostHogMemoryQueueTest { executor.shutdownAndAwaitTermination() } + @Test + fun `ordinary traffic is dropped rather than displacing a parked fatal event`() { + val http = createMockHttp(MockResponse().setResponseCode(500), MockResponse().setBody("{}")) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxQueueSize = 1) + + val fatal = generateFatalEvent().also { it.properties?.put("marker", "fatal_a") } + sut.add(fatal) + assertEquals(1, http.requestCount) + http.takeRequest() + + // The queue is full of exactly one parked crash report; the ordinary event loses. + sut.add(generateEvent("ordinary_1")) + executor.awaitExecution() + + sut.flush() + assertEquals(2, http.requestCount) + val retryBody = http.takeRequest().body.unGzip() + assertTrue("The parked fatal event must survive ordinary traffic", retryBody.contains("fatal_a")) + assertFalse("The ordinary event must have been dropped", retryBody.contains("ordinary_1")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + + @Test + fun `fatal-on-fatal eviction removes the oldest crash report, not the newest`() { + // Fatal records are front-inserted (head is NEWEST), so all-fatal trimming must take the + // tail; taking the head would silently discard the most recent crash. + val http = + createMockHttp( + MockResponse().setResponseCode(500), + MockResponse().setResponseCode(500), + MockResponse().setResponseCode(500), + MockResponse().setBody("{}"), + ) + val sut = getSut(http.url("/").toString(), flushAt = 100, maxQueueSize = 2) + + listOf("fatal_a", "fatal_b", "fatal_c").forEach { marker -> + sut.add(generateFatalEvent().also { it.properties?.put("marker", marker) }) + } + assertEquals(3, http.requestCount) + repeat(3) { http.takeRequest() } + + sut.flush() + assertEquals(4, http.requestCount) + val retryBody = http.takeRequest().body.unGzip() + assertTrue("The newest crash reports must survive", retryBody.contains("fatal_b")) + assertTrue("The newest crash reports must survive", retryBody.contains("fatal_c")) + assertFalse("The oldest crash report is the one trimmed", retryBody.contains("fatal_a")) + + http.shutdown() + executor.shutdownAndAwaitTermination() + } + @Test fun `the fatal path still flushes when the calling thread is already interrupted`() { val http = createMockHttp(MockResponse().setBody("{}")) From dcbee3e5c01078be1da4e204571d20c1e39ab47d Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 05:18:11 +0300 Subject: [PATCH 19/22] docs(server): note the fatal-trim victim choice is best-effort A concurrent failed flush requeues its batch at the head, which can reorder parked fatal events; which of several parked crash reports gets trimmed at capacity is deliberately not age-tracked. Comment-only. --- .../java/com/posthog/server/internal/PostHogMemoryQueue.kt | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index ada289780..b5e455104 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -111,8 +111,11 @@ internal class PostHogMemoryQueue( if (victimIndex >= 0) { events.removeAt(victimIndex) } else { - // All-fatal contents were all front-inserted, so the tail is the oldest; - // maxQueueSize stays a hard bound and the newest crash reports win. + // All-fatal contents were front-inserted, so the tail is normally the + // oldest and the newest crash reports win; maxQueueSize stays a hard bound. + // (Approximate: a concurrent failed flush requeues its batch at the head, + // which can reorder parked fatal events — which of several parked crash + // reports gets trimmed is deliberately best-effort, not age-tracked.) events.removeLastOrNull() } } From 8507cfb26b0710cfcde29dcdc322db07c456e0f4 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Mon, 24 Aug 2026 19:21:13 +0300 Subject: [PATCH 20/22] style(server): drop redundant safe-calls flagged by CodeQL removedEvent is null-guarded right above; K2 smart-casts the local captured by the inline synchronized block, so the safe-call was a useless null check. --- .../java/com/posthog/server/internal/PostHogMemoryQueue.kt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt index 4033bdc9b..b090b0af4 100644 --- a/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogMemoryQueue.kt @@ -92,7 +92,7 @@ internal class PostHogMemoryQueue( return } if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the queue.") @@ -124,7 +124,7 @@ internal class PostHogMemoryQueue( } if (removedEvent != null) { - config.logger.log("Queue is full, the oldest event ${removedEvent?.event} was discarded.") + config.logger.log("Queue is full, the oldest event ${removedEvent.event} was discarded.") } config.logger.log("Event: ${record.event} was added to the front of the queue.") From 0d4f1644fda37d653ee12e7ee34084ce099eeb50 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Tue, 25 Aug 2026 20:30:55 +0300 Subject: [PATCH 21/22] fix(core): roll back a denied uncaught-handler installation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thread.setDefaultUncaughtExceptionHandler can throw (SecurityException under a SecurityManager), and the process-wide install flag was set before that call — a denied install leaked the flag, permanently blocking any later installation, and could leave a half-initialized client. The ownership state now rolls back before the failure propagates to the caller's per-integration handling. --- ...tHogErrorTrackingAutoCaptureIntegration.kt | 13 +++++++- ...ErrorTrackingAutoCaptureIntegrationTest.kt | 32 +++++++++++++++++++ 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt index 42386831c..95e9be715 100644 --- a/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt +++ b/posthog/src/main/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegration.kt @@ -194,8 +194,19 @@ public class PostHogErrorTrackingAutoCaptureIntegration : PostHogIntegration, Th if (!integrationInstalled.compareAndSet(false, true)) { return } + try { + adapterExceptionHandler.setDefaultUncaughtExceptionHandler(this) + } catch (e: Throwable) { + // e.g. SecurityException under a SecurityManager. Roll the ownership state back so a + // denied install can neither capture nor permanently block a later installation, then + // rethrow for the caller (the core client catches and logs per integration; the server + // client catches around its install helper). + integrationInstalled.set(false) + ownsInstallation = false + defaultExceptionHandler = null + throw e + } ownsInstallation = true - adapterExceptionHandler.setDefaultUncaughtExceptionHandler(this) captureEnabled = true config.logger.log("Exception autocapture is enabled.") } diff --git a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt index cab4e715b..b2c0f062d 100644 --- a/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt +++ b/posthog/src/test/java/com/posthog/errortracking/PostHogErrorTrackingAutoCaptureIntegrationTest.kt @@ -17,6 +17,7 @@ import kotlin.test.AfterTest import kotlin.test.BeforeTest import kotlin.test.Test import kotlin.test.assertEquals +import kotlin.test.assertFailsWith internal class PostHogErrorTrackingAutoCaptureIntegrationTest { private val mockConfig = mock() @@ -657,6 +658,37 @@ internal class PostHogErrorTrackingAutoCaptureIntegrationTest { integration.uninstall() } + @Test + fun `a denied handler installation rolls back so a later install can still succeed`() { + // Thread.setDefaultUncaughtExceptionHandler can throw (SecurityException under a + // SecurityManager). The process-wide install flag used to be set before that call, so a + // denied install leaked it and no instance could ever install again. + whenever(mockAdapter.setDefaultUncaughtExceptionHandler(anyOrNull())) + .thenThrow(SecurityException("denied")) + .thenAnswer { + currentHandler = it.getArgument(0) + null + } + + val first = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) + assertFailsWith { first.installWith(RecordingTarget()) } + + // The denied instance must not capture. + val target = RecordingTarget() + first.uncaughtException(Thread.currentThread(), RuntimeException("boom")) + assertEquals(0, target.captured.size) + + // A later install (e.g. after the policy changed) must not be blocked by the failed one. + val second = PostHogErrorTrackingAutoCaptureIntegration(mockConfig, mockAdapter, enabledGate = { true }) + second.installWith(target) + assertEquals(second, currentHandler) + + second.uncaughtException(Thread.currentThread(), RuntimeException("boom")) + assertEquals(1, target.captured.size) + + second.uninstall() + } + @Test fun `uncaughtException applies the layer-supplied fatal policy per thread`() { val target = RecordingTarget() From a57eb1e9dbb4b89036e6e4b853f01547012291b1 Mon Sep 17 00:00:00 2001 From: Catalin Irimie Date: Tue, 25 Aug 2026 20:30:56 +0300 Subject: [PATCH 22/22] refactor(server): move the uncaught-capture wiring out of the client class The capture target, fatal policy and $exception_source stamping now live in an internal PostHogUncaughtExceptionCapture helper; PostHog.setup only calls install(). A denied handler installation logs and leaves the client running with capture disabled instead of failing setup. Changesets trimmed to customer-facing one-liners. --- .changeset/core-uncaught-gate.md | 2 +- .changeset/server-uncaught-exceptions.md | 4 +- .../main/java/com/posthog/server/PostHog.kt | 53 +----------- .../PostHogUncaughtExceptionCapture.kt | 81 +++++++++++++++++++ 4 files changed, 86 insertions(+), 54 deletions(-) create mode 100644 posthog-server/src/main/java/com/posthog/server/internal/PostHogUncaughtExceptionCapture.kt diff --git a/.changeset/core-uncaught-gate.md b/.changeset/core-uncaught-gate.md index c9c8a9f8f..cb5f82c03 100644 --- a/.changeset/core-uncaught-gate.md +++ b/.changeset/core-uncaught-gate.md @@ -2,4 +2,4 @@ 'posthog': patch --- -`PostHogErrorTrackingAutoCaptureIntegration` can now be gated on a caller-supplied strategy instead of the built-in gate (local `errorTrackingConfig.autoCapture` with remote config as a kill-switch): a new `PostHogErrorTrackingAutoCaptureIntegration(config, enabledGate)` constructor lets SDK layers that never fetch remote config (e.g. the server SDK) decide autocapture purely from local config. The uncaught handler also delivers captures through an internal `CaptureTarget` seam (`installWith`) so it can drive clients that are not a core `PostHogInterface`, and when no previous default handler exists it now reproduces the JVM's own `Exception in thread ...` stderr output, so installing capture never hides a crash from log collection. Uncaught-exception events now carry the canonical `mechanism.type` `onuncaughtexception` (previously the noncanonical `UncaughtExceptionHandler`), per the sdk-specs exception-event-metadata spec. Android behavior and the existing `install(PostHogInterface)` path are otherwise unchanged; the additions are internal (`@PostHogInternal`) and visible only because of the multi-module architecture. +Uncaught-exception events now use the canonical `onuncaughtexception` mechanism type. diff --git a/.changeset/server-uncaught-exceptions.md b/.changeset/server-uncaught-exceptions.md index 41b837bac..11c04b331 100644 --- a/.changeset/server-uncaught-exceptions.md +++ b/.changeset/server-uncaught-exceptions.md @@ -2,6 +2,4 @@ 'posthog-server': minor --- -Opt in to capturing uncaught JVM exceptions on the server SDK via `PostHogConfig.captureUncaughtExceptions` (also on the config `Builder`). When enabled, `PostHog` installs a global `Thread.defaultUncaughtExceptionHandler` on setup that captures the crashing exception as an unhandled `$exception` event (mechanism `onuncaughtexception`, `$exception_source: jvm.uncaught_exception_handler`; `$exception_level` is `fatal` for a main-thread crash and `error` for a worker-thread one, whose death the process survives), flushes, and then delegates to the previously registered handler; the handler is removed again on `close()`. Unlike the Android SDK this is gated purely on the local flag — the server SDK never fetches remote config. A fatal `$exception` event takes a dedicated queue path (keyed off the same fatal-event marker the Android SDK's queue uses): the enqueue and the send run as one ordered task on the queue executor, draining the queue batch by batch and ignoring `flushAt`, so a backlog larger than one batch cannot strand the crash event, and the crashing thread blocks on it for at most a bounded timeout (2s). Delivery stays best-effort, the same guarantee class as the Android SDK: the drain can hit the timeout and each HTTP attempt can fail or be cut short by an immediate hard exit. - -The queue's periodic flush timer is now a daemon thread (matching the core SDK's queue), so a set-up client no longer keeps a finished — or crashed — JVM alive until `close()` is called. +Add `captureUncaughtExceptions`: opt in to capturing uncaught JVM exceptions as error tracking events, with a best-effort flush before the process exits. The SDK's flush timer is now a daemon thread and no longer keeps a finished JVM alive until `close()`. diff --git a/posthog-server/src/main/java/com/posthog/server/PostHog.kt b/posthog-server/src/main/java/com/posthog/server/PostHog.kt index e677917d2..b16c68cc7 100644 --- a/posthog-server/src/main/java/com/posthog/server/PostHog.kt +++ b/posthog-server/src/main/java/com/posthog/server/PostHog.kt @@ -6,6 +6,7 @@ import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration import com.posthog.internal.FeatureFlag import com.posthog.server.internal.EvaluationsHost import com.posthog.server.internal.PostHogFeatureFlags +import com.posthog.server.internal.PostHogUncaughtExceptionCapture @Suppress("DEPRECATION") public class PostHog : PostHogStateless(), PostHogInterface { @@ -51,62 +52,19 @@ public class PostHog : PostHogStateless(), PostHogInterface { } // Core setup never installs integrations for the stateless base, so wire the uncaught - // handler explicitly. Gate purely on the local server flag — the server SDK never fetches - // remote config, so the remote-config gate the Android SDK uses can never fire here. + // handler explicitly (all mechanics live in PostHogUncaughtExceptionCapture). // Single-owner by design: the handler is process-wide, so only the first client that opts in // installs it. With several live clients all opting in, closing the owner restores the // previous handler and the remaining clients do not take over — capture stops until a client // is set up again. Server apps use one client per process, so we don't ref-count here. if (config.captureUncaughtExceptions) { getConfig()?.let { coreConfig -> - val integration = - PostHogErrorTrackingAutoCaptureIntegration(coreConfig, { true }, ::isProcessFatal) - // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; - // routing it through captureException preserves those via the shared coercer. A - // fatal-level event takes PostHogMemoryQueue's bounded blocking fatal path inside - // add() (same fatal-record marker the core queue keys on), so capture() itself - // delivers the crash — and everything queued ahead of it — before returning. - integration.installWith( - object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { - override fun capture(throwable: Throwable) { - // $exception_source names the concrete runtime hook per the sdk-specs - // convention (.); the mechanism category - // (onuncaughtexception) rides on the PostHogThrowable. - captureException( - throwable, - null, - mapOf(EXCEPTION_SOURCE_ATTRIBUTE to EXCEPTION_SOURCE_UNCAUGHT_HANDLER), - ) - } - - override fun flush() { - // Deliberately empty. The fatal path above already drains the queue - // within its bounded budget; the public flush() would run another HTTP - // batch inline on the crashing thread with no timeout (e.g. retrying a - // batch a 5xx just requeued), stalling crash delegation past the - // advertised bound. A worker-thread (non-fatal) capture leaves the - // process alive, so the periodic flush delivers it. - } - }, - ) - uncaughtExceptionIntegration = integration + uncaughtExceptionIntegration = PostHogUncaughtExceptionCapture.install(this, coreConfig) } } } } - // The JVM cannot tell whether a thread's death will end the process, so this approximates the - // spec's "expected to terminate" boundary with the main thread: an uncaught exception there is - // fatal, while a worker thread's kills only that thread (level error) and the process lives on. - // Id 1 is the initial thread on mainstream JVMs and "main" its conventional name; either match - // counts, since a missed main thread would silently downgrade a real crash. Known approximation - // limits, accepted rather than censusing live threads inside a crash handler: a worker that - // happens to be the last non-daemon thread does end the process (its crash is still level - // error, async delivery), and a main-thread exception need not end it while other non-daemon - // threads keep running. - @Suppress("DEPRECATION") // Thread.getId is deprecated on JDK 19+ but stable while a thread lives - private fun isProcessFatal(thread: Thread): Boolean = thread.id == 1L || thread.name == "main" - override fun close() { // Same lock as setup so the uninstall + field clear cannot race a concurrent setup() that is // still assigning uncaughtExceptionIntegration; super.close re-acquires the reentrant lock. @@ -488,11 +446,6 @@ public class PostHog : PostHogStateless(), PostHogInterface { } public companion object { - // Event-level capture-integration identity for the uncaught handler, following the - // sdk-specs lowercase . convention. - private const val EXCEPTION_SOURCE_ATTRIBUTE = "\$exception_source" - private const val EXCEPTION_SOURCE_UNCAUGHT_HANDLER = "jvm.uncaught_exception_handler" - /** * Sets up the SDK and returns an instance that you can hold and pass around. * diff --git a/posthog-server/src/main/java/com/posthog/server/internal/PostHogUncaughtExceptionCapture.kt b/posthog-server/src/main/java/com/posthog/server/internal/PostHogUncaughtExceptionCapture.kt new file mode 100644 index 000000000..70904ba1c --- /dev/null +++ b/posthog-server/src/main/java/com/posthog/server/internal/PostHogUncaughtExceptionCapture.kt @@ -0,0 +1,81 @@ +package com.posthog.server.internal + +import com.posthog.PostHogConfig +import com.posthog.errortracking.PostHogErrorTrackingAutoCaptureIntegration +import com.posthog.server.PostHogInterface + +/** + * Wires the shared uncaught-exception integration to the server client, so the client class itself + * carries none of the crash-capture mechanics: the gate (purely the local flag — the server SDK + * never fetches remote config, so the remote-config gate the Android SDK uses can never fire), the + * per-thread fatal policy, and the capture target. + */ +internal object PostHogUncaughtExceptionCapture { + // Event-level capture-integration identity for the uncaught handler, following the + // sdk-specs lowercase . convention. + private const val EXCEPTION_SOURCE_ATTRIBUTE = "\$exception_source" + private const val EXCEPTION_SOURCE_UNCAUGHT_HANDLER = "jvm.uncaught_exception_handler" + + /** + * Installs the handler delivering captures to [client]. Returns the integration to uninstall + * on close, or null when installation failed — an environment that refuses a process-wide + * handler (e.g. a SecurityManager) must not fail client setup. + */ + fun install( + client: PostHogInterface, + coreConfig: PostHogConfig, + ): PostHogErrorTrackingAutoCaptureIntegration? { + val integration = + PostHogErrorTrackingAutoCaptureIntegration(coreConfig, { true }, ::isProcessFatal) + // The uncaught Throwable is a PostHogThrowable carrying fatal/handled=false/mechanism; + // routing it through captureException preserves those via the shared coercer. A + // fatal-level event takes PostHogMemoryQueue's bounded blocking fatal path inside + // add() (same fatal-record marker the core queue keys on), so capture() itself + // delivers the crash — and everything queued ahead of it — before returning. + val target = + object : PostHogErrorTrackingAutoCaptureIntegration.CaptureTarget { + override fun capture(throwable: Throwable) { + // $exception_source names the concrete runtime hook per the sdk-specs + // convention (.); the mechanism category + // (onuncaughtexception) rides on the PostHogThrowable. + client.captureException( + throwable, + null, + mapOf(EXCEPTION_SOURCE_ATTRIBUTE to EXCEPTION_SOURCE_UNCAUGHT_HANDLER), + ) + } + + override fun flush() { + // Deliberately empty. The fatal path above already drains the queue + // within its bounded budget; the public flush() would run another HTTP + // batch inline on the crashing thread with no timeout (e.g. retrying a + // batch a 5xx just requeued), stalling crash delegation past the + // advertised bound. A worker-thread (non-fatal) capture leaves the + // process alive, so the periodic flush delivers it. + } + } + + return try { + integration.installWith(target) + integration + } catch (e: Throwable) { + // Thread.setDefaultUncaughtExceptionHandler can throw (SecurityException under a + // SecurityManager); the integration rolls its ownership state back, and setup + // continues with capture disabled instead of leaving a half-initialized client. + coreConfig.logger.log("Could not install the uncaught-exception handler: $e.") + null + } + } + + // The JVM cannot tell whether a thread's death will end the process, so this approximates the + // spec's "expected to terminate" boundary with the main thread: an uncaught exception there is + // fatal, while a worker thread's kills only that thread (level error) and the process lives on. + // Id 1 is the initial thread on mainstream JVMs and "main" its conventional name; either match + // counts, since a missed main thread would silently downgrade a real crash. Known approximation + // limits, accepted rather than censusing live threads inside a crash handler: a worker that + // happens to be the last live non-daemon thread does end the process (its crash is still level + // error, async delivery), and a main-thread exception need not end it while other non-daemon + // threads keep running. + @Suppress("DEPRECATION") // Thread.getId is deprecated on JDK 19+ but stable while a thread lives + private fun isProcessFatal(thread: Thread): Boolean = thread.id == 1L || thread.name == "main" +}