diff --git a/contents/docs/distributed-tracing/installation/index.mdx b/contents/docs/distributed-tracing/installation/index.mdx index 6f13f897d21c..8e88e4ebc096 100644 --- a/contents/docs/distributed-tracing/installation/index.mdx +++ b/contents/docs/distributed-tracing/installation/index.mdx @@ -6,7 +6,7 @@ import TracingInstallationPlatforms from './_snippets/installation-platforms' PostHog Tracing works with any OpenTelemetry-compatible client – point its OTLP trace exporter at PostHog and you're done, with no PostHog-specific packages. -On Node.js, `posthog-node` can also create and export spans on its own, with no OpenTelemetry dependency. See the [Node.js guide](/docs/distributed-tracing/installation/nodejs) for both routes. +On Node.js and Python, the PostHog SDK can also create and export spans on its own, with no OpenTelemetry dependency. See the [Node.js](/docs/distributed-tracing/installation/nodejs) and [Python](/docs/distributed-tracing/installation/python) guides for both routes. ## Platforms diff --git a/contents/docs/distributed-tracing/installation/python.mdx b/contents/docs/distributed-tracing/installation/python.mdx index 40642332958d..0b27e2f1e4f6 100644 --- a/contents/docs/distributed-tracing/installation/python.mdx +++ b/contents/docs/distributed-tracing/installation/python.mdx @@ -7,9 +7,125 @@ showStepsToc: true import { Steps, Step } from 'components/Docs/Steps' import TracingNextSteps from './_snippets/tracing-next-steps.mdx' +There are two ways to send spans from Python. + +| | `posthog` | OpenTelemetry | +| ------------------------------ | ----------------------------------------------------------- | -------------------------------------------------------------------- | +| **Packages** | The SDK you already use for analytics | `opentelemetry-sdk` and an OTLP exporter | +| **Instrumentation** | Manual – you wrap the operations you care about | Manual, plus auto-instrumentation for Django, Flask, FastAPI, databases, and more | +| **Person and session join** | Automatic inside a PostHog context that identifies them | Set the attributes yourself | + +Pick OpenTelemetry if you already run it, if you use `asyncio` with `AsyncPosthog`, or if you want spans from your web framework and database driver without writing them yourself. Pick `posthog` if PostHog is your only tracing backend and you'd rather instrument a handful of operations by hand than add an exporter pipeline. + +Both routes send OTLP spans to the same endpoint, so you can start with one and switch later without losing your traces. + +## With `posthog` + +> **Minimum version:** `posthog` 7.58.0 or later. + + + + + +```bash +pip install "posthog>=7.58.0" +``` + + + + + +Tracing is off until you set the `traces` option. There's no OpenTelemetry dependency to add. + +```python +from posthog import Posthog + +posthog = Posthog( + "", + host="", + traces={ + "service_name": "checkout-api", + "environment": "production", + }, +) +``` + +| Option | Description | +| --------------------- | -------------------------------------------------------------------- | +| `service_name` | Identifies the service in the Tracing UI. Maps to `service.name` | +| `service_version` | Release version. Maps to `service.version` | +| `environment` | Deployment environment, e.g. `production`. Maps to `deployment.environment` | +| `resource_attributes` | Additional OpenTelemetry resource attributes | + +Use your **project token** (the same one you use for capturing events), not a [personal API key](/docs/api#authentication). Tracing works with the synchronous `Posthog` client and the module-level API, not `AsyncPosthog`. + +See the [Python SDK docs](/docs/libraries/python#configuration) for batching, queue, and span-limit options, and [`before_span_send`](/docs/libraries/python#scrubbing-and-dropping-spans) for scrubbing attributes or dropping spans before they're exported. + + + + + +`start_span` used in a `with` block makes the span active for the block and ends it when the block exits. Spans started inside the block nest underneath it automatically. + +```python +with posthog.start_span("POST /checkout", kind="server") as span: + span.set_attribute("plan", user.plan) + + with posthog.start_span("create-order"): + order = create_order(cart) + with posthog.start_span("charge-card"): + stripe.charge(order) +``` + +If an exception escapes the block, the span records it, its status is set to `error`, and the exception propagates unchanged. + +Span names should be low-cardinality operation names – `GET /users/:id`, not `GET /users/123`. Variable values belong in attributes. + +For work that can't wrap a block, call `start_span` without `with` and call `end()` yourself. See the [Python SDK docs](/docs/libraries/python#distributed-tracing) for the full span API and for continuing a trace across services with W3C `traceparent` headers. + + + + + +Spans created inside a PostHog [context](/docs/libraries/python#contexts) that has a distinct ID or session ID carry `posthogDistinctId` and `sessionId` attributes, which is what makes a trace reachable from a person or a Session Replay recording. + +```python +from posthog import new_context, identify_context, set_context_session + +with new_context(): + identify_context(user.id) + set_context_session(session_id) + + with posthog.start_span("POST /checkout"): + process_order() +``` + +If you use Django, the [contexts middleware](/docs/libraries/django#django-contexts-middleware) sets this up for every request, and reads the `X-POSTHOG-DISTINCT-ID` and `X-POSTHOG-SESSION-ID` headers that [`tracing_headers`](/docs/libraries/js/config#tracing-headers) sends from the browser. + + + + + +Queued spans are exported on an interval, even with `sync_mode` on, so a short-lived process can exit before they're sent. Both `flush()` and `shutdown()` export spans that have already ended. + +```python +def handler(event, context): + with posthog.start_span("handler"): + do_work() + posthog.flush() +``` + +In a serverless handler, call `flush()` before returning. Call `shutdown()` when the process is genuinely exiting. + + + + + +## With OpenTelemetry + - + For the complete SDK reference, see the [OpenTelemetry Python docs](https://opentelemetry.io/docs/languages/python/). @@ -21,7 +137,7 @@ pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http - + You'll need your PostHog project token to authenticate trace requests. This is the same token you use for capturing events with the PostHog SDK. @@ -31,7 +147,7 @@ You can find your project token in [Project settings](https://app.posthog.com/se - + Set up the OpenTelemetry SDK to export spans to PostHog over OTLP HTTP. @@ -68,7 +184,7 @@ OTEL_SERVICE_NAME="my-service" - + Wrap the operations you want to measure in spans, and attach attributes for context. @@ -83,11 +199,17 @@ def checkout(order_id, amount): return "confirmed" ``` +To join these spans to a person or a Session Replay recording, set `posthogDistinctId` and `sessionId` attributes yourself, from the `X-POSTHOG-DISTINCT-ID` and `X-POSTHOG-SESSION-ID` headers that [`tracing_headers`](/docs/libraries/js/config#tracing-headers) sends from the browser. + - + + + + + -Once everything is configured, confirm spans are reaching PostHog: +Whichever route you took: 1. Run your application and trigger the instrumented code 2. Open the PostHog Tracing interface diff --git a/contents/docs/distributed-tracing/start-here.mdx b/contents/docs/distributed-tracing/start-here.mdx index 0cfad9f2ad05..4ed36bb36aa5 100644 --- a/contents/docs/distributed-tracing/start-here.mdx +++ b/contents/docs/distributed-tracing/start-here.mdx @@ -18,7 +18,7 @@ import { QuestLog, QuestLogItem } from "components/Docs/QuestLog"; PostHog Distributed Tracing works with any OpenTelemetry client. Use the OTel SDKs you already have, point your trace exporter at PostHog's HTTP endpoint, and add your project token. -On Node.js you can skip OpenTelemetry entirely: `posthog-node` creates and exports spans itself, and spans made inside a request context carry the person and session they belong to. See the [Node.js guide](/docs/distributed-tracing/installation/nodejs). +On Node.js and Python you can skip OpenTelemetry entirely: `posthog-node` and `posthog` create and export spans themselves, and spans made inside a request context carry the person and session they belong to. See the [Node.js](/docs/distributed-tracing/installation/nodejs) and [Python](/docs/distributed-tracing/installation/python) guides. Set these on your OpenTelemetry SDK: diff --git a/contents/docs/libraries/python/index.mdx b/contents/docs/libraries/python/index.mdx index 9c21dd746374..8514465fa910 100644 --- a/contents/docs/libraries/python/index.mdx +++ b/contents/docs/libraries/python/index.mdx @@ -13,6 +13,7 @@ features: surveys: false aiObservability: true errorTracking: true + tracing: true --- import PythonSdkVersionNote from '../../_snippets/python-sdk-version-note.mdx' @@ -274,6 +275,232 @@ posthog = Posthog( You can configure which variables are captured, masked, or ignored. See the [code variables documentation](/docs/error-tracking/code-variables/python) for detailed configuration options. +## Distributed tracing + +> Requires `posthog` version 7.58.0 or later. + + + +Tracing is new in the Python SDK and its API can still change in a minor release. Spans you send are kept – it's the SDK surface that isn't frozen yet. + + + +Tracing records **spans** – timed units of work – so you can see where time went in a request and how work fans out across your services. Spans created inside a [context](#contexts) automatically carry the person and session they belong to, so a slow trace links back to the person who experienced it. + +Tracing is off until you set the `traces` option. No OpenTelemetry dependency is required. For what you can do with spans once they arrive, see [Distributed tracing](/docs/distributed-tracing/start-here). + +```python +from posthog import Posthog + +posthog = Posthog( + "", + host="", + traces={"service_name": "checkout-api"}, +) +``` + +If you use the module-level API instead, set `posthog.traces = {"service_name": "checkout-api"}` alongside your other options, before you start the first span. + +Set `service_name` – PostHog groups operations by service and span name. + +Tracing is available on the synchronous `Posthog` client and the module-level API. `AsyncPosthog` doesn't support it yet. + +### Creating spans + +`start_span` returns a span. Use it in a `with` block to make it the active span for the block and end it when the block exits. Spans started inside the block nest underneath it automatically. + +```python +with posthog.start_span("POST /checkout", kind="server") as span: + span.set_attribute("plan", user.plan) + + with posthog.start_span("create-order"): + order = create_order(cart) + with posthog.start_span("charge-card"): + stripe.charge(order) +``` + +If an exception escapes the block, the span records it, its status is set to `error`, and the exception propagates unchanged. `KeyboardInterrupt`, `GeneratorExit`, and `asyncio.CancelledError` still end the span but aren't recorded as failures. + +The recorded exception includes the stack trace, which contains file paths from your server. If you'd rather those didn't leave your process, remove `exception.stacktrace` in [`before_span_send`](#scrubbing-and-dropping-spans). + +For work that can't wrap a block, call `start_span` without `with`. **A span started this way isn't active**, so spans started afterwards aren't its children unless you pass `parent` explicitly – and you must call `end()` yourself. + +```python +span = posthog.start_span("background-sync", attributes={"queue": "emails"}) +try: + # Explicitly parent a child to a span that isn't active. + child = posthog.start_span("send-batch", parent=span) + child.end() +finally: + span.end() +``` + +`get_active_span()` returns the span active in the current context, or `None` when there isn't one. + +The active span is tracked with [`contextvars`](https://docs.python.org/3/library/contextvars.html), so it carries across `await` in asyncio code. Threads don't reliably inherit it – pass `parent=span` to continue the trace in a thread you start or a `ThreadPoolExecutor` task. A forked child process starts with no active span, so pass `parent` there too. + +`start_span` always returns a usable span, even when tracing is off, so your code never needs to check whether tracing is enabled. + +### Span names and attributes + +Span names should be low-cardinality operation names – `GET /users/:id`, not `GET /users/123`. Variable values belong in attributes. Strings, booleans, integers, and floats keep their type. Lists and dictionaries are sent as arrays and maps, but PostHog stores them as serialized strings. Setting an attribute to `None` removes it, and any other value is converted to a string. + +```python +with posthog.start_span("GET /users/:id", kind="server") as span: + span.set_attributes({"user.id": user_id, "db.rows": len(rows)}) + span.add_event("cache-miss") + + if not rows: + span.set_status("error", "user not found") +``` + +| Method | Description | +|---|---| +| `set_attribute(key, value)` | Set a single attribute | +| `set_attributes(attributes)` | Merge several attributes at once | +| `add_event(name, attributes=None, timestamp=None)` | Record a timestamped event within the span | +| `set_status(code, message=None)` | Set the outcome: `"ok"` or `"error"`. `"ok"` is final – an exception raised later in the `with` block doesn't override it | +| `record_exception(exception)` | Attach an exception event carrying the type, message, and – for a raised exception – stack trace, and set status to `error`. Use it for exceptions you catch and handle | +| `update_name(name)` | Replace the span name, e.g. once a route template resolves | +| `traceparent()` | This span's W3C `traceparent` header value, or `None` | +| `tracestate()` | This span's W3C `tracestate` value, or `None` when it has none | +| `end(end_time=None)` | End the span and queue it for export. A `with` block does this for you | + +Every method except `traceparent()`, `tracestate()`, and `end()` returns the span, so calls chain. Calls after `end()` are ignored. + +`start_span` takes these keyword arguments: + +| Argument | Description | +|---|---| +| `kind` | What the work is: `"internal"` (default), `"server"` for an inbound request, `"client"` for an outbound call, `"producer"` or `"consumer"` for queue work | +| `attributes` | Attributes to set at span start | +| `parent` | A span, or an inbound W3C `traceparent` string to continue a trace another service started. Defaults to the active span | +| `tracestate` | The W3C `tracestate` accompanying a `traceparent` string. Ignored when `parent` is a span, which inherits its parent's | +| `start_time` | Backdate the span's start, as a `datetime` or seconds since the epoch. The server clamps a start more than 24 hours old to receive time; with `debug` on, the SDK prints a debug message when you pass one | + +### Tracing across services + +Spans use [W3C Trace Context](https://www.w3.org/TR/trace-context/), so a trace can span several services. Pass an inbound `traceparent` header as `parent` to continue a trace another service started, and send `span.traceparent()` onward when you call out. + +```python file=app.py +import requests +from flask import request + + +@app.post("/checkout") +def checkout(): + with posthog.start_span( + "POST /checkout", + kind="server", + parent=request.headers.get("traceparent"), + ) as span: + traceparent = span.traceparent() + + requests.post( + "https://payments.internal/charge", + headers={"traceparent": traceparent} if traceparent else {}, + ) + + return {"status": "ok"} +``` + +A malformed `traceparent` starts a new trace rather than raising. A missing one (`None`) falls back to the active span, if there is one. + +A continued trace propagates the sampled flag it was handed, so a downstream sampler sees the decision the head service made. PostHog itself doesn't sample – a span is recorded and exported whichever way that flag is set. + +### Linking traces to people and sessions + +Spans created inside a [context](#contexts) that has a distinct ID or session ID automatically carry `posthogDistinctId` and `sessionId` attributes, which is what makes a trace reachable from a person or a Session Replay recording. In Django, the [contexts middleware](/docs/libraries/django#django-contexts-middleware) sets these for every request. Elsewhere, set them yourself: + +```python +from posthog import new_context, identify_context, set_context_session + +with new_context(): + identify_context(user.id) + set_context_session(session_id) + + with posthog.start_span("POST /checkout"): + process_order() +``` + +Spans created outside a context with those values omit the attributes. + +### Scrubbing and dropping spans + +`before_span_send` runs on every finished span before it's queued for export. It receives the span as a dict with `name`, `kind`, `status`, `attributes`, `events`, `start_time_ns`, `end_time_ns`, `trace_id`, `span_id`, and `parent_span_id`. Edit it and return it, or return `None` to drop the span entirely. + +```python +def scrub_spans(span): + if span["attributes"].get("http.route") == "/health": + return None + + span["attributes"].pop("http.request.header.authorization", None) + return span + + +posthog = Posthog( + "", + host="", + traces={ + "service_name": "checkout-api", + "before_span_send": scrub_spans, + }, +) +``` + +Attributes are plain Python values, not the OTLP wire encoding. The hook runs after PostHog attaches `posthogDistinctId` and `sessionId`, so those are visible to the hook and can be scrubbed too. An exception's stack trace is on its event, under `event["attributes"]["exception.stacktrace"]`. + +- `trace_id`, `span_id`, and `parent_span_id` are read-only. Rewriting them would orphan child spans that have already been exported, so changes are reverted. +- A hook that raises drops the span rather than exporting it without scrubbing. +- Pass a list to run several hooks in order. The first one to return `None` stops the chain. +- The hook must be a regular function. An `async` hook drops every span. +- If an entry isn't callable, tracing turns off for the client rather than exporting spans the hook was meant to scrub. + +### Span limits + +A span is capped at 128 attributes and 128 events, each event at 128 attributes, and each string attribute value at 8192 characters. The endpoint rejects a span that's too large, and a rejected span is lost whole rather than truncated, so the caps bound a span before it gets there. + +Past the cap, the earliest attributes and events are kept and the number dropped is reported alongside the span, so a truncated span reads as truncated rather than as quietly incomplete. The attributes PostHog attaches itself – `posthogDistinctId` and `sessionId` – don't count toward the cap and are never dropped, so a span at the limit still links back to its person and session. + +The event cap is absolute: an `exception` event the SDK records for you spends an ordinary slot like any other. A span that fills its events and then raises keeps its `error` status but not the exception detail, and reports the loss as a dropped event. Raise `max_events_per_span` on spans that record many events and can also fail. + +The length bound reaches inside a value, including strings nested in lists and dictionaries, and applies to `exception.stacktrace` like any other attribute – a long stack trace keeps its last 8192 characters. All four caps are re-applied after `before_span_send`, so a hook that enriches a span can't push it back over. + +### Configuration + +| Option | Default | Description | +|---|---|---| +| `service_name` | – | Name of the service producing spans. Set this | +| `service_version` | – | Version of the service | +| `environment` | – | Deployment environment, e.g. `production` | +| `resource_attributes` | – | Extra OTLP resource attributes. Takes precedence over the fields above | +| `flush_interval` | `5` | Seconds between exports of queued spans | +| `max_export_batch_size` | `512` | Maximum spans per request | +| `max_queue_size` | `2048` | Maximum spans held in memory. Spans beyond this are dropped | +| `max_live_spans` | `10000` | Maximum spans open at once. At the limit `start_span` returns a span that isn't recorded | +| `max_span_age` | `3600` | Once `max_live_spans` is reached, spans open longer than this many seconds are treated as leaked and never exported | +| `before_span_send` | – | Edit or drop each finished span before export. Return `None` to drop it | +| `max_attributes_per_span` | `128` | Maximum attributes you set on one span | +| `max_events_per_span` | `128` | Maximum events on one span | +| `max_attribute_value_length` | `8192` | Maximum characters in a string attribute value | + +An invalid value falls back to its default with a warning. The exception is `before_span_send`: an entry that isn't callable turns tracing off. + +### Shutdown and short-lived processes + +Spans are exported on a background interval, even with `sync_mode` on. Both `flush()` and `shutdown()` export spans that have already ended. A span still open at `flush()` is exported once it ends; a span still open at `shutdown()` is discarded with a warning, so end your spans before shutting down – a `with` block does this for you. `shutdown()` gives queued spans up to 30 seconds to send. + +In a serverless handler, call `flush()` before returning. Events and spans are flushed concurrently, so it costs one round trip, not two. + +```python +def handler(event, context): + with posthog.start_span("handler"): + do_work() + posthog.flush() +``` + +A script that exits without calling `shutdown()` still gets a brief best-effort flush at exit, but don't rely on it for spans you need. + ## GeoIP properties Before posthog-python v3.0, we added GeoIP properties to all incoming events by default. We also used these properties for feature flag evaluation, based on the IP address of the request. This isn't ideal since they are created based on your server IP address, rather than the user's, leading to incorrect location resolution. @@ -413,6 +640,8 @@ By default, the synchronous `Posthog` client buffers events before sending them - Call `posthog.shutdown()` before the process ends. This blocking call attempts to deliver queued events and cleans up the client. - Enable `sync_mode` when initializing the client so each `posthog.capture()` call attempts delivery before it returns. +If you use [distributed tracing](#distributed-tracing), `sync_mode` doesn't apply to spans. Call `posthog.flush()` before the handler returns – see [Shutdown and short-lived processes](#shutdown-and-short-lived-processes). + ### Asyncio `AsyncPosthog` Keep one `AsyncPosthog` client for the lifetime of your application. Use buffered `capture()` by default, or `await capture_immediate()` when one invocation must wait for an event's delivery attempt. Call `await posthog.shutdown()` once during application cleanup. Don't shut down the client after each request. diff --git a/src/pages/docs/distributed-tracing/index.tsx b/src/pages/docs/distributed-tracing/index.tsx index 262736436671..bac7079e3a34 100644 --- a/src/pages/docs/distributed-tracing/index.tsx +++ b/src/pages/docs/distributed-tracing/index.tsx @@ -13,10 +13,10 @@ export const Content = () => {

PostHog Distributed Tracing works with the OpenTelemetry Protocol (OTLP). Use standard - OpenTelemetry libraries to send spans to PostHog using your project token. On Node.js,{' '} - posthog-node can also create and export spans itself, with no OpenTelemetry - dependency – see the{' '} - Node.js guide. + OpenTelemetry libraries to send spans to PostHog using your project token. On Node.js and + Python, the PostHog SDK can also create and export spans itself, with no OpenTelemetry + dependency – see the Node.js and{' '} + Python guides.

Distributed tracing is currently in beta. Setup details may change before general availability. @@ -30,8 +30,8 @@ export const Content = () => {

  • OpenTelemetry-compatible - Use standard OpenTelemetry SDKs, no PostHog packages - required. Works with any compatible client, or with posthog-node's own span - API. + required. Works with any compatible client, or with the span API in{' '} + posthog-node and posthog for Python.
  • Part of the observability suite - Traces use the same OpenTelemetry ingestion as{' '}