diff --git a/docs/pages/concepts.mdx b/docs/pages/concepts.mdx index 01c00f5d..841f6424 100644 --- a/docs/pages/concepts.mdx +++ b/docs/pages/concepts.mdx @@ -39,7 +39,7 @@ Outpost consists of 3 services that can either be run as a single deployment or Required for log storage. - PostgreSQL -- [ClickHouse (planned)](https://github.com/hookdeck/outpost/issues/52) +- ClickHouse ## Tenant Destination Types diff --git a/docs/pages/guides/deployment.mdx b/docs/pages/guides/deployment.mdx index 2f4f1d1a..c4eb0019 100644 --- a/docs/pages/guides/deployment.mdx +++ b/docs/pages/guides/deployment.mdx @@ -41,7 +41,7 @@ Once we have completed [initial benchmarking](https://github.com/hookdeck/outpos Outpost requires the following dependencies: - Redis -- PostgreSQL +- PostgreSQL or ClickHouse - One of the following message queue systems (for the queues listed below): RabbitMQ, AWS SQS, Azure Service Bus, or GCP Pub/Sub - Log queue - Delivery queue diff --git a/docs/pages/guides/upgrade-v0.13.mdx b/docs/pages/guides/upgrade-v0.13.mdx new file mode 100644 index 00000000..3ad03184 --- /dev/null +++ b/docs/pages/guides/upgrade-v0.13.mdx @@ -0,0 +1,261 @@ +--- +title: "Upgrade to v0.13" +--- + +This guide covers breaking changes and migration steps when upgrading from v0.12 to v0.13. + +## Breaking Changes Overview + +| Change | Impact | Action Required | +| --- | --- | --- | +| [Delivery renamed to Attempt](#delivery-renamed-to-attempt) | All delivery-related endpoints, response types, and config | Update API routes, response handling, and config vars | +| [Route restructuring](#route-restructuring) | Tenant-scoped event, schema, and topic endpoints removed | Update API client paths | +| [New retry mechanism](#new-retry-mechanism) | Retry endpoint moved and changed format | Update retry calls | +| [ID prefix delimiter removed](#id-prefix-delimiter-removed) | Custom ID prefixes | Include delimiter in prefix value (e.g., `evt_` instead of `evt`) | +| [Refined API schema](#refined-api-schema) | Pagination, query filters, and error responses | Update response parsing, query parameters, and error handling | + +## Delivery Renamed to Attempt + +The "delivery" concept in the API has been renamed to "attempt." An attempt represents a single delivery attempt of an event to a destination — the same concept, just a clearer name. + +### Attempt Schema + +```json +{ + "id": "atm_123", + "status": "success", + "delivered_at": "2024-01-01T00:00:05Z", + "code": "200", + "attempt_number": 0, + "manual": false, + "event_id": "evt_123", + "destination_id": "des_456", + "response_data": { + "status_code": 200, + "body": "{\"status\":\"ok\"}", + "headers": { "content-type": "application/json" } + } +} +``` + +- `id`: Unique attempt identifier (default prefix configurable via `IDGEN_ATTEMPT_PREFIX`) +- `status`: `success` or `failed` +- `code`: String response status code or error code +- `attempt_number`: 0 for first attempt, 1+ for retries +- `manual`: Whether this was a manually triggered retry +- `event_id`: The ID of the associated event +- `destination_id`: The destination ID this attempt was sent to +- `response_data`: Response details (only included when `include=response_data` is specified) +- `event`: Expanded event object (only present when `include=event` or `include=event.data` is specified) + +### The `include` Parameter + +The attempt endpoints support an `include` query parameter to expand related data inline: + +| Value | Effect | +| --- | --- | +| `include=event` | Includes event object with `id`, `topic`, `time`, `eligible_for_retry`, `metadata` | +| `include=event.data` | Same as `event` plus the full event `data` payload | +| `include=response_data` | Includes `response_data` with status code, body, and headers | + +Multiple values can be combined: `?include=event.data&include=response_data` + +### Configuration + +| v0.12 | v0.13 | +| --- | --- | +| `IDGEN_DELIVERY_PREFIX` | **Removed** — use `IDGEN_ATTEMPT_PREFIX` | +| `IDGEN_DELIVERY_EVENT_PREFIX` | **Removed** | + +If you had customized `IDGEN_DELIVERY_PREFIX`, update it to `IDGEN_ATTEMPT_PREFIX`. + +## Route Restructuring + +v0.13 restructures the API routes. Events and attempts are now accessed through top-level endpoints or scoped under destinations, rather than deeply nested under tenants. Destination type schemas and topics are no longer tenant-scoped. + +### Removed Routes + +| v0.12 Route | v0.13 Replacement | +| --- | --- | +| `GET /tenants/:tenant_id/events` | `GET /events?tenant_id=:tenant_id` | +| `GET /tenants/:tenant_id/events/:event_id` | `GET /events/:event_id` | +| `GET /tenants/:tenant_id/events/:event_id/deliveries` | `GET /attempts?event_id=:event_id` or `GET /tenants/:tenant_id/destinations/:destination_id/attempts?event_id=:event_id` | +| `GET /tenants/:tenant_id/destinations/:destination_id/events` | `GET /tenants/:tenant_id/destinations/:destination_id/attempts` | +| `GET /tenants/:tenant_id/destinations/:destination_id/events/:event_id` | `GET /tenants/:tenant_id/destinations/:destination_id/attempts/:attempt_id` | +| `POST /tenants/:tenant_id/destinations/:destination_id/events/:event_id/retry` | `POST /retry` with `{ "event_id": "...", "destination_id": "..." }` | +| `GET /tenants/:tenant_id/destination-types` | `GET /destination-types` | +| `GET /tenants/:tenant_id/destination-types/:type` | `GET /destination-types/:type` | +| `GET /tenants/:tenant_id/topics` | `GET /topics` | + +### New Routes + +| Route | Description | +| --- | --- | +| `GET /events` | List events (admin cross-tenant, or filtered by `tenant_id`) | +| `GET /events/:event_id` | Get a specific event by ID | +| `GET /attempts` | List attempts (admin cross-tenant, with filters) | +| `GET /attempts/:attempt_id` | Get a specific attempt by ID | +| `GET /tenants/:tenant_id/destinations/:destination_id/attempts` | List attempts for a destination | +| `GET /tenants/:tenant_id/destinations/:destination_id/attempts/:attempt_id` | Get a specific attempt for a destination | +| `POST /retry` | Retry event delivery | + +**Action:** Update all API client paths to the new routes. + +## New Retry Mechanism + +The retry endpoint has been moved from a deeply nested path to a standalone top-level endpoint with a request body: + +**v0.12:** +``` +POST /tenants/:tenant_id/destinations/:destination_id/events/:event_id/retry +``` + +**v0.13:** +``` +POST /retry +``` +```json +{ + "event_id": "evt_123", + "destination_id": "des_456" +} +``` + +When authenticated with a Tenant JWT, only events belonging to that tenant can be retried. When authenticated with Admin API Key, events from any tenant can be retried. + +**Action:** Update retry calls to use `POST /retry` with `event_id` and `destination_id` in the request body. + +## ID Prefix Delimiter Removed + +ID generation no longer adds a default `_` delimiter between the prefix and the generated ID. The prefix value is now used as-is. + +If you use custom ID prefixes, include the delimiter in the prefix value: + +| v0.12 | v0.13 | +| --- | --- | +| `IDGEN_EVENT_PREFIX=evt` (produces `evt_xxx`) | `IDGEN_EVENT_PREFIX=evt_` (produces `evt_xxx`) | +| `IDGEN_DESTINATION_PREFIX=des` (produces `des_xxx`) | `IDGEN_DESTINATION_PREFIX=des_` (produces `des_xxx`) | + +This applies to all `IDGEN_*_PREFIX` config vars: `IDGEN_EVENT_PREFIX`, `IDGEN_DESTINATION_PREFIX`, and `IDGEN_ATTEMPT_PREFIX`. + +**Action:** Append `_` (or your desired delimiter) to all custom ID prefix values. + +## Refined API Schema + +v0.13 refines the API pagination, query filters, sorting, and error responses. + +### Pagination Response Format + +All paginated list endpoints now return a new response envelope. + +**v0.12:** +```json +{ + "count": 42, + "data": [{ "id": "evt_123", "..." }], + "next": "MTcwNDA2NzIwMA==", + "prev": "" +} +``` + +**v0.13:** +```json +{ + "models": [{ "id": "evt_123", "..." }], + "pagination": { + "order_by": "time", + "dir": "desc", + "limit": 100, + "next": "MTcwNDA2NzIwMA==", + "prev": null + } +} +``` + +Key differences: +- `data` renamed to `models` +- `next`/`prev` cursors moved into a `pagination` object alongside `order_by`, `dir`, and `limit` +- `count` removed from event/attempt list responses (still present on tenant list) +- Empty cursors are now `null` instead of `""` + +**Action:** Update all code that parses paginated responses: +- `response.data` → `response.models` +- `response.next` → `response.pagination.next` +- `response.prev` → `response.pagination.prev` + +### Query Filter Format + +Time filter parameters have been updated to use a structured format: + +**Event and attempt list endpoints:** + +| v0.12 | v0.13 | +| --- | --- | +| `?start=2024-01-01T00:00:00Z` | `?time[gte]=2024-01-01T00:00:00Z` | +| `?end=2024-01-31T23:59:59Z` | `?time[lte]=2024-01-31T23:59:59Z` | + +**Tenant list endpoint:** + +| v0.12 | v0.13 | +| --- | --- | +| *(not available)* | `?created_at[gte]=2024-01-01T00:00:00Z` | +| *(not available)* | `?created_at[lte]=2024-01-31T23:59:59Z` | + +The `time[gte]`/`time[lte]` and `created_at[gte]`/`created_at[lte]` filters support both `YYYY-MM-DD` and full RFC3339 timestamps. + +### Sorting Parameters + +The `order` query parameter on `GET /tenants` has been split into two parameters: + +| v0.12 | v0.13 | +| --- | --- | +| `?order=desc` | `?order_by=created_at&dir=desc` | + +The new `order_by` and `dir` parameters are also available on event and attempt list endpoints: + +| Parameter | Values | Default | +| --- | --- | --- | +| `order_by` | `time` (events/attempts), `created_at` (tenants) | Varies by endpoint | +| `dir` | `asc`, `desc` | `desc` | + +### Error Response Format + +Error responses now include a `status` field and use a consistent structure: + +**v0.12:** +```json +{ + "message": "validation error", + "data": { "email": "required", "password": "min" } +} +``` + +**v0.13:** +```json +{ + "status": 422, + "message": "validation error", + "data": ["email is required", "password must be at least 6 characters"] +} +``` + +Key differences: +- New `status` field mirrors the HTTP status code +- Validation errors in `data` changed from a `{ field: tag }` object to an array of human-readable messages + +## Upgrade Checklist + +1. **Before upgrading:** + - [ ] Update API clients to use `/attempts` routes instead of `/deliveries` and destination-scoped `/events` routes + - [ ] Update retry calls to use `POST /retry` with `event_id` and `destination_id` in request body + - [ ] Update destination-types and topics calls to use unscoped routes (`/destination-types`, `/topics`) + - [ ] Update event access from `GET /tenants/:id/events/:id` to `GET /events/:id` + - [ ] Rename `IDGEN_DELIVERY_PREFIX` to `IDGEN_ATTEMPT_PREFIX` and remove `IDGEN_DELIVERY_EVENT_PREFIX` + - [ ] Append delimiter to all custom `IDGEN_*_PREFIX` config values (e.g., `evt` → `evt_`) + - [ ] Update response parsing for the new pagination envelope (`models`/`pagination`) + - [ ] Update `GET /tenants` sorting from `order` to `order_by` + `dir` + - [ ] Update time filter params from `start`/`end` to `time[gte]`/`time[lte]` + - [ ] Update error response handling if parsing `data` field + +2. **Upgrade:** + - [ ] Update Outpost to v0.13 and restart — migrations run automatically on startup diff --git a/docs/pages/overview.mdx b/docs/pages/overview.mdx index f8527964..1a224869 100644 --- a/docs/pages/overview.mdx +++ b/docs/pages/overview.mdx @@ -19,7 +19,7 @@ Planned destination types include AWS EventBridge and Kafka. Outpost is built and maintained by [Hookdeck](https://hookdeck.com?ref=github-outpost). It's written in Go and distributed as a binary and Docker container under the Apache-2.0 license. -Outpost has minimal dependencies (Redis, PostgreSQL, and one of the supported message queues), is 100% backward compatible with your existing webhooks implementation and is highly optimized for high-throughput, low-cost operation. +Outpost has minimal dependencies (Redis, PostgreSQL or ClickHouse, and one of the supported message queues), is 100% backward compatible with your existing webhooks implementation and is highly optimized for high-throughput, low-cost operation. ## Ethos diff --git a/docs/pages/references/configuration.mdx b/docs/pages/references/configuration.mdx index 528fa31a..82e4a4fc 100644 --- a/docs/pages/references/configuration.mdx +++ b/docs/pages/references/configuration.mdx @@ -57,7 +57,7 @@ Global configurations are provided through env variables or a YAML file. ConfigM | `DESTINATIONS_WEBHOOK_DISABLE_DEFAULT_SIGNATURE_HEADER` | If true, disables adding the default 'X-Outpost-Signature' header to webhook requests. Only applies to 'default' mode. | `false` | No | | `DESTINATIONS_WEBHOOK_DISABLE_DEFAULT_TIMESTAMP_HEADER` | If true, disables adding the default 'X-Outpost-Timestamp' header to webhook requests. Only applies to 'default' mode. | `false` | No | | `DESTINATIONS_WEBHOOK_DISABLE_DEFAULT_TOPIC_HEADER` | If true, disables adding the default 'X-Outpost-Topic' header to webhook requests. Only applies to 'default' mode. | `false` | No | -| `DESTINATIONS_WEBHOOK_HEADER_PREFIX` | Prefix for metadata headers added to webhook requests. Defaults to 'x-outpost-' in 'default' mode and 'webhook-' in 'standard' mode. | `x-outpost-` | No | +| `DESTINATIONS_WEBHOOK_HEADER_PREFIX` | Prefix for metadata headers added to webhook requests. Defaults to 'x-outpost-' in 'default' mode and 'webhook-' in 'standard' mode. | `nil` | No | | `DESTINATIONS_WEBHOOK_MODE` | Webhook mode: 'default' for customizable webhooks or 'standard' for Standard Webhooks specification compliance. Defaults to 'default'. | `nil` | No | | `DESTINATIONS_WEBHOOK_PROXY_URL` | Proxy URL for routing webhook requests through a proxy server. Supports HTTP and HTTPS proxies. When configured, all outgoing webhook traffic will be routed through the specified proxy. | `nil` | No | | `DESTINATIONS_WEBHOOK_SIGNATURE_ALGORITHM` | Algorithm used for signing webhook requests (e.g., 'hmac-sha256'). Only applies to 'default' mode. | `hmac-sha256` | No | @@ -238,7 +238,7 @@ destinations: disable_default_topic_header: false # Prefix for metadata headers added to webhook requests. Defaults to 'x-outpost-' in 'default' mode and 'webhook-' in 'standard' mode. - header_prefix: "x-outpost-" + header_prefix: "" # Webhook mode: 'default' for customizable webhooks or 'standard' for Standard Webhooks specification compliance. Defaults to 'default'. mode: "" diff --git a/docs/pages/references/roadmap.mdx b/docs/pages/references/roadmap.mdx index 32d77a57..2de574bf 100644 --- a/docs/pages/references/roadmap.mdx +++ b/docs/pages/references/roadmap.mdx @@ -13,47 +13,105 @@ title: "Outpost Roadmap" - [Amazon EventBridge](https://github.com/hookdeck/outpost/issues/201) - [Kafka](https://github.com/hookdeck/outpost/issues/141) -## Later +## Releases -### Core Functionality +### v0.13.0 + +#### Core Functionality + +- ✅ [ClickHouse log storage](https://github.com/hookdeck/outpost/issues/52) + +### v0.12.0 + +See [v0.12.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.12.0). + +#### Core Functionality + +- ✅ List tenants API with RediSearch +- ✅ `/tenants` prefix for tenant-scoped API routes +- ✅ Redis migration flow improvements + +### v0.11.0 + +See [v0.11.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.11.0). + +#### Fixes + +- ✅ Webhook connection error retry +- ✅ Retry ID collision for multi-destination events +- ✅ Dragonfly compatibility + +### v0.10.0 + +See [v0.10.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.10.0). + +#### Core Functionality + +- ✅ Destination filtering +- ✅ Custom headers for webhook destinations + +### v0.9.0 -- [ClickHouse log storage](https://github.com/hookdeck/outpost/issues/52) +See [v0.9.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.9.0). -## Previous Notable Milestones +#### Core Functionality + +- ✅ Publish endpoint `duplicate` field for idempotency detection +- ✅ Configurable retry monitor polling (`RETRY_POLL_BACKOFF_MS`) +- ✅ Self-managed message queue infrastructure (`MQS_AUTO_PROVISION`) +- ✅ Structured health check endpoint + +### v0.8.0 + +See [v0.8.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.8.0). + +#### Core Functionality + +- ✅ [Standard Webhooks](https://github.com/standard-webhooks/standard-webhooks) support +- ✅ Configurable ID generation (UUID v4, UUID v7, nanoid) +- ✅ Custom retry schedules (`RETRY_SCHEDULE`) +- ✅ Destination `metadata` and `delivery_metadata` fields +- ✅ Configurable idempotency key TTL +- ✅ Tenant `metadata` field -## v0.7.0 +### v0.7.0 See [v0.7.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.7.0). -### Destination Types +#### Destination Types -- [GCP Pub/Sub](https://github.com/hookdeck/outpost/issues/140) +- ✅ [GCP Pub/Sub](https://github.com/hookdeck/outpost/issues/140) -## v0.6.0 +#### Core Functionality + +- ✅ Webhook delivery through proxy +- ✅ Multi-deployment support (`deployment_id`) + +### v0.6.0 See [v0.6.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.6.0). -### Core Functionality +#### Core Functionality -- [Redis cluster support](https://github.com/hookdeck/outpost/pull/465) +- ✅ [Redis cluster support](https://github.com/hookdeck/outpost/pull/465) -## v0.5.0 +### v0.5.0 See [v0.5.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.5.0). -### Destination Types +#### Destination Types -- [AWS S3](https://github.com/orgs/hookdeck/projects/21/views/1?filterQuery=s3&pane=issue&itemId=113373337&issue=hookdeck%7Coutpost%7C418) +- ✅ [AWS S3](https://github.com/hookdeck/outpost/issues/418) ### v0.4.0 See [v0.4.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.4.0). -### Publish Message Queues +#### Publish Message Queues - ✅ [Azure Service Bus](https://github.com/hookdeck/outpost/pull/435) -### Destination Types +#### Destination Types - ✅ [Azure Service Bus](https://github.com/hookdeck/outpost/issues/241) @@ -82,7 +140,7 @@ See [v0.1.0 release notes](https://github.com/hookdeck/outpost/releases/tag/v0.1 - ✅ [Hookdeck Event Gateway](https://github.com/hookdeck/outpost/issues/136) - ✅ [Amazon Kinesis](https://github.com/hookdeck/outpost/issues/305) -### ALPHA +### Alpha #### Core Functionality