Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Olake-UI offers an intuitive web interface to configure, monitor, and manage you
- [Server Readme](/server/README.md)
- [UI Figma Design](https://www.figma.com/design/FwLnU97I8LjtYNREPyYofc/Olake-Design-Community?node-id=1-46&p=f&t=y3BIsLTUaXhHwYLG-0)
- [API Contracts](/api-contract.md)
- [Declarative CLI Bundles](/home/sabino/Downloads/olake-ui-fork/docs/declarative-cli-bundles.md)

## Running with Docker Compose

Expand Down Expand Up @@ -121,6 +122,15 @@ curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-c

To upgrade from legacy `docker-compose.yml` that was used before **Jan 30th 2026** , follow this [documentation](https://olake.io/docs/install/olake-ui/#:~:text=To%20move%20from,Legacy%20Setup.).

## Declarative apply and export

OLake UI can now consume a declarative CLI bundle and export an existing UI job back into the same bundle shape.

- apply endpoint: `POST /api/v1/project/{projectid}/jobs/apply-cli-bundle`
- export endpoint: `GET /api/v1/project/{projectid}/jobs/{id}/export-cli-bundle`

The bundle contract and examples are documented in [docs/declarative-cli-bundles.md](/home/sabino/Downloads/olake-ui-fork/docs/declarative-cli-bundles.md).

## Contributing

We ❤️ contributions big or small check our [Bounty Program](https://olake.io/docs/community/issues-and-prs#goodies). As always, thanks to our amazing contributors!.
Expand Down
67 changes: 67 additions & 0 deletions api-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -724,9 +724,76 @@ http://localhost:8080
}
```

### Apply Declarative CLI Bundle

- **Endpoint**: `/api/v1/project/:projectid/jobs/apply-cli-bundle`
- **Method**: POST
- **Description**: Preview or apply a declarative OLake CLI bundle into the current project
- **Headers**: `Authorization: Bearer <token>`
- **Query Params**:
- `dry_run=true|false`
- `prune=true|false` (reserved for future prune behavior)
- **Request Body**: `multipart/form-data`
- `bundle`: `.zip` or `.tar.gz` archive containing `source.json`, `destination.json`, `streams.json`, and optional `state.json` / `olake-ui.json`
- **Response**:

```json
{
"success": true,
"message": "bundle preview generated successfully",
"data": {
"dry_run": true,
"prune": false,
"bundle": "mongodb-orders",
"effective": {
"apply_identity": "mongodb-orders",
"job_name": "mongodb-orders-cdc",
"source_name": "mongodb-orders-source",
"source_type": "mongodb",
"source_version": "v0.5.1",
"destination_name": "parquet-minio-local",
"destination_type": "parquet",
"destination_version": "v0.5.1",
"frequency": "0 * * * *",
"activate": true
},
"source": {
"action": "updated",
"name": "mongodb-orders-source",
"fields": ["config"]
},
"destination": {
"action": "unchanged",
"name": "parquet-minio-local"
},
"job": {
"action": "created",
"name": "mongodb-orders-cdc",
"fields": ["name", "source", "destination", "frequency", "activate", "streams_config"]
},
"state": {
"action": "preserved"
}
}
}
```

### Export Declarative CLI Bundle

- **Endpoint**: `/api/v1/project/:projectid/jobs/:id/export-cli-bundle`
- **Method**: GET
- **Description**: Export an existing UI job into a declarative CLI bundle archive
- **Headers**: `Authorization: Bearer <token>`
- **Query Params**:
- `format=zip|tar.gz` (default `zip`)
- `include_state=true|false` (default `false`)
- **Response**:
- binary archive download with:
- `source.json`
- `destination.json`
- `streams.json`
- `olake-ui.json`
- optional `state.json`

### Delete Job

Expand Down
99 changes: 99 additions & 0 deletions docs/declarative-cli-bundles.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Declarative CLI Bundles

OLake UI can now reconcile jobs from a file bundle that matches the OLake CLI contract and export existing UI jobs back into the same bundle shape.

## Bundle layout

Required files:

- `source.json`
- `destination.json`
- `streams.json`

Optional files:

- `state.json`
- `olake-ui.json`

Example layout:

```text
mongodb-orders/
├── source.json
├── destination.json
├── streams.json
└── olake-ui.json
```

## `olake-ui.json`

`olake-ui.json` is the only server-side overlay. It carries the metadata that OLake UI needs but the raw CLI files do not include.

```json
{
"apply_identity": "mongodb-orders",
"job_name": "mongodb-orders-cdc",
"source_name": "mongodb-orders-source",
"source_type": "mongodb",
"source_version": "v0.5.1",
"destination_name": "parquet-minio-local",
"destination_type": "parquet",
"destination_version": "v0.5.1",
"frequency": "0 * * * *",
"activate": true
}
```

Notes:

- `source_type`, `source_version`, and `destination_version` are effectively required for server-side apply.
- `destination_type` can be omitted when `destination.json` has a top-level `type`.
- If names are omitted, the server infers deterministic defaults from the bundle name.
- If `state.json` is omitted, the server preserves existing UI state during apply.

## Apply API

Preview:

```bash
curl -sS -b cookies.txt \
-F bundle=@mongodb-orders.zip \
"http://localhost:8000/api/v1/project/123/jobs/apply-cli-bundle?dry_run=true" | jq
```

Apply:

```bash
curl -sS -b cookies.txt \
-F bundle=@mongodb-orders.zip \
"http://localhost:8000/api/v1/project/123/jobs/apply-cli-bundle" | jq
```

Apply is idempotent:

- same bundle + same current state = `unchanged`
- config drift = `updated`
- missing resource = `created`
- omitted `state.json` = existing state is `preserved`

## Export API

Export a UI job back into a CLI bundle:

```bash
curl -sS -b cookies.txt \
"http://localhost:8000/api/v1/project/123/jobs/42/export-cli-bundle?format=zip" \
-o mongodb-orders.zip
```

Include checkpoint state when you want a reconstruction bundle:

```bash
curl -sS -b cookies.txt \
"http://localhost:8000/api/v1/project/123/jobs/42/export-cli-bundle?format=zip&include_state=true" \
-o mongodb-orders-with-state.zip
```

## Example bundle

See [examples/cli-bundles/mongodb-parquet](/home/sabino/Downloads/olake-ui-fork/examples/cli-bundles/mongodb-parquet).
11 changes: 11 additions & 0 deletions examples/cli-bundles/mongodb-parquet/destination.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"type": "PARQUET",
"writer": {
"s3_bucket": "bronze",
"s3_region": "us-east-1",
"s3_path": "/orders",
"s3_access_key": "minioadmin",
"s3_secret_key": "minioadmin",
"s3_endpoint": "http://minio:9000"
}
}
12 changes: 12 additions & 0 deletions examples/cli-bundles/mongodb-parquet/olake-ui.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"apply_identity": "mongodb-parquet-example",
"job_name": "mongodb-orders-cdc",
"source_name": "mongodb-orders-source",
"source_type": "mongodb",
"source_version": "v0.5.1",
"destination_name": "parquet-minio-local",
"destination_type": "parquet",
"destination_version": "v0.5.1",
"frequency": "0 * * * *",
"activate": true
}
10 changes: 10 additions & 0 deletions examples/cli-bundles/mongodb-parquet/source.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"hosts": ["mongo:27017"],
"username": "olake",
"password": "olake",
"authdb": "admin",
"replica_set": "rs0",
"srv": false,
"database": "orders",
"use_iam": false
}
48 changes: 48 additions & 0 deletions examples/cli-bundles/mongodb-parquet/streams.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{
"selected_streams": {
"orders": [
{
"stream_name": "orders",
"partition_regex": "",
"normalization": true,
"append_mode": false
}
]
},
"streams": [
{
"stream": {
"name": "orders",
"namespace": "orders",
"type_schema": {
"properties": {
"_id": {
"type": [
"string"
],
"destination_column_name": "_id"
},
"updated_at": {
"type": [
"timestamp_milli",
"null"
],
"destination_column_name": "updated_at"
}
}
},
"supported_sync_modes": [
"cdc",
"full_refresh"
],
"source_defined_primary_key": [
"_id"
],
"available_cursor_fields": [
"updated_at"
],
"sync_mode": "cdc"
}
}
]
}
4 changes: 4 additions & 0 deletions server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ make create-user username=admin password=yourpassword email=admin@example.com

All API Endpoints: [Postman](https://solar-capsule-662043.postman.co/workspace/Olake-Server~ad9c900c-0376-42e2-adf2-e3137b92b325/collection/24907154-6eaf11b3-4e36-4ec3-a05a-3fa3720125ee?action=share&creator=24907154&active-environment=24907154-dcc91e95-6699-48cb-bbe0-e0e92b9800bd)

Declarative bundle docs and examples: [../docs/declarative-cli-bundles.md](/home/sabino/Downloads/olake-ui-fork/docs/declarative-cli-bundles.md)

### Authentication

- POST `/login` - User login
Expand All @@ -97,8 +99,10 @@ All API Endpoints: [Postman](https://solar-capsule-662043.postman.co/workspace/O

- GET `/jobs` - Get all jobs
- POST `/jobs` - Create a new job
- POST `/jobs/apply-cli-bundle` - Preview or apply a declarative CLI bundle
- PUT `/jobs/:id` - Update a job
- DELETE `/jobs/:id` - Delete a job
- GET `/jobs/:id/export-cli-bundle` - Export a job back into a CLI bundle

### Users

Expand Down
Loading