Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ sled = "0.34"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }

# Async
async-trait = "0.1"

# Utilities
uuid = { version = "1.11", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
Expand Down
87 changes: 84 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,62 @@ Events Tree:
- type, service_id, data, timestamp
```

### Per-Service Logging Architecture

Each service has its stdout and stderr captured and stored in a **bounded ring buffer** maintained by the daemon:

**Implementation Details:**
- **Buffer:** In-memory bounded ring buffer (default: 10,000 lines per service)
- **Rotation:** Automatic - oldest entries are dropped when buffer is full
- **Storage:** Each entry includes timestamp, stream (stdout/stderr), and message
- **Access:** Via CLI (`fabricks service logs`) or REST API (`GET /v1/services/:id/logs`)
- **Addressing:** Logs can be retrieved by service ID or service name

**Architecture:**
```
┌──────────────────────────────────────────────┐
│ WASM Service Instance │
│ │
│ stdout ──┐ │
│ │ │
│ stderr ──┼──────────────────────────────────┼───> Captured by Daemon
│ │ │
└───────────┴──────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ Daemon: Per-Service Log Buffer │
│ │
│ BoundedVecDeque<LogEntry> (10k lines) │
│ │
│ LogEntry { │
│ timestamp: DateTime<Utc>, │
│ stream: "stdout" | "stderr", │
│ message: String │
│ } │
│ │
│ Auto-rotation when full │
└──────────────────────────────────────────────┘
┌───────────────────────────┐
│ GET /v1/services/:id/logs│
│ ?tail=N │
└───────────────────────────┘
┌───────────────────────────┐
│ fabricks service logs │
│ <id> -n N --format text │
└───────────────────────────┘
```

**Benefits:**
- Logs accessible without needing to read daemon stdout
- CLI users can view service-specific logs
- Bounded memory usage (no unbounded log growth)
- Fast retrieval by service ID or name

---

## Network Architecture
Expand Down Expand Up @@ -715,18 +771,43 @@ fn can_connect(from_service: &Service, to_host: &str, to_port: u16) -> bool {
// 1. Check if services share a network
let shared_network = from_service.networks.iter()
.any(|net| to_service.networks.contains(net));

// 2. Check capability grant
let has_capability = from_service.capabilities.network.connect
.contains(&format!("{}:{}", to_host, to_port));

// 3. Check policy
let policy_allows = policy_engine.allows(from_service, to_service);

shared_network && has_capability && policy_allows
}
```

### Default Network

The daemon automatically creates a **default network** at startup with the following characteristics:

- **Name:** `"default"`
- **Access:** `NetworkAccess::External` - allows external internet access
- **Purpose:** Provides a convenient default for standalone services that need external connectivity

**Secure by Default:**

Services that don't explicitly join any network are **internal-only** (no external access). This secure-by-default approach requires explicit opt-in to external connectivity:

```bash
# Internal-only (no external access)
fabricks service run ./my-service

# With external access via default network
fabricks service run --network default ./my-service

# Multiple networks (default + custom)
fabricks service run --network default --network application ./my-service
```

For mortar compositions, services must explicitly list networks in the `fabricks-mortar.toml` configuration.

---

## Security Architecture
Expand Down
97 changes: 72 additions & 25 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -725,12 +725,22 @@ fabricks service run [OPTIONS] <PATH|TAG>
--env-file <FILE> Read environment from file
-v, --volume <SRC:DEST> Mount volumes (can be used multiple times)
--name <NAME> Assign a name to the instance
--network <NETWORK> Connect to network
--network <NETWORK> Connect to network (can be specified multiple times)
--rm Automatically remove when stopped
-d, --detach Run in background
--restart <POLICY> Restart policy [default: no] [possible: no, on-failure, always]
```

**Network Access:**

By default, services run through the daemon are **internal-only** (no external access) for security. To enable network access:

- **For external access:** Use `--network default` to join the default network which has `NetworkAccess::External`
- **For internal service-to-service communication:** Specify custom network names with `--network <name>`
- **Multiple networks:** Specify `--network` multiple times to join multiple networks

The **default network** is automatically created at daemon startup with external access enabled.

**How it Works:**

1. **Resolution Phase:**
Expand All @@ -753,14 +763,20 @@ fabricks service run ./examples/nodejs-hello
# Run by OCI tag
fabricks service run nodejs-hello:1.0.0

# Run with port mapping
fabricks service run -p 8080:8089 ./examples/nodejs-hello
# Run with port mapping and network access
fabricks service run -p 8080:8089 --network default ./examples/nodejs-hello

# Run with environment variables
fabricks service run -e LOG_LEVEL=debug nodejs-hello:1.0.0
fabricks service run -e LOG_LEVEL=debug --network default nodejs-hello:1.0.0

# Run in background with auto-restart
fabricks service run -d --restart on-failure nodejs-hello:1.0.0
# Run in background with auto-restart and external access
fabricks service run -d --restart on-failure --network default nodejs-hello:1.0.0

# Run on multiple networks (default + custom)
fabricks service run --network default --network application ./examples/nodejs-hello

# Run internal-only (no external access - secure by default)
fabricks service run ./examples/internal-worker
```

**Output:**
Expand Down Expand Up @@ -916,34 +932,65 @@ fabricks service logs [OPTIONS] <SERVICE>

**Options:**
```
--follow Follow log output
--tail <N> Number of lines to show from end [default: all]
--since <TIME> Show logs since timestamp or duration
--timestamps Show timestamps
--instance <ID> Show logs from specific instance
-n, --tail <N> Number of lines to show from end [default: all]
--format <FORMAT> Output format [default: text] [possible: text, json]
--follow Follow log output (future)
--since <TIME> Show logs since timestamp or duration (future)
--timestamps Show timestamps (future)
--instance <ID> Show logs from specific instance (future)
```

**Log Capture:**

Service stdout and stderr are captured by the daemon in a **per-service bounded ring buffer** (default: 10,000 lines). Logs are stored in memory and rotated automatically when the buffer is full. This allows CLI users to view logs without needing access to daemon stdout.

**Examples:**
```bash
# View service logs
# View all service logs
fabricks service logs product-service

# Follow logs
fabricks service logs --follow product-service
# View by service ID
fabricks service logs srv_abc123

# Last 100 lines
fabricks service logs --tail 100 product-service

# Logs from specific instance
fabricks service logs --instance product-service-0 product-service
```

**Output:**
```
2025-01-15T10:23:45Z [INFO] Starting server on :8080
2025-01-15T10:23:45Z [INFO] Connected to database
2025-01-15T10:24:00Z [INFO] Request: GET /products
2025-01-15T10:24:01Z [INFO] Response: 200 OK (15ms)
fabricks service logs -n 100 product-service

# JSON format (structured output)
fabricks service logs --format json product-service

# Text format (human-readable, default)
fabricks service logs --format text product-service
```

**Output (text format):**
```
2025-01-15T10:23:45Z [stdout] Starting server on :8080
2025-01-15T10:23:45Z [stdout] Connected to database
2025-01-15T10:24:00Z [stdout] Request: GET /products
2025-01-15T10:24:01Z [stdout] Response: 200 OK (15ms)
2025-01-15T10:24:02Z [stderr] Warning: deprecated API usage
```

**Output (json format):**
```json
[
{
"timestamp": "2025-01-15T10:23:45Z",
"stream": "stdout",
"message": "Starting server on :8080"
},
{
"timestamp": "2025-01-15T10:23:45Z",
"stream": "stdout",
"message": "Connected to database"
},
{
"timestamp": "2025-01-15T10:24:02Z",
"stream": "stderr",
"message": "Warning: deprecated API usage"
}
]
```

---
Expand Down
83 changes: 54 additions & 29 deletions docs/daemon-api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -585,7 +585,8 @@ Runs a module from OCI storage by tag or reference. This is the primary endpoint
"reference": "hello-http:0.1.0",
"args": [],
"env_vars": [],
"no_capabilities": false
"no_capabilities": false,
"networks": ["default"]
}
```

Expand All @@ -595,6 +596,7 @@ Runs a module from OCI storage by tag or reference. This is the primary endpoint
| `args` | array | No | Command-line arguments to pass to the module |
| `env_vars` | array | No | Environment variable overrides as `[key, value]` tuples |
| `no_capabilities` | boolean | No | Disable capability enforcement (default: false) |
| `networks` | array | No | Array of network names to join (e.g., `["default", "application"]`). If omitted or empty, service is internal-only. Use `["default"]` for external access. |

**Response:**
```json
Expand Down Expand Up @@ -840,58 +842,81 @@ curl --unix-socket /var/run/fabricks.sock \

### Get Service Logs

**Endpoint:** `GET /v1/services/{id}/logs`
**Endpoint:** `GET /v1/services/:id/logs`

Retrieves logs for a service. The `:id` parameter can be either a service ID or service name.

**Query Parameters:**
- `follow` - Stream logs (default: false)
- `tail` - Number of lines from end (default: all)
- `since` - Timestamp or duration
- `timestamps` - Include timestamps (default: false)
- `instance` - Specific instance ID
- `tail` - Number of lines from end (default: all, 0 means all)

**Log Storage:**

Service stdout and stderr are captured by the daemon in a **per-service bounded ring buffer** (default: 10,000 lines). Logs are stored in memory with automatic rotation when the buffer is full. Each log entry includes:
- Timestamp (when the log was captured)
- Stream (`"stdout"` or `"stderr"`)
- Message (the actual log line)

**Response (non-streaming):**
**Response:**
```json
{
"status": "success",
"data": {
"logs": [
"id": "srv_abc123",
"entries": [
{
"timestamp": "2025-01-15T10:23:45Z",
"instance": "api-service-a1b2c3-0",
"message": "[INFO] Starting server on :8080"
"timestamp": "2025-01-15T10:23:45.123456789Z",
"stream": "stdout",
"message": "Starting server on :8080"
},
{
"timestamp": "2025-01-15T10:23:46Z",
"instance": "api-service-a1b2c3-1",
"message": "[INFO] Starting server on :8080"
"timestamp": "2025-01-15T10:23:45.234567890Z",
"stream": "stdout",
"message": "Connected to database"
},
{
"timestamp": "2025-01-15T10:24:00.345678901Z",
"stream": "stderr",
"message": "Warning: deprecated API usage"
}
]
],
"count": 3
}
}
```

**Response (streaming with `follow=true`):**
**Response Fields:**
- `id` - Service ID
- `entries` - Array of log entries
- `timestamp` - RFC3339 timestamp with nanosecond precision
- `stream` - Either `"stdout"` or `"stderr"`
- `message` - Log message text
- `count` - Number of log entries returned

Server-Sent Events (SSE) stream:
```
event: log
data: {"timestamp":"2025-01-15T10:23:45Z","instance":"api-service-a1b2c3-0","message":"[INFO] Request processed"}
**cURL Examples:**
```bash
# Get all logs by service name
curl --unix-socket /var/run/fabricks.sock \
"http://localhost/v1/services/my-service/logs"

event: log
data: {"timestamp":"2025-01-15T10:23:46Z","instance":"api-service-a1b2c3-1","message":"[INFO] Request processed"}
```
# Get all logs by service ID
curl --unix-socket /var/run/fabricks.sock \
"http://localhost/v1/services/srv_abc123/logs"

**cURL Example:**
```bash
# Get last 100 lines
curl --unix-socket /var/run/fabricks.sock \
"http://localhost/v1/services/api-service-a1b2c3/logs?tail=100"
"http://localhost/v1/services/my-service/logs?tail=100"

# Stream logs
# Get all logs (explicit)
curl --unix-socket /var/run/fabricks.sock \
"http://localhost/v1/services/api-service-a1b2c3/logs?follow=true"
"http://localhost/v1/services/my-service/logs?tail=0"
```

**Notes:**
- If service is not found, returns 404
- If service has no logs yet, returns empty `entries` array with `count: 0`
- Logs are per-service (not per-instance) - all instances write to the same log buffer
- Future: streaming with SSE, filtering by instance, time-based filtering

---

### Get Service Stats
Expand Down
Loading
Loading