From ae6febe273b5e92e1b3d29764cf018da3dd07b7e Mon Sep 17 00:00:00 2001 From: Richard Kiene Date: Sun, 1 Feb 2026 11:06:12 -0700 Subject: [PATCH] Add Go/TinyGo support, daemon shutdown, and log capture (#31) Implements comprehensive Go support using TinyGo, graceful daemon shutdown, and per-service log capture with bounded ring buffers. Go Builder: - Add GoBuilder using TinyGo 0.32+ with wasip2 target - Support both command and HTTP service types - Add toolchain verification for TinyGo and wasm-tools - Add go-hello and go-http working examples Daemon Shutdown: - Add 'fabricks daemon stop' CLI command - Add POST /v1/daemon/shutdown API endpoint - Implement graceful shutdown with 5s timeout - Clean service and resource cleanup on shutdown Log Capture: - Add per-service bounded ring buffer (10k lines default) - Capture stdout/stderr with timestamps and stream type - Add 'fabricks service logs' CLI command - Add GET /v1/services/:id/logs API endpoint - Support retrieval by service ID or name Runtime Improvements: - Fix fuel metering: add default 10 billion fuel limit - Add LogCaptureSink and LogWriter for output capture - Add run_with_output() method for custom stdio - Remove excessive debug logging Network Access: - Add auto-created "default" network with external access - Make standalone services internal-only by default (secure by default) - Require explicit --network flag for external access - Support multiple networks via repeated --network flag Documentation: - Update architecture.md with log capture design - Update CLI reference with daemon stop and service logs - Update API reference with shutdown and logs endpoints - Document network access patterns and security model --- Cargo.toml | 3 + docs/architecture.md | 87 ++++++- docs/cli-reference.md | 97 ++++++-- docs/daemon-api-reference.md | 83 ++++--- docs/fabrickfile-mortar-reference.md | 14 ++ examples/go-hello/Fabrickfile | 16 ++ .../go-hello/examples/go-hello/Fabrickfile | 16 ++ .../examples/go-hello/go-hello.core.wasm | Bin 0 -> 508155 bytes examples/go-hello/examples/go-hello/main.go | 7 + .../go-hello/examples/go-http/Fabrickfile | 22 ++ examples/go-hello/examples/go-http/main.go | 20 ++ examples/go-hello/go-hello.wasm | Bin 0 -> 850662 bytes examples/go-hello/go.mod | 3 + examples/go-hello/main.go | 7 + examples/go-http/Fabrickfile | 22 ++ examples/go-http/go-http.wasm | Bin 0 -> 3196339 bytes examples/go-http/go.mod | 3 + examples/go-http/main.go | 20 ++ fabricks-e2e/tests/daemon_api.rs | 7 +- fabricks-runtime/Cargo.toml | 2 + fabricks-runtime/src/lib.rs | 2 + fabricks-runtime/src/output.rs | 227 ++++++++++++++++++ fabricks-runtime/src/runtime.rs | 103 ++++++++ fabricks-runtime/tests/output_capture_test.rs | 123 ++++++++++ fabricks/src/builders/go.rs | 186 ++++++++++++++ fabricks/src/builders/mod.rs | 19 +- fabricks/src/cli.rs | 33 +++ fabricks/src/commands/daemon.rs | 13 + fabricks/src/commands/run.rs | 1 + fabricks/src/commands/service.rs | 197 +++++++++------ fabricks/src/daemon_client.rs | 59 +++++ fabricksd/src/api/handlers/daemon.rs | 22 ++ fabricksd/src/api/handlers/services.rs | 81 ++++++- fabricksd/src/api/router.rs | 5 + fabricksd/src/main.rs | 2 +- fabricksd/src/network/manager.rs | 27 +++ fabricksd/src/service/handle.rs | 59 ++++- fabricksd/src/service/logs.rs | 168 +++++++++++++ fabricksd/src/service/manager.rs | 26 ++ fabricksd/src/service/mod.rs | 2 + fabricksd/src/shutdown.rs | 21 +- fabricksd/src/state.rs | 4 + 42 files changed, 1644 insertions(+), 165 deletions(-) create mode 100644 examples/go-hello/Fabrickfile create mode 100644 examples/go-hello/examples/go-hello/Fabrickfile create mode 100644 examples/go-hello/examples/go-hello/go-hello.core.wasm create mode 100644 examples/go-hello/examples/go-hello/main.go create mode 100644 examples/go-hello/examples/go-http/Fabrickfile create mode 100644 examples/go-hello/examples/go-http/main.go create mode 100644 examples/go-hello/go-hello.wasm create mode 100644 examples/go-hello/go.mod create mode 100644 examples/go-hello/main.go create mode 100644 examples/go-http/Fabrickfile create mode 100644 examples/go-http/go-http.wasm create mode 100644 examples/go-http/go.mod create mode 100644 examples/go-http/main.go create mode 100644 fabricks-runtime/src/output.rs create mode 100644 fabricks-runtime/tests/output_capture_test.rs create mode 100644 fabricks/src/builders/go.rs create mode 100644 fabricksd/src/service/logs.rs diff --git a/Cargo.toml b/Cargo.toml index 1f2a32c..310a7bd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -81,6 +81,9 @@ sled = "0.34" tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] } +# Async +async-trait = "0.1" + # Utilities uuid = { version = "1.11", features = ["v4", "serde"] } chrono = { version = "0.4", features = ["serde"] } diff --git a/docs/architecture.md b/docs/architecture.md index 8b3cdc0..a2c4bff 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -688,6 +688,62 @@ Events Tree: - type, service_id, data, timestamp ``` +### Per-Service Logging Architecture + +Each service has its stdout and stderr captured and stored in a **bounded ring buffer** maintained by the daemon: + +**Implementation Details:** +- **Buffer:** In-memory bounded ring buffer (default: 10,000 lines per service) +- **Rotation:** Automatic - oldest entries are dropped when buffer is full +- **Storage:** Each entry includes timestamp, stream (stdout/stderr), and message +- **Access:** Via CLI (`fabricks service logs`) or REST API (`GET /v1/services/:id/logs`) +- **Addressing:** Logs can be retrieved by service ID or service name + +**Architecture:** +``` +┌──────────────────────────────────────────────┐ +│ WASM Service Instance │ +│ │ +│ stdout ──┐ │ +│ │ │ +│ stderr ──┼──────────────────────────────────┼───> Captured by Daemon +│ │ │ +└───────────┴──────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────┐ +│ Daemon: Per-Service Log Buffer │ +│ │ +│ BoundedVecDeque (10k lines) │ +│ │ +│ LogEntry { │ +│ timestamp: DateTime, │ +│ stream: "stdout" | "stderr", │ +│ message: String │ +│ } │ +│ │ +│ Auto-rotation when full │ +└──────────────────────────────────────────────┘ + │ + ▼ + ┌───────────────────────────┐ + │ GET /v1/services/:id/logs│ + │ ?tail=N │ + └───────────────────────────┘ + │ + ▼ + ┌───────────────────────────┐ + │ fabricks service logs │ + │ -n N --format text │ + └───────────────────────────┘ +``` + +**Benefits:** +- Logs accessible without needing to read daemon stdout +- CLI users can view service-specific logs +- Bounded memory usage (no unbounded log growth) +- Fast retrieval by service ID or name + --- ## Network Architecture @@ -715,18 +771,43 @@ fn can_connect(from_service: &Service, to_host: &str, to_port: u16) -> bool { // 1. Check if services share a network let shared_network = from_service.networks.iter() .any(|net| to_service.networks.contains(net)); - + // 2. Check capability grant let has_capability = from_service.capabilities.network.connect .contains(&format!("{}:{}", to_host, to_port)); - + // 3. Check policy let policy_allows = policy_engine.allows(from_service, to_service); - + shared_network && has_capability && policy_allows } ``` +### Default Network + +The daemon automatically creates a **default network** at startup with the following characteristics: + +- **Name:** `"default"` +- **Access:** `NetworkAccess::External` - allows external internet access +- **Purpose:** Provides a convenient default for standalone services that need external connectivity + +**Secure by Default:** + +Services that don't explicitly join any network are **internal-only** (no external access). This secure-by-default approach requires explicit opt-in to external connectivity: + +```bash +# Internal-only (no external access) +fabricks service run ./my-service + +# With external access via default network +fabricks service run --network default ./my-service + +# Multiple networks (default + custom) +fabricks service run --network default --network application ./my-service +``` + +For mortar compositions, services must explicitly list networks in the `fabricks-mortar.toml` configuration. + --- ## Security Architecture diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 24302f9..bef6fd1 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -725,12 +725,22 @@ fabricks service run [OPTIONS] --env-file Read environment from file -v, --volume Mount volumes (can be used multiple times) --name Assign a name to the instance - --network Connect to network + --network Connect to network (can be specified multiple times) --rm Automatically remove when stopped -d, --detach Run in background --restart Restart policy [default: no] [possible: no, on-failure, always] ``` +**Network Access:** + +By default, services run through the daemon are **internal-only** (no external access) for security. To enable network access: + +- **For external access:** Use `--network default` to join the default network which has `NetworkAccess::External` +- **For internal service-to-service communication:** Specify custom network names with `--network ` +- **Multiple networks:** Specify `--network` multiple times to join multiple networks + +The **default network** is automatically created at daemon startup with external access enabled. + **How it Works:** 1. **Resolution Phase:** @@ -753,14 +763,20 @@ fabricks service run ./examples/nodejs-hello # Run by OCI tag fabricks service run nodejs-hello:1.0.0 -# Run with port mapping -fabricks service run -p 8080:8089 ./examples/nodejs-hello +# Run with port mapping and network access +fabricks service run -p 8080:8089 --network default ./examples/nodejs-hello # Run with environment variables -fabricks service run -e LOG_LEVEL=debug nodejs-hello:1.0.0 +fabricks service run -e LOG_LEVEL=debug --network default nodejs-hello:1.0.0 -# Run in background with auto-restart -fabricks service run -d --restart on-failure nodejs-hello:1.0.0 +# Run in background with auto-restart and external access +fabricks service run -d --restart on-failure --network default nodejs-hello:1.0.0 + +# Run on multiple networks (default + custom) +fabricks service run --network default --network application ./examples/nodejs-hello + +# Run internal-only (no external access - secure by default) +fabricks service run ./examples/internal-worker ``` **Output:** @@ -916,34 +932,65 @@ fabricks service logs [OPTIONS] **Options:** ``` - --follow Follow log output - --tail Number of lines to show from end [default: all] - --since