From b0844f9248c2cc517e0a0ed62bebfa795d0fc080 Mon Sep 17 00:00:00 2001 From: Bora Oztekin Date: Wed, 19 Aug 2026 23:15:22 +0000 Subject: [PATCH 1/2] docs(function-autoscaler): add function autoscaler docs for self-hosted Document deployment, metrics dependencies, health checks, and troubleshooting for the self-hosted Function Autoscaler. Update observability and Helmfile guidance for bundled VictoriaMetrics and existing metrics backends. Signed-off-by: Bora Oztekin --- docs/user/autoscaling/architecture.md | 64 ++- docs/user/autoscaling/index.md | 40 +- docs/user/autoscaling/observability.md | 10 +- docs/user/autoscaling/operations.md | 90 ++-- docs/user/helmfile-installation.md | 39 +- docs/user/observability.md | 565 ++++++++++--------------- 6 files changed, 379 insertions(+), 429 deletions(-) diff --git a/docs/user/autoscaling/architecture.md b/docs/user/autoscaling/architecture.md index 532aa36ba..96f221d5f 100644 --- a/docs/user/autoscaling/architecture.md +++ b/docs/user/autoscaling/architecture.md @@ -1,27 +1,35 @@ # Function Autoscaler Architecture -The function autoscaler is a Rust service deployed as a horizontally scaled Kubernetes Deployment. It reads utilization and request metrics from a Prometheus-compatible timeseries database, stores discovered functions and coordination state in Cassandra, and writes desired instance counts to the NVCF API. Cassandra lightweight transactions handle leader election and short-lived per-function locks. +The Function Autoscaler runs as a Kubernetes Deployment in the control-plane +cluster. It reads metrics from a PromQL-compatible backend, stores coordination +state in Cassandra, and writes desired instance counts to the NVCF API. -The work is split into two loops. A leader-elected discovery loop scans the timeseries database for active function versions and upserts them into Cassandra. A scaling loop runs on every replica, but each replica only handles the functions whose IDs hash into its assigned buckets, so the active set is sharded across replicas. +The work is split into two loops. A leader-elected discovery loop scans the +timeseries database for active function versions and upserts them into +Cassandra. A scaling loop runs on every replica, but each replica only handles +the functions whose IDs hash into its assigned buckets, so the active set is +sharded across replicas. ## Sequence Diagram ```mermaid sequenceDiagram - participant Workers as Workers / Invocation Services - participant TSDB as Time Series DB + participant Services as NVCF metrics endpoints + participant Collector as OpenTelemetry Collector + participant TSDB as Metrics backend participant Autoscaler as Function Autoscaler participant Cassandra as Cassandra participant NVCF as NVCF Service - Workers->>TSDB: Emit utilization and instance metrics + Collector->>Services: Scrape selected metrics + Collector->>TSDB: Remote write - Note over Autoscaler,Cassandra: Discovery loop (~15s, leader-elected) + Note over Autoscaler,Cassandra: Periodic discovery loop, leader-elected Autoscaler->>TSDB: Query active functions TSDB-->>Autoscaler: Function set Autoscaler->>Cassandra: Upsert newly discovered functions - Note over Autoscaler,NVCF: Scaling loop (~30s, per-bucket) + Note over Autoscaler,NVCF: Periodic scaling loop, per-bucket Autoscaler->>Cassandra: Read active functions for this node's buckets Autoscaler->>TSDB: Query current instances and utilization history TSDB-->>Autoscaler: Metrics @@ -32,22 +40,43 @@ sequenceDiagram Autoscaler->>Cassandra: Write predicted count, refresh function TTL ``` -The discovery loop runs on one leader-elected replica. The scaling loop runs on every replica, but each replica only processes the function buckets assigned to it. +The discovery loop runs on one leader-elected replica. The scaling loop runs on +every replica, but each replica only processes its assigned function buckets. -## Timeseries Database +## Deployment order -The function autoscaler is a read-only client of a Prometheus-compatible timeseries store. It calls the `/api/v1/query_range` HTTP endpoint and uses PromQL for every metric query, so any backend that implements that interface works: upstream Prometheus, Thanos, Grafana Mimir, or VictoriaMetrics. The reference NVCF deployments point at VictoriaMetrics via the `timeseries_db_url` setting. +With the default `control` profile, the observability stage installs the shared +metrics components and backend. The final control-plane stage installs State +Metrics, then the Function Autoscaler. The autoscaler depends on State Metrics. -The function autoscaler does not run a scrape config of its own and does not write samples. Before it can do anything useful, the rest of the data plane has to be feeding the same store: +The shared metrics stage is skipped for `disabled`. The Function Autoscaler is +installed only for `control` and `all`. -- Worker pods export utilization and instance count metrics (`nvcf_worker_service_worker_thread_busy_seconds_total`, `nvcf_worker_service_worker_thread_count_total`, instance gauges). -- Invocation services and the gRPC proxy export request counters (`function_request`, `function_request_total`) labeled by `function_id`, `function_version_id`, and `nca_id`. These labels are how the discovery loop finds active function versions. +## Metrics backend -For a self-hosted control plane, you need three things in place before bringing the function autoscaler online: +The autoscaler is a read-only client of a PromQL-compatible backend. It uses +range queries to discover active functions and read instance, request, and +utilization metrics. -1. A Prometheus-compatible store reachable from the function autoscaler pod. -2. A scrape configuration (or remote-write feed) covering the worker pods and the invocation-plane services. -3. The resulting query endpoint passed in as `timeseries_db_url`. The function autoscaler reports `not ready` on its readiness probe until that endpoint responds. +The autoscaler does not scrape metrics. The metrics for the selected invocation +path must reach the backend that it queries. These include: + +- State Metrics instance, concurrency, request latency, and function metadata. +- Invocation Service and gRPC Proxy request counters used to discover active + function versions. +- Worker thread count and busy-time metrics used for worker-based utilization. +- LLM API Gateway request duration metrics used for LLM functions. + +For a split deployment, any worker metrics used for scaling must reach the +backend that the autoscaler queries. See +[Cluster Monitoring](../cluster-management/monitoring.md) for compute-plane +metrics endpoints. + +The backend can be bundled VictoriaMetrics or an existing PromQL-compatible +service. See [Observability Configuration](../observability.md) for backend, +endpoint, and authentication settings. + +The autoscaler reports `not ready` until the query endpoint responds. ## Coordination and Self-Healing @@ -62,3 +91,4 @@ Coordination relies on Cassandra TTLs to recover from failures without operator - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API. - [Function Autoscaler Operations](./operations.md) for health endpoints and common issues. - [Function Autoscaler Observability](./observability.md) for emitted metrics, traces, and logs. +- [Observability Configuration](../observability.md) for profiles and metrics backend settings. diff --git a/docs/user/autoscaling/index.md b/docs/user/autoscaling/index.md index 525d7bf7b..5e544e35c 100644 --- a/docs/user/autoscaling/index.md +++ b/docs/user/autoscaling/index.md @@ -1,33 +1,50 @@ # Function Autoscaling -The NVCF Function Autoscaler is a distributed Rust service that monitors function utilization and uses it to determine the ideal instance count per function on the NVCF control plane. It runs as a horizontally scaled deployment on the same Kubernetes cluster as the rest of the control plane. - -On an interval, the function autoscaler reads metrics from the timeseries database, decides how many instances each function should have, and calls the NVCF API to apply that decision. - -The function autoscaler depends on a Prometheus-compatible timeseries database fed by the worker pods and invocation-plane services. Without it, the service reports `not ready` and makes no scaling decisions. See [Timeseries database](./architecture.md#timeseries-database) for the required metrics and endpoints. +The NVCF Function Autoscaler reads function metrics, calculates a desired +instance count, and sends that count to the NVCF API. It runs in the +self-hosted control-plane cluster. ## Function Autoscaler vs Horizontal Pod Autoscaler -Function autoscaling is distinct from Kubernetes horizontal pod autoscaling (HPA). HPA scales pods within a single cluster, so it cannot reach NVCF worker pods that are spread across multiple clusters. Function autoscaling orchestrates scaling across clusters using global load patterns. +Function autoscaling is distinct from Kubernetes horizontal pod autoscaling +(HPA). HPA scales a Kubernetes workload in one cluster. The Function +Autoscaler sets the desired instance count for an NVCF function version, which +can run across NVCF compute clusters. ## Key Functionality -- Discovers active functions from invocation and worker metrics in the timeseries database and persists the active set in Cassandra. -- Periodically computes a desired instance count per function from recent utilization and the function's scaling policy. +- Discovers active functions from invocation and worker metrics in the + timeseries database and persists the active set in Cassandra. +- Periodically computes a desired instance count per function from recent + utilization and the function's scaling policy. - Applies the desired count by calling the NVCF API's predictions endpoint. -- Coordinates work across replicas using hash-based bucket assignment and Cassandra Lightweight Transaction (LWT) distributed locks. +- Coordinates work across replicas using hash-based bucket assignment and + Cassandra lightweight transaction (LWT) locks. + +## Self-hosted deployment + +The self-managed control-plane stack defaults to the `control` observability +profile. The `control` and `all` profiles install the Function Autoscaler. The +`compute` and `disabled` profiles do not. + +State Metrics must be enabled for `control` and `all`. With the default +component modes, the control-plane stack also installs the shared collector and +VictoriaMetrics. See [Observability Configuration](../observability.md) for +profile and backend settings. ## Architecture Overview ```mermaid flowchart LR - Workers[Workers / Invocation Services] --> TSDB[(Time Series DB)] + Services[Metrics endpoints] --> Collector[OpenTelemetry Collector] + Collector --> TSDB[(VictoriaMetrics or external backend)] TSDB --> Autoscaler[Function Autoscaler] Autoscaler <--> Cassandra[(Cassandra)] Autoscaler --> NVCF[NVCF API] ``` -See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequence diagram and the bucket model. +See [Architecture](./architecture.md#sequence-diagram) for the end-to-end +sequence and bucket model. ## See Also @@ -35,3 +52,4 @@ See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequen - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API. - [Function Autoscaler Operations](./operations.md) for health endpoints and operational guidance. - [Function Autoscaler Observability](./observability.md) for the metrics, traces, and logs emitted by the service. +- [Observability Configuration](../observability.md) for profiles and metrics backend configuration. diff --git a/docs/user/autoscaling/observability.md b/docs/user/autoscaling/observability.md index 131a30129..eea90a583 100644 --- a/docs/user/autoscaling/observability.md +++ b/docs/user/autoscaling/observability.md @@ -1,8 +1,14 @@ # Function Autoscaler Observability -The function autoscaler emits structured logs, Prometheus metrics that explain dependency health statuses and scaling decisions, and OpenTelemetry spans for outbound calls to its dependencies. The Prometheus exporter serves metrics on the address configured in `server.metrics.exporters`. The local settings file at `crates/server/resources/settings-local.yaml` uses `0.0.0.0:41338`. +The Function Autoscaler emits structured logs, Prometheus metrics, and +OpenTelemetry spans. The chart exposes its Prometheus exporter through the +`function-autoscaler` service on the `metrics` port, which defaults to `41338`. +The shared stack's default monitors do not include this service. Add a monitor +or scrape target for it to collect these metrics. -Job and namespace labels follow the standard NVCF naming convention for the cluster that runs the function autoscaler. +These service metrics describe the autoscaler itself. They are separate from +the function metrics that the autoscaler reads from VictoriaMetrics or an +external backend. ## Metric reference diff --git a/docs/user/autoscaling/operations.md b/docs/user/autoscaling/operations.md index 08e74d5b5..e18c1e72c 100644 --- a/docs/user/autoscaling/operations.md +++ b/docs/user/autoscaling/operations.md @@ -1,65 +1,73 @@ # Function Autoscaler Operations -This page covers operating the function autoscaler after deployment, including health probes, common operational issues, and pointers to the Helm chart values. For log filter syntax, metrics, and traces, see [Function Autoscaler Observability](./observability.md). +The self-managed stack deploys the Function Autoscaler for the `control` and +`all` observability profiles. State Metrics must remain enabled for both. See +[Observability Configuration](../observability.md) for profile and metrics +backend settings. -## Health endpoints - -The function autoscaler exposes three HTTP health endpoints. Their exact paths differ from the rest of the NVCF control plane: liveness and readiness are namespaced under `/admin/health/`. - -| Endpoint | Purpose | Use as | -|----------|---------|--------| -| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. | -| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. | -| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. | - -The liveness probe deliberately does not check Cassandra or the timeseries database. Restarting the pod when those are unreachable does not help, so the function autoscaler stays running and lets readiness flip instead. +Apply an environment change from the self-managed stack directory: -## Common operational issues +```bash +make apply HELMFILE_ENV= +``` -### Cassandra connection failures +## Verify the deployment -Symptoms: readiness flips to 503, `/health` reports the `cassandra_client` component as unhealthy, log lines from `rs_autoscaler::cassandra` show connection errors. +Check State Metrics and the Function Autoscaler: -Checks: +```bash +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=state-metrics +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=function-autoscaler +kubectl rollout status deployment/function-autoscaler -n nvcf +``` -- SSL certificates are mounted at the path expected by `cassandra.ssl`. The function autoscaler container expects the cert directory to exist; create `/etc/app/config` if it is missing. -- Credentials in the secrets file are valid for the configured keyspace. -- The contact points resolve from the pod's network namespace. +Confirm the resolved PromQL endpoint. This ConfigMap does not contain the +backend credentials: -### Timeseries database query failures +```bash +kubectl get configmap -n nvcf function-autoscaler-env \ + -o jsonpath='{.data.TIMESERIES_DB__TIMESERIES_DB_URL}{"\n"}' +``` -Symptoms: `nvcf_autoscaler.timeseries_db.requests_total` shows a rising error count, `auth_failure_total` or `server_side_failure_total` is non-zero, log lines from `rs_autoscaler::timeseries_db` show 4xx or 5xx responses. +For the bundled backend, the result should point to `vmsingle` in the +configured monitoring namespace. For an existing backend, it should match +`metricsBackend.promqlEndpoint`. -Checks: - -- `timeseries_db.timeseries_db_url` is reachable from the pod. -- The bearer token in the secrets file is current. Token rotation is the most common cause of `auth_failure_total` spikes. -- Query time ranges fit the retention window of the backing store. - -### NVCF API errors - -Symptoms: `nvcf_autoscaler.nvcf_api.request_duration_milliseconds` shows a sustained rise in 4xx or 5xx, scaling decisions stop applying. +## Health endpoints -Checks: +The Function Autoscaler exposes three health endpoints: -- The OAuth2 token endpoint is reachable and the client credentials in the secrets file are valid. -- The functions being scaled are still in a deployable status. Functions in unexpected states are skipped, not retried. -- `nvcf_api.disable_auth` is set as intended for the deployment. Leave it `false` whenever the NVCF API enforces authentication. +| Endpoint | Purpose | Use as | +| --- | --- | --- | +| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. | +| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. | +| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. | -### Discovery is stalled +Inspect the detailed endpoint through the service: -Symptoms: the active function set in Cassandra stops growing despite traffic to new functions, `nvcf_autoscaler.distributed_lock.acquisition_failures_total` is rising across all replicas. +```bash +kubectl port-forward -n nvcf service/function-autoscaler 8181:8181 +curl http://127.0.0.1:8181/health +``` -Checks: +The liveness probe does not check Cassandra or the metrics backend. Dependency +failures change readiness instead. -- Inspect the `locks` table for the discovery lock row and its TTL. If the row never expires, the previous leader may have stopped refreshing without releasing it. -- Confirm at least one replica's `nvcf_autoscaler.distributed_lock` gauge reports the leader state. -- Restart the holding replica if the cluster is otherwise healthy. The lock expires within `discovery_lock_duration_seconds`. +## Troubleshooting -See [Architecture](./architecture.md#cassandra-lightweight-transactions-lwts) for the lock state machine. +| Symptom | Check | +| --- | --- | +| Function Autoscaler is not installed | Use the `control` or `all` profile. Keep `stateMetrics.enabled: true`. | +| `cassandra_client` is unhealthy | Check contact-point DNS, credentials, and the configured TLS files. | +| `timeseries_db_client` is unhealthy | Check the resolved PromQL endpoint, authentication mode, credentials, and backend retention. | +| Scaling decisions are not applied | Check NVCF API authentication and function status. | +| Discovery does not find active functions | Confirm the backend contains the request and worker metrics listed in [Architecture](./architecture.md#metrics-backend). Check the discovery lock metrics and TTL. | ## See also - [Function Autoscaler Observability](./observability.md) for the metrics and traces referenced in the symptoms above. - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds and policy via the NVCF API. - [Architecture](./architecture.md) for the component layout these symptoms map to. +- [Observability Configuration](../observability.md) for shared stack settings. diff --git a/docs/user/helmfile-installation.md b/docs/user/helmfile-installation.md index 0a37bb874..ad1ce0cc9 100644 --- a/docs/user/helmfile-installation.md +++ b/docs/user/helmfile-installation.md @@ -33,14 +33,15 @@ ls ## Namespace Requirements -Each control-plane Helm chart must be installed into a specific namespace. These -namespace assignments are fixed and must not be changed because -service-to-service cluster DNS addressing and Vault (OpenBao) authentication -claims depend on this layout. +Each control-plane Helm chart must be installed into a specific namespace. The +control-plane namespace assignments are fixed because service-to-service DNS +addressing and Vault (OpenBao) authentication claims depend on them. The +observability stack uses `monitoring` by default, but its namespace is +configurable. | Namespace | Services | | --- | --- | -| `nvcf` | api, invocation-service, grpc-proxy, notary-service, reval, state-metrics | +| `nvcf` | api, invocation-service, grpc-proxy, notary-service, reval, state-metrics, function-autoscaler | | `api-keys` | api-keys, admin-issuer-proxy | | `ess` | ess-api | | `sis` | sis | @@ -48,12 +49,14 @@ claims depend on this layout. | `cassandra-system` | cassandra | | `nats-system` | nats | | `cert-manager` | cert-manager | +| `monitoring` (default) | OpenTelemetry Operator, collector, default monitors, VictoriaMetrics | | `envoy-gateway-system` | ingress (nvcf-gateway-routes) | Installing a chart into the wrong namespace will cause authentication failures such as `error validating claims: claim "/kubernetes.io/namespace" does not match any associated bound claim values`. -If you see this error, verify that every release is deployed in the namespace shown above. +If you see this error, verify that each control-plane release uses the required +namespace and each observability release uses its configured namespace. @@ -280,6 +283,18 @@ global: # collectorPort: # collectorProtocol: +# Install control-plane monitors, the bundled metrics backend, and the +# Function Autoscaler. +observability: + profile: control + +victoriaMetrics: + server: + persistentVolume: + enabled: true + size: 16Gi + storageClass: "gp3" # Customize to your storage class. + fakeGpuOperator: enabled: false # If deploying locally with no GPUs, true ubuntu: @@ -339,6 +354,18 @@ request-router host and port that worker pods can reach. See [LLM Function Enablement](./llm-function-enablement.md) for the complete addon configuration. +#### `observability` Configuration + +The self-managed control-plane stack defaults to +`observability.profile: control`. This installs the shared metrics components, +VictoriaMetrics, State Metrics, and the Function Autoscaler. Set the +VictoriaMetrics storage class for the target cluster. + +To use a customer-managed backend or change component ownership, see +[Observability Configuration](./observability.md). For autoscaler health and +backend checks, see +[Function Autoscaler Operations](./autoscaling/operations.md). + #### `domain` and `ingress` Configuration The `domain` and `ingress` sections of the environment file are used to configure the external access to the NVCF control plane. diff --git a/docs/user/observability.md b/docs/user/observability.md index 0a6520801..397222e0d 100644 --- a/docs/user/observability.md +++ b/docs/user/observability.md @@ -1,414 +1,275 @@ # Observability Configuration -This page provides guidance on configuring observability for self-hosted NVCF control-plane, including metrics, logging, and tracing. +The self-managed stack can collect NVCF metrics and write them to a bundled or +customer-managed backend. Logs and traces use separate configuration. -## Find the answer to your question +## Observability profiles -Common operator questions and where to look on this page or in linked references. +Set one profile in the Helmfile environment: -| Question | Where to look | -|----------|---------------| -| How do I see application-level NVCF stats (number of functions, queue depth, request latency)? | [State Metrics Service metrics](./metrics/state-metrics/metrics.md). The page documents per-function instance count, queue depth, and request latency, plus other function-level signals. | -| How do I debug a single request end-to-end? | Combine the per-hop signals: enable tracing per [Tracing Configuration](#tracing-configuration), correlate with the [Metrics Overview](./metrics/metrics-index.md) for each service in the request path, and tail the matching service logs. A consolidated hop-by-hop walkthrough is in development. | -| Where are per-service metrics? | [Metrics Overview](./metrics/metrics-index.md). | -| Where are gRPC proxy metrics? | [gRPC Proxy metrics](./metrics/grpc-proxy/metrics.md). The page documents client connection counts, NATS pipe health, gRPC worker session-attach latency, and HTTP RED metrics. | -| How do I add custom spans or metrics in a Kit application? | Use the OpenTelemetry API directly, the OmniTrace helper, the Carbonite static metrics API, or the `omni::observability::IMeter` interface. Refer to the Omniverse Kit and Carbonite documentation for details. | -| Where are reference dashboards? | [Example dashboards](./example-dashboards.md) and the [Dashboards](#dashboards) section below. | - -## Overview - -Self-hosted NVCF control-plane observability enables users to monitor the health and performance of their NVCF deployment. The observability solution is designed to be: - -- **Cloud-agnostic**: Works in any Kubernetes environment (cloud provider, on-premises, or air-gapped) -- **Offline-capable**: Fully functional in isolated networks without external dependencies -- **Bring-Your-Own (BYO)**: Integrates with your existing observability platforms -- **No vendor lock-in**: Uses open standards (Prometheus, OpenTelemetry, OTLP) - -The observability solution currently provides: - -- [Metrics Collection]: Prometheus-compatible metrics from all control-plane services -- [Logging]: Logs emitted to stdout/stderr for easy collection -- [Tracing]: Distributed tracing via OTLP to your collector -- [Dashboards]: Reference Grafana dashboards for key metrics - - -**Looking for a quick start?** If you want to quickly deploy example observability components -to explore metrics, logs, and dashboards, see [self-hosted-example-dashboards](./example-dashboards.md). - -The example deployments are designed for development and testing only, and are not suitable -for production use. For production deployments, follow the guidance on this page to integrate -with your own observability infrastructure. - - - -## Early Access Phase - -NVCF self-hosted observability is currently in Early Access (EA). During EA, NVCF provides interfaces and documentation for you to integrate with your own observability backend: - -**What's Provided:** - -- Documented metrics for critical control-plane services -- Example scrape targets for prometheus-operator ServiceMonitor configuration -- Metrics exposed via Prometheus-compatible endpoints -- Logs emitted to stdout/stderr for easy collection -- Configuration and deployment documentation -- Example dashboards for key metrics - -**Your Responsibility:** - -- Deploy and manage your own observability backend (Prometheus, Grafana, Loki, Elasticsearch, etc.) -- Configure metrics scraping from control-plane services -- Deploy log collectors (e.g., Fluentd, Promtail, OTel Collector) to aggregate logs -- Set up your preferred visualization and alerting tools - -## Control-Plane Services - -The following control-plane services expose metrics and logs for monitoring: - -**Core NVCF Services:** - -- **NVCF API**: Main API for function management and invocation -- **Invocation Service**: Handles function invocation requests -- **SPOT Instance Service (SIS)**: Manages worker pod and cluster state -- **State Metrics Service**: Aggregates and exports NVCF-specific metrics - -**Supporting Services:** - -- **Cassandra (C\*)**: Primary database for control-plane state -- **OpenBao/Vault**: Secret management and S2S authentication -- **Encrypted Secrets Service (ESS)**: Function and account secrets -- **NATS Core**: Pub/sub messaging -- **NATS JetStream**: Persistent messaging - -**Worker Pod Components:** - -- **Utils Container**: Proxy to NATS from user applications -- **Init Container**: Setup and resource loading -- **Inference Container**: Inference workload - -## Architecture - -### Metrics Collection - -All control-plane services expose Prometheus-compatible metrics endpoints. You can scrape these metrics using: +```yaml +observability: + profile: control +``` -- **Prometheus Operator**: Create ServiceMonitor resources based on the provided scrape targets -- **Prometheus**: Configure scrape targets manually -- **OpenTelemetry Collector**: Use the Prometheus receiver +The control-plane and compute-plane stacks use the same profile names for +different parts of observability. A split deployment normally uses `control` +on the control-plane cluster and `compute` on each compute cluster. + +| Profile | Shared stack monitor defaults | Function Autoscaler in control plane | NVCA observability defaults in compute plane | +| --- | --- | --- | --- | +| `disabled` | None | Not installed | Disabled | +| `control` | Control-plane services | Installed | Disabled | +| `compute` | NVCA, DCGM, and worker pods | Not installed | Enabled | +| `all` | Control-plane and compute-plane targets | Installed | Enabled | + +The `all` profile is intended for a cluster that contains both control-plane +and compute-plane targets. + +The self-managed control-plane stack defaults to `control` and delegates an +enabled profile to the shared observability stack. The compute-plane stack +defaults to `compute`. It enables the NVCA OpenTelemetry Collector sidecar and +the `BYOObservability` feature gate, but does not install the shared +observability stack or VictoriaMetrics. + +When the shared observability stack runs, an enabled profile installs these +components by default: + +- Prometheus Operator custom resource definitions for `ServiceMonitor` and + `PodMonitor`. +- OpenTelemetry Operator. +- OpenTelemetry Collector with Target Allocator and discovery role-based access + control (RBAC). +- VictoriaMetrics. +- Default NVCF monitor resources. + +The self-managed control-plane stack installs State Metrics when +`stateMetrics.enabled` is `true`. For `control` and `all`, it also installs the +Function Autoscaler and requires State Metrics. + +`global.observability.metrics.enabled` is not the profile selector. Some +service charts still use it to enable their own metric exports or PodMonitors. +Set it separately when those service metrics are needed. + +## Shared metrics flow + +```mermaid +flowchart LR + Targets["NVCF metrics endpoints"] --> Monitors["ServiceMonitor and PodMonitor"] + Monitors --> Collector["OpenTelemetry Collector"] + Collector --> Backend["VictoriaMetrics or external backend"] + Backend --> Autoscaler["Function Autoscaler"] + Backend --> Queries["PromQL queries and dashboards"] +``` -**Metrics Documentation:** +The Target Allocator discovers monitors labeled +`nvcf.nvidia.com/observability-target: "true"`. The collector scrapes the +selected endpoints and sends samples through Prometheus remote write. -Detailed metrics documentation is available for each service, including metric names, -types, labels, and descriptions. See the per-service metrics reference under the -`Metrics` section. +Within the shared stack, the default control-plane monitors select State +Metrics, Invocation Service, gRPC Proxy, and LLM API Gateway. The default +compute-plane monitors select NVCA, DCGM, and NVCA-managed worker pods. -### Logging +## Bundled VictoriaMetrics -**Log Format:** +The default backend is a single VictoriaMetrics instance in the `monitoring` +namespace. The stack derives these endpoints: -- All services emit logs to stdout/stderr (standard for Kubernetes) -- Sensitive data redaction must be configured by the log collector +```text +remote write: http://vmsingle.monitoring.svc.cluster.local:8428/api/v1/write +PromQL: http://vmsingle.monitoring.svc.cluster.local:8428 +``` -**Log Collection:** +The default storage settings are: -You can collect logs using any Kubernetes-compatible log aggregator: +```yaml +victoriaMetrics: + server: + retentionPeriod: "1" + persistentVolume: + enabled: true + size: 16Gi + storageClass: "" +``` -- Fluentd or Fluent Bit -- Promtail (for Loki) -- Filebeat (for Elasticsearch) -- OpenTelemetry Collector (filelog receiver) +Set `storageClass` for the target cluster. If you change +`observability.namespace` or `victoriaMetrics.namespace`, the stack derives the +service addresses from that namespace. -**System Logs:** +The bundled VictoriaMetrics service has a cluster-local endpoint and uses no +application-level authentication. Keep it cluster-local unless you add network +and access controls. -System logs are available at standard UNIX locations and from the systemd journal. +## Existing metrics backend -### Tracing (Available in GA) +Use `metricsBackend.mode: existing` when the stack should write to and query a +customer-managed backend: -Distributed tracing support via OpenTelemetry Protocol (OTLP) is planned for a future release: +```yaml +observability: + profile: control + +metricsBackend: + mode: existing + type: external + remoteWriteEndpoint: https://metrics.example.com/write + promqlEndpoint: https://metrics.example.com + authentication: + mode: none +``` -- Key flows will be instrumented with OpenTelemetry SDK -- Traces will be exportable via OTLP (HTTP or gRPC) -- Configurable sampling strategies -- Support for any OTLP-compatible backend (Jaeger, Tempo, Zipkin, etc.) -- Tracing is configurable via Helm values under `global.observability.tracing` +`remoteWriteEndpoint` is required for an existing backend. `promqlEndpoint` is +also required for `control` and `all` because the Function Autoscaler queries +it. -## Configuration +For the Function Autoscaler's PromQL client, authentication modes are `none`, +`token`, and `mtls`. Token authentication requires `authnEndpoint`. mTLS +requires `clientCertificatePath` and `clientPrivateKeyPath`. -You configure observability by integrating with your own backend: +These settings apply only to the Function Autoscaler's PromQL client. They do +not configure collector remote-write authentication or mount credentials and +certificates. Configure remote-write authentication separately under +`collector.config.exporters.prometheusremotewrite`. -### Metrics Scraping +## Component ownership -Metrics export is opt-in and disabled by default. Enable it in your Helmfile -environment before configuring scrape targets: +Profiles set defaults. Override a component only when another deployment owns +it: -```yaml -global: - observability: - metrics: - enabled: true -``` +| Mode | Meaning | +| --- | --- | +| `install` | The NVCF observability stack installs the component. | +| `existing` | The component is managed outside this stack. | +| `disabled` | The component is not used. | -Use Prometheus Operator with the provided ServiceMonitor examples: +For example, keep customer-managed Prometheus Operator CRDs, OpenTelemetry +Operator, and metrics backend: ```yaml -# Example ServiceMonitor for NVCF API -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: nvcf-api - namespace: nvcf -spec: - selector: - matchLabels: - app: nvcf-api - endpoints: - # Endpoint created based on the scrape target in the - # per-service metrics documentation - - port: metrics - interval: 30s - path: /metrics +observability: + profile: control + components: + prometheusOperatorCrds: + mode: existing + otelOperator: + mode: existing + +metricsBackend: + mode: existing + type: external + remoteWriteEndpoint: https://metrics.example.com/write + promqlEndpoint: https://metrics.example.com ``` -Or configure Prometheus scrape targets manually in your prometheus.yml. +The configurable components are: -#### Application-level NVCF stats +- `observability.components.prometheusOperatorCrds` +- `observability.components.otelOperator` +- `observability.components.collector` +- `observability.components.targetAllocator` +- `observability.components.discoveryRbac` +- `metricsBackend` -The State Metrics Service exposes per-function signals you can query in -Prometheus. The following PromQL examples cover the three most common -operator questions. Metric names and labels are sourced from -[State Metrics Service metrics](./metrics/state-metrics/metrics.md). +Helmfile rejects combinations that leave an installed component without a +required dependency. -Number of registered functions: +## Monitor overrides -```promql -# nvcf_function_info is emitted per function with descriptive labels. -# Dedupe by function_id so multiple label series do not inflate the count. -count(count by (function_id) (nvcf_function_info)) -``` - -Queue depth per function: - -```promql -# nvcf_function_queue_depth is a gauge keyed by function_id. -sum by (function_id, name) (nvcf_function_queue_depth) -``` - -Function request latency (p50 and p95) over a 5 minute window: - -```promql -# p50 -histogram_quantile( - 0.50, - sum by (le, function_id) (rate(function_request_latency_bucket[5m])) -) +Profiles select monitor groups, but each group and target can be overridden: -# p95 -histogram_quantile( - 0.95, - sum by (le, function_id) (rate(function_request_latency_bucket[5m])) -) +```yaml +observability: + profile: control + +defaultMonitors: + controlPlane: + enabled: true + computePlane: + enabled: false + worker: + enabled: false ``` -### Log Collection - -Deploy a log collector as a DaemonSet to ship logs to your backend: - -```bash -# Example: Deploy Promtail for Loki -kubectl apply -f promtail-daemonset.yaml +The shared collector discovers targets only in its Kubernetes cluster. The +compute-plane stack does not install this shared collector. For a split +deployment, configure compute-plane collection separately and make any worker +metrics used for autoscaling available to the control-plane backend. See +[Cluster Monitoring](./cluster-management/monitoring.md). -# Example: Deploy Fluentd or Fluent Bit -kubectl apply -f fluentd-daemonset.yaml -``` +## Dashboards -Configure your log collector to: +The shared stack does not install a dashboard UI. Query the bundled or external +backend with a PromQL-compatible tool. The +[Example Dashboards](./example-dashboards.md) guide deploys a separate +development reference stack with its own metrics components. Review component +ownership before using both stacks in one cluster. -- Tail logs from all namespaces -- Add metadata labels (pod name, namespace, service) -- Forward to your log aggregation backend (Loki, Elasticsearch, etc.) +## Logs -### Tracing Configuration +NVCF services write logs to standard output and standard error. The metrics +profile does not install a log backend. Use a Kubernetes log collector such as +Fluent Bit, Fluentd, Promtail, or an OpenTelemetry Collector configured for +logs. -Enable distributed tracing by setting Helm values under -`global.observability.tracing`. The control-plane exports traces via OTLP -to your own OTLP-compatible collector. Set `collectorEndpoint`, -`collectorPort`, and `collectorProtocol` to match your collector's address. -`collectorProtocol` is the endpoint URI scheme expected by the stack, not the -OTLP transport. +## Tracing configuration -Helm overrides example: +Control-plane services can export traces to an OpenTelemetry Protocol (OTLP) +endpoint configured under `global.observability.tracing`: ```yaml global: observability: tracing: enabled: true - collectorEndpoint: "otel-collector-gateway-collector.observability.svc.cluster.local" + collectorEndpoint: otel-collector.monitoring.svc.cluster.local collectorPort: 4317 collectorProtocol: http ``` -Configuration fields: +`collectorProtocol` supplies the URI scheme used by the stack. It does not +select the OTLP transport. -- `enabled`: Set to `true` to enable OTLP trace export from control-plane - services. -- `collectorEndpoint`: DNS name or address of your OTLP collector (e.g., - OpenTelemetry Collector, Jaeger collector). Use a Kubernetes service DNS name - such as `..svc.cluster.local` when the collector runs - in-cluster. -- `collectorPort`: Port on which the collector accepts OTLP traffic (e.g., - 4317 for gRPC, 4318 for HTTP depending on your collector setup). -- `collectorProtocol`: URI scheme used to build the collector endpoint - (`http` or `https`). This value does not select the OTLP transport. +## Verify -Ensure your collector is deployed and reachable from the NVCF control-plane -namespace, and that it forwards traces to your backend (Jaeger, Tempo, Zipkin, -or another OTLP-compatible system). +Check the shared components: -## Dashboards - -Reference Grafana dashboards are provided for control-plane services showing critical metrics for key services: - -- ESS (Encrypted Secrets Service) - -- Cassandra - -- Vault - -- Invocation Service - -- NVCF API - -- SIS (SPOT Instance Service) - -- Worker Pods (Utils Container, Init Container, Inference Container) - - - Note: Worker Pods are deployed in the backend cluster, not the control-plane - cluster, but their configuration is globally controlled as part of the control-plane - -- State Metrics Service +```bash +kubectl get pods,pvc -n monitoring +kubectl get opentelemetrycollector -A +kubectl get servicemonitor,podmonitor -A +``` -**Dashboard Location:** +Replace `monitoring` if the observability components use another namespace. -Dashboards are provided in native Grafana JSON format for [file-provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards). +For `control` and `all`, also check: -Load dashboards into Grafana by placing them in `/etc/grafana/provisioning/dashboards/` on startup. +```bash +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=state-metrics +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=function-autoscaler +``` -Published dashboards will be available in the -[NVCF examples](https://github.com/NVIDIA/nvcf/tree/main/examples) public GitHub repository. +See [Function Autoscaler Operations](./autoscaling/operations.md) for backend +and health checks. ## Troubleshooting -For troubleshooting common observability issues: - -**Metrics not appearing:** - -1. Verify the service is exposing metrics: - - ```bash - # Port-forward to the service metrics port - kubectl port-forward -n nvcf svc/nvcf-api 8080:8080 - - # In another terminal, curl the metrics endpoint - curl http://localhost:8080/metrics - ``` - -2. Check ServiceMonitor or scrape configuration: - - ```bash - # Verify ServiceMonitor exists - kubectl get servicemonitor -n nvcf - - # Check ServiceMonitor details - kubectl describe servicemonitor nvcf-api -n nvcf - ``` - -3. Verify network policies allow scraping: - - ```bash - # List network policies that might block traffic - kubectl get networkpolicy -n nvcf - - # Test connectivity from Prometheus namespace - kubectl run -n --rm -it debug \ - --image=curlimages/curl --restart=Never -- \ - curl http://nvcf-api.nvcf.svc.cluster.local:8080/metrics - ``` - -4. Check service logs for errors: - - ```bash - # Check for metrics-related errors - kubectl logs -n nvcf deployment/nvcf-api | grep -i metric - ``` - -**Logs not being collected:** - -1. Verify log collector DaemonSet is running: - - ```bash - # Check DaemonSet status (e.g., for Fluentd/Fluent Bit) - # Note: Namespaces may be different depending on the log collector deployment - kubectl get daemonset -n logging - kubectl get pods -n logging -l app=fluent-bit - ``` - -2. Check collector can access pod logs: - - ```bash - # Verify log collector has proper volume mounts - kubectl describe daemonset fluent-bit -n logging | grep -A5 Mounts - - # Check collector logs for errors - kubectl logs -n logging -l app=fluent-bit --tail=50 - ``` - -3. Verify log backend is reachable: - - ```bash - # Test connectivity to log backend (e.g. Loki) - kubectl run -n logging --rm -it debug \ - --image=curlimages/curl --restart=Never -- \ - curl -v http://loki.logging.svc.cluster.local:3100/ready - ``` - -4. Check for log redaction or filtering rules: - - ```bash - # Review collector configuration - kubectl get configmap fluent-bit-config -n logging -o yaml - - # Check if logs are being dropped - kubectl logs -n logging -l app=fluent-bit | grep -i "drop\|filter" - ``` - -## Security - -**Metrics Endpoints:** - -- Metrics endpoints should be accessed over HTTP in-cluster only - - - Any external access should be SSL/TLS or mTLS secured with a reverse proxy or other ingress controller, or - - Aggregated locally and exposed via a secured otel-collector - -- All sensitive log data should be redacted by the log collector (currently, this is the responsibility of the log collector, not the service) - - - Example implementation by OTEL Collector: [Log Redaction](https://opentelemetry.io/docs/languages/dotnet/logs/redaction/) - -- User-provided observability backend should be properly secured with RBAC, TLS/SSL, and other security best practices. - -## Related Documentation - -- [OpenTelemetry documentation](https://opentelemetry.io/docs/) -- [Prometheus documentation](https://prometheus.io/docs/) - -## Version Compatibility - -NVCF self-hosted control-plane observability is compatible with: - -- Supported versions are the latest Kubernetes minor release and the two prior minor releases (N-2). See official Kubernetes docs for current supported [versions](https://kubernetes.io/releases/version-skew-policy/#supported-versions). -- Any Prometheus-compatible metrics collection system -- Any log aggregation system that can collect from Kubernetes stdout/stderr or read - from the filesystem (depending on K8s cluster configuration) - -For the latest compatibility information, see the release notes. +| Symptom | Check | +| --- | --- | +| No observability releases | Confirm that `observability.profile` is not `disabled`. | +| VictoriaMetrics pod is pending | Check the persistent volume claim and configured storage class. | +| Metrics backend has no samples | Check the monitor labels, Target Allocator, collector logs, and remote-write endpoint. | +| Worker metrics are missing in a split deployment | Check compute-plane collection and connectivity to the backend queried by the autoscaler. | +| Function Autoscaler is not installed | Use the `control` or `all` profile and keep State Metrics enabled. | +| Function Autoscaler is not ready | Check Cassandra and the PromQL endpoint with the autoscaler health endpoint. | +| External backend authentication fails | Check the selected authentication mode and its required endpoints or certificate paths. | + +## Related documentation + +- [Metrics Overview](./metrics/metrics-index.md) +- [Function Autoscaling](./autoscaling/index.md) +- [Function Autoscaler Observability](./autoscaling/observability.md) +- [Cluster Monitoring](./cluster-management/monitoring.md) +- [Example Dashboards](./example-dashboards.md) +- [Control Plane Operations](./control-plane-operations.md) From fdd86fa6e86a8d2ee07dcb72e87a6aa82ad3881f Mon Sep 17 00:00:00 2001 From: Bora Oztekin Date: Thu, 20 Aug 2026 05:08:52 +0000 Subject: [PATCH 2/2] docs(observability): preserve existing configuration guidance Restore the existing observability page structure and examples. Add the self-managed metrics stack details in one focused section and update only statements affected by the new defaults. Signed-off-by: Bora Oztekin --- docs/user/observability.md | 586 ++++++++++++++++++++++++------------- 1 file changed, 389 insertions(+), 197 deletions(-) diff --git a/docs/user/observability.md b/docs/user/observability.md index 397222e0d..192b04fdc 100644 --- a/docs/user/observability.md +++ b/docs/user/observability.md @@ -1,114 +1,137 @@ # Observability Configuration -The self-managed stack can collect NVCF metrics and write them to a bundled or -customer-managed backend. Logs and traces use separate configuration. +This page provides guidance on configuring observability for self-hosted NVCF control-plane, including metrics, logging, and tracing. -## Observability profiles +## Find the answer to your question -Set one profile in the Helmfile environment: +Common operator questions and where to look on this page or in linked references. -```yaml -observability: - profile: control -``` +| Question | Where to look | +|----------|---------------| +| How do I see application-level NVCF stats (number of functions, queue depth, request latency)? | [State Metrics Service metrics](./metrics/state-metrics/metrics.md). The page documents per-function instance count, queue depth, and request latency, plus other function-level signals. | +| How do I debug a single request end-to-end? | Combine the per-hop signals: enable tracing per [Tracing Configuration](#tracing-configuration), correlate with the [Metrics Overview](./metrics/metrics-index.md) for each service in the request path, and tail the matching service logs. A consolidated hop-by-hop walkthrough is in development. | +| Where are per-service metrics? | [Metrics Overview](./metrics/metrics-index.md). | +| Where are gRPC proxy metrics? | [gRPC Proxy metrics](./metrics/grpc-proxy/metrics.md). The page documents client connection counts, NATS pipe health, gRPC worker session-attach latency, and HTTP RED metrics. | +| How do I add custom spans or metrics in a Kit application? | Use the OpenTelemetry API directly, the OmniTrace helper, the Carbonite static metrics API, or the `omni::observability::IMeter` interface. Refer to the Omniverse Kit and Carbonite documentation for details. | +| Where are reference dashboards? | [Example dashboards](./example-dashboards.md) and the [Dashboards](#dashboards) section below. | +| How do I configure the shared metrics stack? | See [Self-managed metrics stack](#self-managed-metrics-stack). | -The control-plane and compute-plane stacks use the same profile names for -different parts of observability. A split deployment normally uses `control` -on the control-plane cluster and `compute` on each compute cluster. +## Overview -| Profile | Shared stack monitor defaults | Function Autoscaler in control plane | NVCA observability defaults in compute plane | -| --- | --- | --- | --- | -| `disabled` | None | Not installed | Disabled | -| `control` | Control-plane services | Installed | Disabled | -| `compute` | NVCA, DCGM, and worker pods | Not installed | Enabled | -| `all` | Control-plane and compute-plane targets | Installed | Enabled | +Self-hosted NVCF control-plane observability enables users to monitor the health and performance of their NVCF deployment. The observability solution is designed to be: -The `all` profile is intended for a cluster that contains both control-plane -and compute-plane targets. - -The self-managed control-plane stack defaults to `control` and delegates an -enabled profile to the shared observability stack. The compute-plane stack -defaults to `compute`. It enables the NVCA OpenTelemetry Collector sidecar and -the `BYOObservability` feature gate, but does not install the shared -observability stack or VictoriaMetrics. - -When the shared observability stack runs, an enabled profile installs these -components by default: - -- Prometheus Operator custom resource definitions for `ServiceMonitor` and - `PodMonitor`. -- OpenTelemetry Operator. -- OpenTelemetry Collector with Target Allocator and discovery role-based access - control (RBAC). -- VictoriaMetrics. -- Default NVCF monitor resources. - -The self-managed control-plane stack installs State Metrics when -`stateMetrics.enabled` is `true`. For `control` and `all`, it also installs the -Function Autoscaler and requires State Metrics. - -`global.observability.metrics.enabled` is not the profile selector. Some -service charts still use it to enable their own metric exports or PodMonitors. -Set it separately when those service metrics are needed. - -## Shared metrics flow - -```mermaid -flowchart LR - Targets["NVCF metrics endpoints"] --> Monitors["ServiceMonitor and PodMonitor"] - Monitors --> Collector["OpenTelemetry Collector"] - Collector --> Backend["VictoriaMetrics or external backend"] - Backend --> Autoscaler["Function Autoscaler"] - Backend --> Queries["PromQL queries and dashboards"] -``` +- **Cloud-agnostic**: Works in any Kubernetes environment (cloud provider, on-premises, or air-gapped) +- **Offline-capable**: Fully functional in isolated networks without external dependencies +- **Bring-Your-Own (BYO)**: Integrates with your existing observability platforms +- **No vendor lock-in**: Uses open standards (Prometheus, OpenTelemetry, OTLP) -The Target Allocator discovers monitors labeled -`nvcf.nvidia.com/observability-target: "true"`. The collector scrapes the -selected endpoints and sends samples through Prometheus remote write. +The observability solution currently provides: -Within the shared stack, the default control-plane monitors select State -Metrics, Invocation Service, gRPC Proxy, and LLM API Gateway. The default -compute-plane monitors select NVCA, DCGM, and NVCA-managed worker pods. +- [Metrics Collection]: Prometheus-compatible metrics from all control-plane services +- [Logging]: Logs emitted to stdout/stderr for easy collection +- [Tracing]: Distributed tracing via OTLP to your collector +- [Dashboards]: Reference Grafana dashboards for key metrics -## Bundled VictoriaMetrics + +**Looking for a quick start?** If you want to quickly deploy example observability components +to explore metrics, logs, and dashboards, see [self-hosted-example-dashboards](./example-dashboards.md). -The default backend is a single VictoriaMetrics instance in the `monitoring` -namespace. The stack derives these endpoints: +The example deployments are designed for development and testing only, and are not suitable +for production use. For production deployments, use the self-managed metrics stack or +integrate with your own observability infrastructure. -```text -remote write: http://vmsingle.monitoring.svc.cluster.local:8428/api/v1/write -PromQL: http://vmsingle.monitoring.svc.cluster.local:8428 -``` + -The default storage settings are: +## Early Access Phase -```yaml -victoriaMetrics: - server: - retentionPeriod: "1" - persistentVolume: - enabled: true - size: 16Gi - storageClass: "" -``` +NVCF self-hosted observability is currently in Early Access (EA). During EA, NVCF provides interfaces and documentation for you to integrate with your own observability backend: -Set `storageClass` for the target cluster. If you change -`observability.namespace` or `victoriaMetrics.namespace`, the stack derives the -service addresses from that namespace. +**What's Provided:** -The bundled VictoriaMetrics service has a cluster-local endpoint and uses no -application-level authentication. Keep it cluster-local unless you add network -and access controls. +- Documented metrics for critical control-plane services +- Example scrape targets for prometheus-operator ServiceMonitor configuration +- Metrics exposed via Prometheus-compatible endpoints +- Shared metrics collection with bundled VictoriaMetrics or an existing backend +- Logs emitted to stdout/stderr for easy collection +- Configuration and deployment documentation +- Example dashboards for key metrics -## Existing metrics backend +**Your Responsibility:** -Use `metricsBackend.mode: existing` when the stack should write to and query a -customer-managed backend: +- Configure storage for bundled VictoriaMetrics or connect an existing backend +- Configure compute-plane collection for split deployments +- Deploy log collectors (e.g., Fluentd, Promtail, OTel Collector) to aggregate logs +- Set up your preferred visualization and alerting tools -```yaml -observability: - profile: control +## Control-Plane Services + +The following control-plane services expose metrics and logs for monitoring: + +**Core NVCF Services:** + +- **NVCF API**: Main API for function management and invocation +- **Invocation Service**: Handles function invocation requests +- **SPOT Instance Service (SIS)**: Manages worker pod and cluster state +- **State Metrics Service**: Aggregates and exports NVCF-specific metrics +- **Function Autoscaler**: Calculates desired function instance counts +**Supporting Services:** + +- **Cassandra (C\*)**: Primary database for control-plane state +- **OpenBao/Vault**: Secret management and S2S authentication +- **Encrypted Secrets Service (ESS)**: Function and account secrets +- **NATS Core**: Pub/sub messaging +- **NATS JetStream**: Persistent messaging + +**Worker Pod Components:** + +- **Utils Container**: Proxy to NATS from user applications +- **Init Container**: Setup and resource loading +- **Inference Container**: Inference workload + +## Architecture + +### Metrics Collection + +All control-plane services expose Prometheus-compatible metrics endpoints. You can scrape these metrics using: + +- **Prometheus Operator**: Create ServiceMonitor resources based on the provided scrape targets +- **Prometheus**: Configure scrape targets manually +- **OpenTelemetry Collector**: Use the Prometheus receiver + +**Metrics Documentation:** + +Detailed metrics documentation is available for each service, including metric names, +types, labels, and descriptions. See the per-service metrics reference under the +`Metrics` section. + +### Self-managed metrics stack + +The Helmfile stack uses an observability profile to select the default metrics +components and monitor targets: + +| Profile | Shared monitor defaults | Function Autoscaler | NVCA observability defaults | +| --- | --- | --- | --- | +| `disabled` | None | Not installed | Disabled | +| `control` | Control-plane services | Installed | Disabled | +| `compute` | NVCA, DCGM, and worker pods | Not installed | Enabled | +| `all` | Control-plane and compute-plane targets | Installed | Enabled | + +The control-plane stack defaults to `control`. The compute-plane stack defaults +to `compute`. Use `all` when both sets of targets run in the same cluster. + +The default `control` profile installs the Prometheus Operator custom resource +definitions, OpenTelemetry Operator, collector with Target Allocator, default +control-plane monitors, and VictoriaMetrics. It also installs the Function +Autoscaler and requires State Metrics. + +The bundled VictoriaMetrics instance runs in `monitoring` by default. Set its +storage class in the Helmfile environment. See +[Helmfile Installation](./helmfile-installation.md#observability-configuration). + +Use `metricsBackend.mode: existing` to connect a customer-managed backend: + +```yaml metricsBackend: mode: existing type: external @@ -118,158 +141,327 @@ metricsBackend: mode: none ``` -`remoteWriteEndpoint` is required for an existing backend. `promqlEndpoint` is -also required for `control` and `all` because the Function Autoscaler queries -it. +The collector requires the remote-write endpoint. The `control` and `all` +profiles also require the PromQL endpoint because the Function Autoscaler +queries it. The autoscaler supports `none`, `token`, and `mtls` authentication +for PromQL queries. Configure collector remote-write authentication separately. -For the Function Autoscaler's PromQL client, authentication modes are `none`, -`token`, and `mtls`. Token authentication requires `authnEndpoint`. mTLS -requires `clientCertificatePath` and `clientPrivateKeyPath`. +Profiles set defaults. Components can use `install`, `existing`, or `disabled` +mode when another deployment owns them. -These settings apply only to the Function Autoscaler's PromQL client. They do -not configure collector remote-write authentication or mount credentials and -certificates. Configure remote-write authentication separately under -`collector.config.exporters.prometheusremotewrite`. +The shared collector discovers targets only in its Kubernetes cluster. In a +split deployment, configure compute-plane collection separately and make any +worker metrics used for autoscaling available to the control-plane backend. See +[Cluster Monitoring](./cluster-management/monitoring.md). -## Component ownership +### Logging -Profiles set defaults. Override a component only when another deployment owns -it: +**Log Format:** -| Mode | Meaning | -| --- | --- | -| `install` | The NVCF observability stack installs the component. | -| `existing` | The component is managed outside this stack. | -| `disabled` | The component is not used. | +- All services emit logs to stdout/stderr (standard for Kubernetes) +- Sensitive data redaction must be configured by the log collector -For example, keep customer-managed Prometheus Operator CRDs, OpenTelemetry -Operator, and metrics backend: +**Log Collection:** -```yaml -observability: - profile: control - components: - prometheusOperatorCrds: - mode: existing - otelOperator: - mode: existing +You can collect logs using any Kubernetes-compatible log aggregator: -metricsBackend: - mode: existing - type: external - remoteWriteEndpoint: https://metrics.example.com/write - promqlEndpoint: https://metrics.example.com -``` +- Fluentd or Fluent Bit +- Promtail (for Loki) +- Filebeat (for Elasticsearch) +- OpenTelemetry Collector (filelog receiver) + +**System Logs:** + +System logs are available at standard UNIX locations and from the systemd journal. + +### Tracing (Available in GA) -The configurable components are: +Distributed tracing support via OpenTelemetry Protocol (OTLP) is planned for a future release: -- `observability.components.prometheusOperatorCrds` -- `observability.components.otelOperator` -- `observability.components.collector` -- `observability.components.targetAllocator` -- `observability.components.discoveryRbac` -- `metricsBackend` +- Key flows will be instrumented with OpenTelemetry SDK +- Traces will be exportable via OTLP (HTTP or gRPC) +- Configurable sampling strategies +- Support for any OTLP-compatible backend (Jaeger, Tempo, Zipkin, etc.) +- Tracing is configurable via Helm values under `global.observability.tracing` -Helmfile rejects combinations that leave an installed component without a -required dependency. +## Configuration -## Monitor overrides +You can use the shared metrics stack or integrate with your own backend. -Profiles select monitor groups, but each group and target can be overridden: +### Metrics Scraping + +The observability profile configures the shared collector and default monitors. +Some service charts also use `global.observability.metrics.enabled` to enable +their own metrics exports or PodMonitors. Set it separately when those service +metrics are needed. + +Use Prometheus Operator with the provided ServiceMonitor examples: ```yaml -observability: - profile: control - -defaultMonitors: - controlPlane: - enabled: true - computePlane: - enabled: false - worker: - enabled: false +# Example ServiceMonitor for NVCF API +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: nvcf-api + namespace: nvcf +spec: + selector: + matchLabels: + app: nvcf-api + endpoints: + # Endpoint created based on the scrape target in the + # per-service metrics documentation + - port: metrics + interval: 30s + path: /metrics ``` -The shared collector discovers targets only in its Kubernetes cluster. The -compute-plane stack does not install this shared collector. For a split -deployment, configure compute-plane collection separately and make any worker -metrics used for autoscaling available to the control-plane backend. See -[Cluster Monitoring](./cluster-management/monitoring.md). +Or configure Prometheus scrape targets manually in your prometheus.yml. -## Dashboards +#### Application-level NVCF stats -The shared stack does not install a dashboard UI. Query the bundled or external -backend with a PromQL-compatible tool. The -[Example Dashboards](./example-dashboards.md) guide deploys a separate -development reference stack with its own metrics components. Review component -ownership before using both stacks in one cluster. +The State Metrics Service exposes per-function signals you can query in +Prometheus. The following PromQL examples cover the three most common +operator questions. Metric names and labels are sourced from +[State Metrics Service metrics](./metrics/state-metrics/metrics.md). -## Logs +Number of registered functions: -NVCF services write logs to standard output and standard error. The metrics -profile does not install a log backend. Use a Kubernetes log collector such as -Fluent Bit, Fluentd, Promtail, or an OpenTelemetry Collector configured for -logs. +```promql +# nvcf_function_info is emitted per function with descriptive labels. +# Dedupe by function_id so multiple label series do not inflate the count. +count(count by (function_id) (nvcf_function_info)) +``` + +Queue depth per function: + +```promql +# nvcf_function_queue_depth is a gauge keyed by function_id. +sum by (function_id, name) (nvcf_function_queue_depth) +``` -## Tracing configuration +Function request latency (p50 and p95) over a 5 minute window: + +```promql +# p50 +histogram_quantile( + 0.50, + sum by (le, function_id) (rate(function_request_latency_bucket[5m])) +) + +# p95 +histogram_quantile( + 0.95, + sum by (le, function_id) (rate(function_request_latency_bucket[5m])) +) +``` -Control-plane services can export traces to an OpenTelemetry Protocol (OTLP) -endpoint configured under `global.observability.tracing`: +### Log Collection + +Deploy a log collector as a DaemonSet to ship logs to your backend: + +```bash +# Example: Deploy Promtail for Loki +kubectl apply -f promtail-daemonset.yaml + +# Example: Deploy Fluentd or Fluent Bit +kubectl apply -f fluentd-daemonset.yaml +``` + +Configure your log collector to: + +- Tail logs from all namespaces +- Add metadata labels (pod name, namespace, service) +- Forward to your log aggregation backend (Loki, Elasticsearch, etc.) + +### Tracing Configuration + +Enable distributed tracing by setting Helm values under +`global.observability.tracing`. The control-plane exports traces via OTLP +to your own OTLP-compatible collector. Set `collectorEndpoint`, +`collectorPort`, and `collectorProtocol` to match your collector's address. +`collectorProtocol` is the endpoint URI scheme expected by the stack, not the +OTLP transport. + +Helm overrides example: ```yaml global: observability: tracing: enabled: true - collectorEndpoint: otel-collector.monitoring.svc.cluster.local + collectorEndpoint: "otel-collector-gateway-collector.observability.svc.cluster.local" collectorPort: 4317 collectorProtocol: http ``` -`collectorProtocol` supplies the URI scheme used by the stack. It does not -select the OTLP transport. +Configuration fields: -## Verify +- `enabled`: Set to `true` to enable OTLP trace export from control-plane + services. +- `collectorEndpoint`: DNS name or address of your OTLP collector (e.g., + OpenTelemetry Collector, Jaeger collector). Use a Kubernetes service DNS name + such as `..svc.cluster.local` when the collector runs + in-cluster. +- `collectorPort`: Port on which the collector accepts OTLP traffic (e.g., + 4317 for gRPC, 4318 for HTTP depending on your collector setup). +- `collectorProtocol`: URI scheme used to build the collector endpoint + (`http` or `https`). This value does not select the OTLP transport. -Check the shared components: +Ensure your collector is deployed and reachable from the NVCF control-plane +namespace, and that it forwards traces to your backend (Jaeger, Tempo, Zipkin, +or another OTLP-compatible system). -```bash -kubectl get pods,pvc -n monitoring -kubectl get opentelemetrycollector -A -kubectl get servicemonitor,podmonitor -A -``` +## Dashboards -Replace `monitoring` if the observability components use another namespace. +Reference Grafana dashboards are provided for control-plane services showing critical metrics for key services: -For `control` and `all`, also check: +- ESS (Encrypted Secrets Service) -```bash -kubectl get deployment -n nvcf \ - -l app.kubernetes.io/instance=state-metrics -kubectl get deployment -n nvcf \ - -l app.kubernetes.io/instance=function-autoscaler -``` +- Cassandra -See [Function Autoscaler Operations](./autoscaling/operations.md) for backend -and health checks. +- Vault + +- Invocation Service + +- NVCF API + +- SIS (SPOT Instance Service) + +- Worker Pods (Utils Container, Init Container, Inference Container) + + - Note: Worker Pods are deployed in the backend cluster, not the control-plane + cluster, but their configuration is globally controlled as part of the control-plane + +- State Metrics Service + +**Dashboard Location:** + +Dashboards are provided in native Grafana JSON format for [file-provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#dashboards). + +Load dashboards into Grafana by placing them in `/etc/grafana/provisioning/dashboards/` on startup. + +Published dashboards will be available in the +[NVCF examples](https://github.com/NVIDIA/nvcf/tree/main/examples) public GitHub repository. ## Troubleshooting -| Symptom | Check | -| --- | --- | -| No observability releases | Confirm that `observability.profile` is not `disabled`. | -| VictoriaMetrics pod is pending | Check the persistent volume claim and configured storage class. | -| Metrics backend has no samples | Check the monitor labels, Target Allocator, collector logs, and remote-write endpoint. | -| Worker metrics are missing in a split deployment | Check compute-plane collection and connectivity to the backend queried by the autoscaler. | -| Function Autoscaler is not installed | Use the `control` or `all` profile and keep State Metrics enabled. | -| Function Autoscaler is not ready | Check Cassandra and the PromQL endpoint with the autoscaler health endpoint. | -| External backend authentication fails | Check the selected authentication mode and its required endpoints or certificate paths. | +For troubleshooting common observability issues: + +**Metrics not appearing:** + +1. Verify the service is exposing metrics: + + ```bash + # Port-forward to the service metrics port + kubectl port-forward -n nvcf svc/nvcf-api 8080:8080 + + # In another terminal, curl the metrics endpoint + curl http://localhost:8080/metrics + ``` + +2. Check ServiceMonitor or scrape configuration: -## Related documentation + ```bash + # Verify ServiceMonitor exists + kubectl get servicemonitor -n nvcf + + # Check ServiceMonitor details + kubectl describe servicemonitor nvcf-api -n nvcf + ``` + +3. Verify network policies allow scraping: + + ```bash + # List network policies that might block traffic + kubectl get networkpolicy -n nvcf + + # Test connectivity from Prometheus namespace + kubectl run -n --rm -it debug \ + --image=curlimages/curl --restart=Never -- \ + curl http://nvcf-api.nvcf.svc.cluster.local:8080/metrics + ``` + +4. Check service logs for errors: + + ```bash + # Check for metrics-related errors + kubectl logs -n nvcf deployment/nvcf-api | grep -i metric + ``` + +For shared stack or Function Autoscaler issues, see +[Function Autoscaler Operations](./autoscaling/operations.md). + +**Logs not being collected:** + +1. Verify log collector DaemonSet is running: + + ```bash + # Check DaemonSet status (e.g., for Fluentd/Fluent Bit) + # Note: Namespaces may be different depending on the log collector deployment + kubectl get daemonset -n logging + kubectl get pods -n logging -l app=fluent-bit + ``` + +2. Check collector can access pod logs: + + ```bash + # Verify log collector has proper volume mounts + kubectl describe daemonset fluent-bit -n logging | grep -A5 Mounts + + # Check collector logs for errors + kubectl logs -n logging -l app=fluent-bit --tail=50 + ``` + +3. Verify log backend is reachable: + + ```bash + # Test connectivity to log backend (e.g. Loki) + kubectl run -n logging --rm -it debug \ + --image=curlimages/curl --restart=Never -- \ + curl -v http://loki.logging.svc.cluster.local:3100/ready + ``` + +4. Check for log redaction or filtering rules: + + ```bash + # Review collector configuration + kubectl get configmap fluent-bit-config -n logging -o yaml + + # Check if logs are being dropped + kubectl logs -n logging -l app=fluent-bit | grep -i "drop\|filter" + ``` + +## Security + +**Metrics Endpoints:** + +- Metrics endpoints should be accessed over HTTP in-cluster only + + - Any external access should be SSL/TLS or mTLS secured with a reverse proxy or other ingress controller, or + - Aggregated locally and exposed via a secured otel-collector + +- All sensitive log data should be redacted by the log collector (currently, this is the responsibility of the log collector, not the service) + + - Example implementation by OTEL Collector: [Log Redaction](https://opentelemetry.io/docs/languages/dotnet/logs/redaction/) + +- User-provided observability backend should be properly secured with RBAC, TLS/SSL, and other security best practices. + +## Related Documentation -- [Metrics Overview](./metrics/metrics-index.md) - [Function Autoscaling](./autoscaling/index.md) - [Function Autoscaler Observability](./autoscaling/observability.md) - [Cluster Monitoring](./cluster-management/monitoring.md) -- [Example Dashboards](./example-dashboards.md) -- [Control Plane Operations](./control-plane-operations.md) +- [OpenTelemetry documentation](https://opentelemetry.io/docs/) +- [Prometheus documentation](https://prometheus.io/docs/) + +## Version Compatibility + +NVCF self-hosted control-plane observability is compatible with: + +- Supported versions are the latest Kubernetes minor release and the two prior minor releases (N-2). See official Kubernetes docs for current supported [versions](https://kubernetes.io/releases/version-skew-policy/#supported-versions). +- Any Prometheus-compatible metrics collection system +- Any log aggregation system that can collect from Kubernetes stdout/stderr or read + from the filesystem (depending on K8s cluster configuration) + +For the latest compatibility information, see the release notes.