diff --git a/docs/user/autoscaling/architecture.md b/docs/user/autoscaling/architecture.md index 532aa36ba..96f221d5f 100644 --- a/docs/user/autoscaling/architecture.md +++ b/docs/user/autoscaling/architecture.md @@ -1,27 +1,35 @@ # Function Autoscaler Architecture -The function autoscaler is a Rust service deployed as a horizontally scaled Kubernetes Deployment. It reads utilization and request metrics from a Prometheus-compatible timeseries database, stores discovered functions and coordination state in Cassandra, and writes desired instance counts to the NVCF API. Cassandra lightweight transactions handle leader election and short-lived per-function locks. +The Function Autoscaler runs as a Kubernetes Deployment in the control-plane +cluster. It reads metrics from a PromQL-compatible backend, stores coordination +state in Cassandra, and writes desired instance counts to the NVCF API. -The work is split into two loops. A leader-elected discovery loop scans the timeseries database for active function versions and upserts them into Cassandra. A scaling loop runs on every replica, but each replica only handles the functions whose IDs hash into its assigned buckets, so the active set is sharded across replicas. +The work is split into two loops. A leader-elected discovery loop scans the +timeseries database for active function versions and upserts them into +Cassandra. A scaling loop runs on every replica, but each replica only handles +the functions whose IDs hash into its assigned buckets, so the active set is +sharded across replicas. ## Sequence Diagram ```mermaid sequenceDiagram - participant Workers as Workers / Invocation Services - participant TSDB as Time Series DB + participant Services as NVCF metrics endpoints + participant Collector as OpenTelemetry Collector + participant TSDB as Metrics backend participant Autoscaler as Function Autoscaler participant Cassandra as Cassandra participant NVCF as NVCF Service - Workers->>TSDB: Emit utilization and instance metrics + Collector->>Services: Scrape selected metrics + Collector->>TSDB: Remote write - Note over Autoscaler,Cassandra: Discovery loop (~15s, leader-elected) + Note over Autoscaler,Cassandra: Periodic discovery loop, leader-elected Autoscaler->>TSDB: Query active functions TSDB-->>Autoscaler: Function set Autoscaler->>Cassandra: Upsert newly discovered functions - Note over Autoscaler,NVCF: Scaling loop (~30s, per-bucket) + Note over Autoscaler,NVCF: Periodic scaling loop, per-bucket Autoscaler->>Cassandra: Read active functions for this node's buckets Autoscaler->>TSDB: Query current instances and utilization history TSDB-->>Autoscaler: Metrics @@ -32,22 +40,43 @@ sequenceDiagram Autoscaler->>Cassandra: Write predicted count, refresh function TTL ``` -The discovery loop runs on one leader-elected replica. The scaling loop runs on every replica, but each replica only processes the function buckets assigned to it. +The discovery loop runs on one leader-elected replica. The scaling loop runs on +every replica, but each replica only processes its assigned function buckets. -## Timeseries Database +## Deployment order -The function autoscaler is a read-only client of a Prometheus-compatible timeseries store. It calls the `/api/v1/query_range` HTTP endpoint and uses PromQL for every metric query, so any backend that implements that interface works: upstream Prometheus, Thanos, Grafana Mimir, or VictoriaMetrics. The reference NVCF deployments point at VictoriaMetrics via the `timeseries_db_url` setting. +With the default `control` profile, the observability stage installs the shared +metrics components and backend. The final control-plane stage installs State +Metrics, then the Function Autoscaler. The autoscaler depends on State Metrics. -The function autoscaler does not run a scrape config of its own and does not write samples. Before it can do anything useful, the rest of the data plane has to be feeding the same store: +The shared metrics stage is skipped for `disabled`. The Function Autoscaler is +installed only for `control` and `all`. -- Worker pods export utilization and instance count metrics (`nvcf_worker_service_worker_thread_busy_seconds_total`, `nvcf_worker_service_worker_thread_count_total`, instance gauges). -- Invocation services and the gRPC proxy export request counters (`function_request`, `function_request_total`) labeled by `function_id`, `function_version_id`, and `nca_id`. These labels are how the discovery loop finds active function versions. +## Metrics backend -For a self-hosted control plane, you need three things in place before bringing the function autoscaler online: +The autoscaler is a read-only client of a PromQL-compatible backend. It uses +range queries to discover active functions and read instance, request, and +utilization metrics. -1. A Prometheus-compatible store reachable from the function autoscaler pod. -2. A scrape configuration (or remote-write feed) covering the worker pods and the invocation-plane services. -3. The resulting query endpoint passed in as `timeseries_db_url`. The function autoscaler reports `not ready` on its readiness probe until that endpoint responds. +The autoscaler does not scrape metrics. The metrics for the selected invocation +path must reach the backend that it queries. These include: + +- State Metrics instance, concurrency, request latency, and function metadata. +- Invocation Service and gRPC Proxy request counters used to discover active + function versions. +- Worker thread count and busy-time metrics used for worker-based utilization. +- LLM API Gateway request duration metrics used for LLM functions. + +For a split deployment, any worker metrics used for scaling must reach the +backend that the autoscaler queries. See +[Cluster Monitoring](../cluster-management/monitoring.md) for compute-plane +metrics endpoints. + +The backend can be bundled VictoriaMetrics or an existing PromQL-compatible +service. See [Observability Configuration](../observability.md) for backend, +endpoint, and authentication settings. + +The autoscaler reports `not ready` until the query endpoint responds. ## Coordination and Self-Healing @@ -62,3 +91,4 @@ Coordination relies on Cassandra TTLs to recover from failures without operator - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API. - [Function Autoscaler Operations](./operations.md) for health endpoints and common issues. - [Function Autoscaler Observability](./observability.md) for emitted metrics, traces, and logs. +- [Observability Configuration](../observability.md) for profiles and metrics backend settings. diff --git a/docs/user/autoscaling/index.md b/docs/user/autoscaling/index.md index 525d7bf7b..5e544e35c 100644 --- a/docs/user/autoscaling/index.md +++ b/docs/user/autoscaling/index.md @@ -1,33 +1,50 @@ # Function Autoscaling -The NVCF Function Autoscaler is a distributed Rust service that monitors function utilization and uses it to determine the ideal instance count per function on the NVCF control plane. It runs as a horizontally scaled deployment on the same Kubernetes cluster as the rest of the control plane. - -On an interval, the function autoscaler reads metrics from the timeseries database, decides how many instances each function should have, and calls the NVCF API to apply that decision. - -The function autoscaler depends on a Prometheus-compatible timeseries database fed by the worker pods and invocation-plane services. Without it, the service reports `not ready` and makes no scaling decisions. See [Timeseries database](./architecture.md#timeseries-database) for the required metrics and endpoints. +The NVCF Function Autoscaler reads function metrics, calculates a desired +instance count, and sends that count to the NVCF API. It runs in the +self-hosted control-plane cluster. ## Function Autoscaler vs Horizontal Pod Autoscaler -Function autoscaling is distinct from Kubernetes horizontal pod autoscaling (HPA). HPA scales pods within a single cluster, so it cannot reach NVCF worker pods that are spread across multiple clusters. Function autoscaling orchestrates scaling across clusters using global load patterns. +Function autoscaling is distinct from Kubernetes horizontal pod autoscaling +(HPA). HPA scales a Kubernetes workload in one cluster. The Function +Autoscaler sets the desired instance count for an NVCF function version, which +can run across NVCF compute clusters. ## Key Functionality -- Discovers active functions from invocation and worker metrics in the timeseries database and persists the active set in Cassandra. -- Periodically computes a desired instance count per function from recent utilization and the function's scaling policy. +- Discovers active functions from invocation and worker metrics in the + timeseries database and persists the active set in Cassandra. +- Periodically computes a desired instance count per function from recent + utilization and the function's scaling policy. - Applies the desired count by calling the NVCF API's predictions endpoint. -- Coordinates work across replicas using hash-based bucket assignment and Cassandra Lightweight Transaction (LWT) distributed locks. +- Coordinates work across replicas using hash-based bucket assignment and + Cassandra lightweight transaction (LWT) locks. + +## Self-hosted deployment + +The self-managed control-plane stack defaults to the `control` observability +profile. The `control` and `all` profiles install the Function Autoscaler. The +`compute` and `disabled` profiles do not. + +State Metrics must be enabled for `control` and `all`. With the default +component modes, the control-plane stack also installs the shared collector and +VictoriaMetrics. See [Observability Configuration](../observability.md) for +profile and backend settings. ## Architecture Overview ```mermaid flowchart LR - Workers[Workers / Invocation Services] --> TSDB[(Time Series DB)] + Services[Metrics endpoints] --> Collector[OpenTelemetry Collector] + Collector --> TSDB[(VictoriaMetrics or external backend)] TSDB --> Autoscaler[Function Autoscaler] Autoscaler <--> Cassandra[(Cassandra)] Autoscaler --> NVCF[NVCF API] ``` -See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequence diagram and the bucket model. +See [Architecture](./architecture.md#sequence-diagram) for the end-to-end +sequence and bucket model. ## See Also @@ -35,3 +52,4 @@ See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequen - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API. - [Function Autoscaler Operations](./operations.md) for health endpoints and operational guidance. - [Function Autoscaler Observability](./observability.md) for the metrics, traces, and logs emitted by the service. +- [Observability Configuration](../observability.md) for profiles and metrics backend configuration. diff --git a/docs/user/autoscaling/observability.md b/docs/user/autoscaling/observability.md index 131a30129..eea90a583 100644 --- a/docs/user/autoscaling/observability.md +++ b/docs/user/autoscaling/observability.md @@ -1,8 +1,14 @@ # Function Autoscaler Observability -The function autoscaler emits structured logs, Prometheus metrics that explain dependency health statuses and scaling decisions, and OpenTelemetry spans for outbound calls to its dependencies. The Prometheus exporter serves metrics on the address configured in `server.metrics.exporters`. The local settings file at `crates/server/resources/settings-local.yaml` uses `0.0.0.0:41338`. +The Function Autoscaler emits structured logs, Prometheus metrics, and +OpenTelemetry spans. The chart exposes its Prometheus exporter through the +`function-autoscaler` service on the `metrics` port, which defaults to `41338`. +The shared stack's default monitors do not include this service. Add a monitor +or scrape target for it to collect these metrics. -Job and namespace labels follow the standard NVCF naming convention for the cluster that runs the function autoscaler. +These service metrics describe the autoscaler itself. They are separate from +the function metrics that the autoscaler reads from VictoriaMetrics or an +external backend. ## Metric reference diff --git a/docs/user/autoscaling/operations.md b/docs/user/autoscaling/operations.md index 08e74d5b5..e18c1e72c 100644 --- a/docs/user/autoscaling/operations.md +++ b/docs/user/autoscaling/operations.md @@ -1,65 +1,73 @@ # Function Autoscaler Operations -This page covers operating the function autoscaler after deployment, including health probes, common operational issues, and pointers to the Helm chart values. For log filter syntax, metrics, and traces, see [Function Autoscaler Observability](./observability.md). +The self-managed stack deploys the Function Autoscaler for the `control` and +`all` observability profiles. State Metrics must remain enabled for both. See +[Observability Configuration](../observability.md) for profile and metrics +backend settings. -## Health endpoints - -The function autoscaler exposes three HTTP health endpoints. Their exact paths differ from the rest of the NVCF control plane: liveness and readiness are namespaced under `/admin/health/`. - -| Endpoint | Purpose | Use as | -|----------|---------|--------| -| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. | -| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. | -| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. | - -The liveness probe deliberately does not check Cassandra or the timeseries database. Restarting the pod when those are unreachable does not help, so the function autoscaler stays running and lets readiness flip instead. +Apply an environment change from the self-managed stack directory: -## Common operational issues +```bash +make apply HELMFILE_ENV= +``` -### Cassandra connection failures +## Verify the deployment -Symptoms: readiness flips to 503, `/health` reports the `cassandra_client` component as unhealthy, log lines from `rs_autoscaler::cassandra` show connection errors. +Check State Metrics and the Function Autoscaler: -Checks: +```bash +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=state-metrics +kubectl get deployment -n nvcf \ + -l app.kubernetes.io/instance=function-autoscaler +kubectl rollout status deployment/function-autoscaler -n nvcf +``` -- SSL certificates are mounted at the path expected by `cassandra.ssl`. The function autoscaler container expects the cert directory to exist; create `/etc/app/config` if it is missing. -- Credentials in the secrets file are valid for the configured keyspace. -- The contact points resolve from the pod's network namespace. +Confirm the resolved PromQL endpoint. This ConfigMap does not contain the +backend credentials: -### Timeseries database query failures +```bash +kubectl get configmap -n nvcf function-autoscaler-env \ + -o jsonpath='{.data.TIMESERIES_DB__TIMESERIES_DB_URL}{"\n"}' +``` -Symptoms: `nvcf_autoscaler.timeseries_db.requests_total` shows a rising error count, `auth_failure_total` or `server_side_failure_total` is non-zero, log lines from `rs_autoscaler::timeseries_db` show 4xx or 5xx responses. +For the bundled backend, the result should point to `vmsingle` in the +configured monitoring namespace. For an existing backend, it should match +`metricsBackend.promqlEndpoint`. -Checks: - -- `timeseries_db.timeseries_db_url` is reachable from the pod. -- The bearer token in the secrets file is current. Token rotation is the most common cause of `auth_failure_total` spikes. -- Query time ranges fit the retention window of the backing store. - -### NVCF API errors - -Symptoms: `nvcf_autoscaler.nvcf_api.request_duration_milliseconds` shows a sustained rise in 4xx or 5xx, scaling decisions stop applying. +## Health endpoints -Checks: +The Function Autoscaler exposes three health endpoints: -- The OAuth2 token endpoint is reachable and the client credentials in the secrets file are valid. -- The functions being scaled are still in a deployable status. Functions in unexpected states are skipped, not retried. -- `nvcf_api.disable_auth` is set as intended for the deployment. Leave it `false` whenever the NVCF API enforces authentication. +| Endpoint | Purpose | Use as | +| --- | --- | --- | +| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. | +| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. | +| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. | -### Discovery is stalled +Inspect the detailed endpoint through the service: -Symptoms: the active function set in Cassandra stops growing despite traffic to new functions, `nvcf_autoscaler.distributed_lock.acquisition_failures_total` is rising across all replicas. +```bash +kubectl port-forward -n nvcf service/function-autoscaler 8181:8181 +curl http://127.0.0.1:8181/health +``` -Checks: +The liveness probe does not check Cassandra or the metrics backend. Dependency +failures change readiness instead. -- Inspect the `locks` table for the discovery lock row and its TTL. If the row never expires, the previous leader may have stopped refreshing without releasing it. -- Confirm at least one replica's `nvcf_autoscaler.distributed_lock` gauge reports the leader state. -- Restart the holding replica if the cluster is otherwise healthy. The lock expires within `discovery_lock_duration_seconds`. +## Troubleshooting -See [Architecture](./architecture.md#cassandra-lightweight-transactions-lwts) for the lock state machine. +| Symptom | Check | +| --- | --- | +| Function Autoscaler is not installed | Use the `control` or `all` profile. Keep `stateMetrics.enabled: true`. | +| `cassandra_client` is unhealthy | Check contact-point DNS, credentials, and the configured TLS files. | +| `timeseries_db_client` is unhealthy | Check the resolved PromQL endpoint, authentication mode, credentials, and backend retention. | +| Scaling decisions are not applied | Check NVCF API authentication and function status. | +| Discovery does not find active functions | Confirm the backend contains the request and worker metrics listed in [Architecture](./architecture.md#metrics-backend). Check the discovery lock metrics and TTL. | ## See also - [Function Autoscaler Observability](./observability.md) for the metrics and traces referenced in the symptoms above. - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds and policy via the NVCF API. - [Architecture](./architecture.md) for the component layout these symptoms map to. +- [Observability Configuration](../observability.md) for shared stack settings. diff --git a/docs/user/helmfile-installation.md b/docs/user/helmfile-installation.md index 0a37bb874..ad1ce0cc9 100644 --- a/docs/user/helmfile-installation.md +++ b/docs/user/helmfile-installation.md @@ -33,14 +33,15 @@ ls ## Namespace Requirements -Each control-plane Helm chart must be installed into a specific namespace. These -namespace assignments are fixed and must not be changed because -service-to-service cluster DNS addressing and Vault (OpenBao) authentication -claims depend on this layout. +Each control-plane Helm chart must be installed into a specific namespace. The +control-plane namespace assignments are fixed because service-to-service DNS +addressing and Vault (OpenBao) authentication claims depend on them. The +observability stack uses `monitoring` by default, but its namespace is +configurable. | Namespace | Services | | --- | --- | -| `nvcf` | api, invocation-service, grpc-proxy, notary-service, reval, state-metrics | +| `nvcf` | api, invocation-service, grpc-proxy, notary-service, reval, state-metrics, function-autoscaler | | `api-keys` | api-keys, admin-issuer-proxy | | `ess` | ess-api | | `sis` | sis | @@ -48,12 +49,14 @@ claims depend on this layout. | `cassandra-system` | cassandra | | `nats-system` | nats | | `cert-manager` | cert-manager | +| `monitoring` (default) | OpenTelemetry Operator, collector, default monitors, VictoriaMetrics | | `envoy-gateway-system` | ingress (nvcf-gateway-routes) | Installing a chart into the wrong namespace will cause authentication failures such as `error validating claims: claim "/kubernetes.io/namespace" does not match any associated bound claim values`. -If you see this error, verify that every release is deployed in the namespace shown above. +If you see this error, verify that each control-plane release uses the required +namespace and each observability release uses its configured namespace. @@ -280,6 +283,18 @@ global: # collectorPort: # collectorProtocol: +# Install control-plane monitors, the bundled metrics backend, and the +# Function Autoscaler. +observability: + profile: control + +victoriaMetrics: + server: + persistentVolume: + enabled: true + size: 16Gi + storageClass: "gp3" # Customize to your storage class. + fakeGpuOperator: enabled: false # If deploying locally with no GPUs, true ubuntu: @@ -339,6 +354,18 @@ request-router host and port that worker pods can reach. See [LLM Function Enablement](./llm-function-enablement.md) for the complete addon configuration. +#### `observability` Configuration + +The self-managed control-plane stack defaults to +`observability.profile: control`. This installs the shared metrics components, +VictoriaMetrics, State Metrics, and the Function Autoscaler. Set the +VictoriaMetrics storage class for the target cluster. + +To use a customer-managed backend or change component ownership, see +[Observability Configuration](./observability.md). For autoscaler health and +backend checks, see +[Function Autoscaler Operations](./autoscaling/operations.md). + #### `domain` and `ingress` Configuration The `domain` and `ingress` sections of the environment file are used to configure the external access to the NVCF control plane. diff --git a/docs/user/observability.md b/docs/user/observability.md index 0a6520801..192b04fdc 100644 --- a/docs/user/observability.md +++ b/docs/user/observability.md @@ -14,6 +14,7 @@ Common operator questions and where to look on this page or in linked references | Where are gRPC proxy metrics? | [gRPC Proxy metrics](./metrics/grpc-proxy/metrics.md). The page documents client connection counts, NATS pipe health, gRPC worker session-attach latency, and HTTP RED metrics. | | How do I add custom spans or metrics in a Kit application? | Use the OpenTelemetry API directly, the OmniTrace helper, the Carbonite static metrics API, or the `omni::observability::IMeter` interface. Refer to the Omniverse Kit and Carbonite documentation for details. | | Where are reference dashboards? | [Example dashboards](./example-dashboards.md) and the [Dashboards](#dashboards) section below. | +| How do I configure the shared metrics stack? | See [Self-managed metrics stack](#self-managed-metrics-stack). | ## Overview @@ -36,8 +37,8 @@ The observability solution currently provides: to explore metrics, logs, and dashboards, see [self-hosted-example-dashboards](./example-dashboards.md). The example deployments are designed for development and testing only, and are not suitable -for production use. For production deployments, follow the guidance on this page to integrate -with your own observability infrastructure. +for production use. For production deployments, use the self-managed metrics stack or +integrate with your own observability infrastructure. @@ -50,14 +51,15 @@ NVCF self-hosted observability is currently in Early Access (EA). During EA, NVC - Documented metrics for critical control-plane services - Example scrape targets for prometheus-operator ServiceMonitor configuration - Metrics exposed via Prometheus-compatible endpoints +- Shared metrics collection with bundled VictoriaMetrics or an existing backend - Logs emitted to stdout/stderr for easy collection - Configuration and deployment documentation - Example dashboards for key metrics **Your Responsibility:** -- Deploy and manage your own observability backend (Prometheus, Grafana, Loki, Elasticsearch, etc.) -- Configure metrics scraping from control-plane services +- Configure storage for bundled VictoriaMetrics or connect an existing backend +- Configure compute-plane collection for split deployments - Deploy log collectors (e.g., Fluentd, Promtail, OTel Collector) to aggregate logs - Set up your preferred visualization and alerting tools @@ -71,6 +73,7 @@ The following control-plane services expose metrics and logs for monitoring: - **Invocation Service**: Handles function invocation requests - **SPOT Instance Service (SIS)**: Manages worker pod and cluster state - **State Metrics Service**: Aggregates and exports NVCF-specific metrics +- **Function Autoscaler**: Calculates desired function instance counts **Supporting Services:** @@ -102,6 +105,55 @@ Detailed metrics documentation is available for each service, including metric n types, labels, and descriptions. See the per-service metrics reference under the `Metrics` section. +### Self-managed metrics stack + +The Helmfile stack uses an observability profile to select the default metrics +components and monitor targets: + +| Profile | Shared monitor defaults | Function Autoscaler | NVCA observability defaults | +| --- | --- | --- | --- | +| `disabled` | None | Not installed | Disabled | +| `control` | Control-plane services | Installed | Disabled | +| `compute` | NVCA, DCGM, and worker pods | Not installed | Enabled | +| `all` | Control-plane and compute-plane targets | Installed | Enabled | + +The control-plane stack defaults to `control`. The compute-plane stack defaults +to `compute`. Use `all` when both sets of targets run in the same cluster. + +The default `control` profile installs the Prometheus Operator custom resource +definitions, OpenTelemetry Operator, collector with Target Allocator, default +control-plane monitors, and VictoriaMetrics. It also installs the Function +Autoscaler and requires State Metrics. + +The bundled VictoriaMetrics instance runs in `monitoring` by default. Set its +storage class in the Helmfile environment. See +[Helmfile Installation](./helmfile-installation.md#observability-configuration). + +Use `metricsBackend.mode: existing` to connect a customer-managed backend: + +```yaml +metricsBackend: + mode: existing + type: external + remoteWriteEndpoint: https://metrics.example.com/write + promqlEndpoint: https://metrics.example.com + authentication: + mode: none +``` + +The collector requires the remote-write endpoint. The `control` and `all` +profiles also require the PromQL endpoint because the Function Autoscaler +queries it. The autoscaler supports `none`, `token`, and `mtls` authentication +for PromQL queries. Configure collector remote-write authentication separately. + +Profiles set defaults. Components can use `install`, `existing`, or `disabled` +mode when another deployment owns them. + +The shared collector discovers targets only in its Kubernetes cluster. In a +split deployment, configure compute-plane collection separately and make any +worker metrics used for autoscaling available to the control-plane backend. See +[Cluster Monitoring](./cluster-management/monitoring.md). + ### Logging **Log Format:** @@ -134,19 +186,14 @@ Distributed tracing support via OpenTelemetry Protocol (OTLP) is planned for a f ## Configuration -You configure observability by integrating with your own backend: +You can use the shared metrics stack or integrate with your own backend. ### Metrics Scraping -Metrics export is opt-in and disabled by default. Enable it in your Helmfile -environment before configuring scrape targets: - -```yaml -global: - observability: - metrics: - enabled: true -``` +The observability profile configures the shared collector and default monitors. +Some service charts also use `global.observability.metrics.enabled` to enable +their own metrics exports or PodMonitors. Set it separately when those service +metrics are needed. Use Prometheus Operator with the provided ServiceMonitor examples: @@ -342,6 +389,9 @@ For troubleshooting common observability issues: kubectl logs -n nvcf deployment/nvcf-api | grep -i metric ``` +For shared stack or Function Autoscaler issues, see +[Function Autoscaler Operations](./autoscaling/operations.md). + **Logs not being collected:** 1. Verify log collector DaemonSet is running: @@ -399,6 +449,9 @@ For troubleshooting common observability issues: ## Related Documentation +- [Function Autoscaling](./autoscaling/index.md) +- [Function Autoscaler Observability](./autoscaling/observability.md) +- [Cluster Monitoring](./cluster-management/monitoring.md) - [OpenTelemetry documentation](https://opentelemetry.io/docs/) - [Prometheus documentation](https://prometheus.io/docs/) @@ -406,7 +459,7 @@ For troubleshooting common observability issues: NVCF self-hosted control-plane observability is compatible with: -- Supported versions are the latest Kubernetes minor release and the two prior minor releases (N-2). See official Kubernetes docs for current supported [versions](https://kubernetes.io/releases/version-skew-policy/#supported-versions). +- Supported versions are the latest Kubernetes minor release and the two prior minor releases (N-2). See official Kubernetes docs for current supported [versions](https://kubernetes.io/releases/version-skew-policy/#supported-versions). - Any Prometheus-compatible metrics collection system - Any log aggregation system that can collect from Kubernetes stdout/stderr or read from the filesystem (depending on K8s cluster configuration)