docs(function-autoscaler): add function autoscaler docs for self-hosted

docs/user/autoscaling/architecture.md

@@ -1,27 +1,35 @@
 # Function Autoscaler Architecture
 
-The function autoscaler is a Rust service deployed as a horizontally scaled Kubernetes Deployment. It reads utilization and request metrics from a Prometheus-compatible timeseries database, stores discovered functions and coordination state in Cassandra, and writes desired instance counts to the NVCF API. Cassandra lightweight transactions handle leader election and short-lived per-function locks.
+The Function Autoscaler runs as a Kubernetes Deployment in the control-plane
+cluster. It reads metrics from a PromQL-compatible backend, stores coordination
+state in Cassandra, and writes desired instance counts to the NVCF API.
 
-The work is split into two loops. A leader-elected discovery loop scans the timeseries database for active function versions and upserts them into Cassandra. A scaling loop runs on every replica, but each replica only handles the functions whose IDs hash into its assigned buckets, so the active set is sharded across replicas.
+The work is split into two loops. A leader-elected discovery loop scans the
+timeseries database for active function versions and upserts them into
+Cassandra. A scaling loop runs on every replica, but each replica only handles
+the functions whose IDs hash into its assigned buckets, so the active set is
+sharded across replicas.
 
 ## Sequence Diagram
 
 ```mermaid
 sequenceDiagram
-    participant Workers as Workers / Invocation Services
-    participant TSDB as Time Series DB
+    participant Services as NVCF metrics endpoints
+    participant Collector as OpenTelemetry Collector
+    participant TSDB as Metrics backend
     participant Autoscaler as Function Autoscaler
     participant Cassandra as Cassandra
     participant NVCF as NVCF Service
 
-    Workers->>TSDB: Emit utilization and instance metrics
+    Collector->>Services: Scrape selected metrics
+    Collector->>TSDB: Remote write
 
-    Note over Autoscaler,Cassandra: Discovery loop (~15s, leader-elected)
+    Note over Autoscaler,Cassandra: Periodic discovery loop, leader-elected
     Autoscaler->>TSDB: Query active functions
     TSDB-->>Autoscaler: Function set
     Autoscaler->>Cassandra: Upsert newly discovered functions
 
-    Note over Autoscaler,NVCF: Scaling loop (~30s, per-bucket)
+    Note over Autoscaler,NVCF: Periodic scaling loop, per-bucket
     Autoscaler->>Cassandra: Read active functions for this node's buckets
     Autoscaler->>TSDB: Query current instances and utilization history
     TSDB-->>Autoscaler: Metrics
@@ -32,22 +40,43 @@ sequenceDiagram
     Autoscaler->>Cassandra: Write predicted count, refresh function TTL
 ```
 
-The discovery loop runs on one leader-elected replica. The scaling loop runs on every replica, but each replica only processes the function buckets assigned to it.
+The discovery loop runs on one leader-elected replica. The scaling loop runs on
+every replica, but each replica only processes its assigned function buckets.
 
-## Timeseries Database
+## Deployment order
 
-The function autoscaler is a read-only client of a Prometheus-compatible timeseries store. It calls the `/api/v1/query_range` HTTP endpoint and uses PromQL for every metric query, so any backend that implements that interface works: upstream Prometheus, Thanos, Grafana Mimir, or VictoriaMetrics. The reference NVCF deployments point at VictoriaMetrics via the `timeseries_db_url` setting.
+With the default `control` profile, the observability stage installs the shared
+metrics components and backend. The final control-plane stage installs State
+Metrics, then the Function Autoscaler. The autoscaler depends on State Metrics.
 
-The function autoscaler does not run a scrape config of its own and does not write samples. Before it can do anything useful, the rest of the data plane has to be feeding the same store:
+The shared metrics stage is skipped for `disabled`. The Function Autoscaler is
+installed only for `control` and `all`.
 
-- Worker pods export utilization and instance count metrics (`nvcf_worker_service_worker_thread_busy_seconds_total`, `nvcf_worker_service_worker_thread_count_total`, instance gauges).
-- Invocation services and the gRPC proxy export request counters (`function_request`, `function_request_total`) labeled by `function_id`, `function_version_id`, and `nca_id`. These labels are how the discovery loop finds active function versions.
+## Metrics backend
 
-For a self-hosted control plane, you need three things in place before bringing the function autoscaler online:
+The autoscaler is a read-only client of a PromQL-compatible backend. It uses
+range queries to discover active functions and read instance, request, and
+utilization metrics.
 
-1. A Prometheus-compatible store reachable from the function autoscaler pod.
-2. A scrape configuration (or remote-write feed) covering the worker pods and the invocation-plane services.
-3. The resulting query endpoint passed in as `timeseries_db_url`. The function autoscaler reports `not ready` on its readiness probe until that endpoint responds.
+The autoscaler does not scrape metrics. The metrics for the selected invocation
+path must reach the backend that it queries. These include:
+
+- State Metrics instance, concurrency, request latency, and function metadata.
+- Invocation Service and gRPC Proxy request counters used to discover active
+  function versions.
+- Worker thread count and busy-time metrics used for worker-based utilization.
+- LLM API Gateway request duration metrics used for LLM functions.
+
+For a split deployment, any worker metrics used for scaling must reach the
+backend that the autoscaler queries. See
+[Cluster Monitoring](../cluster-management/monitoring.md) for compute-plane
+metrics endpoints.
+
+The backend can be bundled VictoriaMetrics or an existing PromQL-compatible
+service. See [Observability Configuration](../observability.md) for backend,
+endpoint, and authentication settings.
+
+The autoscaler reports `not ready` until the query endpoint responds.
 
 ## Coordination and Self-Healing
 
@@ -62,3 +91,4 @@ Coordination relies on Cassandra TTLs to recover from failures without operator
 - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API.
 - [Function Autoscaler Operations](./operations.md) for health endpoints and common issues.
 - [Function Autoscaler Observability](./observability.md) for emitted metrics, traces, and logs.
+- [Observability Configuration](../observability.md) for profiles and metrics backend settings.

docs/user/autoscaling/index.md

@@ -1,37 +1,55 @@
 # Function Autoscaling
 
-The NVCF Function Autoscaler is a distributed Rust service that monitors function utilization and uses it to determine the ideal instance count per function on the NVCF control plane. It runs as a horizontally scaled deployment on the same Kubernetes cluster as the rest of the control plane.
-
-On an interval, the function autoscaler reads metrics from the timeseries database, decides how many instances each function should have, and calls the NVCF API to apply that decision.
-
-The function autoscaler depends on a Prometheus-compatible timeseries database fed by the worker pods and invocation-plane services. Without it, the service reports `not ready` and makes no scaling decisions. See [Timeseries database](./architecture.md#timeseries-database) for the required metrics and endpoints.
+The NVCF Function Autoscaler reads function metrics, calculates a desired
+instance count, and sends that count to the NVCF API. It runs in the
+self-hosted control-plane cluster.
 
 ## Function Autoscaler vs Horizontal Pod Autoscaler
 
-Function autoscaling is distinct from Kubernetes horizontal pod autoscaling (HPA). HPA scales pods within a single cluster, so it cannot reach NVCF worker pods that are spread across multiple clusters. Function autoscaling orchestrates scaling across clusters using global load patterns.
+Function autoscaling is distinct from Kubernetes horizontal pod autoscaling
+(HPA). HPA scales a Kubernetes workload in one cluster. The Function
+Autoscaler sets the desired instance count for an NVCF function version, which
+can run across NVCF compute clusters.
 
 ## Key Functionality
 
-- Discovers active functions from invocation and worker metrics in the timeseries database and persists the active set in Cassandra.
-- Periodically computes a desired instance count per function from recent utilization and the function's scaling policy.
+- Discovers active functions from invocation and worker metrics in the
+  timeseries database and persists the active set in Cassandra.
+- Periodically computes a desired instance count per function from recent
+  utilization and the function's scaling policy.
 - Applies the desired count by calling the NVCF API's predictions endpoint.
-- Coordinates work across replicas using hash-based bucket assignment and Cassandra Lightweight Transaction (LWT) distributed locks.
+- Coordinates work across replicas using hash-based bucket assignment and
+  Cassandra lightweight transaction (LWT) locks.
+
+## Self-hosted deployment
+
+The self-managed control-plane stack defaults to the `control` observability
+profile. The `control` and `all` profiles install the Function Autoscaler. The
+`compute` and `disabled` profiles do not.
+
+State Metrics must be enabled for `control` and `all`. With the default
+component modes, the control-plane stack also installs the shared collector and
+VictoriaMetrics. See [Observability Configuration](../observability.md) for
+profile and backend settings.
 
 ## Architecture Overview
 
 ```mermaid
 flowchart LR
-    Workers[Workers / Invocation Services] --> TSDB[(Time Series DB)]
+    Services[Metrics endpoints] --> Collector[OpenTelemetry Collector]
+    Collector --> TSDB[(VictoriaMetrics or external backend)]
     TSDB --> Autoscaler[Function Autoscaler]
     Autoscaler <--> Cassandra[(Cassandra)]
     Autoscaler --> NVCF[NVCF API]
 ```
 
-See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequence diagram and the bucket model.
+See [Architecture](./architecture.md#sequence-diagram) for the end-to-end
+sequence and bucket model.
 
 ## See Also
 
 - [Architecture](./architecture.md) for components, data flow, and the Cassandra LWT lock behavior that elects the discovery leader.
 - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API.
 - [Function Autoscaler Operations](./operations.md) for health endpoints and operational guidance.
 - [Function Autoscaler Observability](./observability.md) for the metrics, traces, and logs emitted by the service.
+- [Observability Configuration](../observability.md) for profiles and metrics backend configuration.

docs/user/autoscaling/observability.md

@@ -1,8 +1,14 @@
 # Function Autoscaler Observability
 
-The function autoscaler emits structured logs, Prometheus metrics that explain dependency health statuses and scaling decisions, and OpenTelemetry spans for outbound calls to its dependencies. The Prometheus exporter serves metrics on the address configured in `server.metrics.exporters`. The local settings file at `crates/server/resources/settings-local.yaml` uses `0.0.0.0:41338`.
+The Function Autoscaler emits structured logs, Prometheus metrics, and
+OpenTelemetry spans. The chart exposes its Prometheus exporter through the
+`function-autoscaler` service on the `metrics` port, which defaults to `41338`.
+The shared stack's default monitors do not include this service. Add a monitor
+or scrape target for it to collect these metrics.
 
-Job and namespace labels follow the standard NVCF naming convention for the cluster that runs the function autoscaler.
+These service metrics describe the autoscaler itself. They are separate from
+the function metrics that the autoscaler reads from VictoriaMetrics or an
+external backend.
 
 ## Metric reference
 

docs/user/autoscaling/operations.md

@@ -1,65 +1,73 @@
 # Function Autoscaler Operations
 
-This page covers operating the function autoscaler after deployment, including health probes, common operational issues, and pointers to the Helm chart values. For log filter syntax, metrics, and traces, see [Function Autoscaler Observability](./observability.md).
+The self-managed stack deploys the Function Autoscaler for the `control` and
+`all` observability profiles. State Metrics must remain enabled for both. See
+[Observability Configuration](../observability.md) for profile and metrics
+backend settings.
 
-## Health endpoints
-
-The function autoscaler exposes three HTTP health endpoints. Their exact paths differ from the rest of the NVCF control plane: liveness and readiness are namespaced under `/admin/health/`.
-
-| Endpoint | Purpose | Use as |
-|----------|---------|--------|
-| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. |
-| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. |
-| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. |
-
-The liveness probe deliberately does not check Cassandra or the timeseries database. Restarting the pod when those are unreachable does not help, so the function autoscaler stays running and lets readiness flip instead.
+Apply an environment change from the self-managed stack directory:
 
-## Common operational issues
+```bash
+make apply HELMFILE_ENV=<environment-name>
+```
 
-### Cassandra connection failures
+## Verify the deployment
 
-Symptoms: readiness flips to 503, `/health` reports the `cassandra_client` component as unhealthy, log lines from `rs_autoscaler::cassandra` show connection errors.
+Check State Metrics and the Function Autoscaler:
 
-Checks:
+```bash
+kubectl get deployment -n nvcf \
+  -l app.kubernetes.io/instance=state-metrics
+kubectl get deployment -n nvcf \
+  -l app.kubernetes.io/instance=function-autoscaler
+kubectl rollout status deployment/function-autoscaler -n nvcf
+```
 
-- SSL certificates are mounted at the path expected by `cassandra.ssl`. The function autoscaler container expects the cert directory to exist; create `/etc/app/config` if it is missing.
-- Credentials in the secrets file are valid for the configured keyspace.
-- The contact points resolve from the pod's network namespace.
+Confirm the resolved PromQL endpoint. This ConfigMap does not contain the
+backend credentials:
 
-### Timeseries database query failures
+```bash
+kubectl get configmap -n nvcf function-autoscaler-env \
+  -o jsonpath='{.data.TIMESERIES_DB__TIMESERIES_DB_URL}{"\n"}'
+```
 
-Symptoms: `nvcf_autoscaler.timeseries_db.requests_total` shows a rising error count, `auth_failure_total` or `server_side_failure_total` is non-zero, log lines from `rs_autoscaler::timeseries_db` show 4xx or 5xx responses.
+For the bundled backend, the result should point to `vmsingle` in the
+configured monitoring namespace. For an existing backend, it should match
+`metricsBackend.promqlEndpoint`.
 
-Checks:
-
-- `timeseries_db.timeseries_db_url` is reachable from the pod.
-- The bearer token in the secrets file is current. Token rotation is the most common cause of `auth_failure_total` spikes.
-- Query time ranges fit the retention window of the backing store.
-
-### NVCF API errors
-
-Symptoms: `nvcf_autoscaler.nvcf_api.request_duration_milliseconds` shows a sustained rise in 4xx or 5xx, scaling decisions stop applying.
+## Health endpoints
 
-Checks:
+The Function Autoscaler exposes three health endpoints:
 
-- The OAuth2 token endpoint is reachable and the client credentials in the secrets file are valid.
-- The functions being scaled are still in a deployable status. Functions in unexpected states are skipped, not retried.
-- `nvcf_api.disable_auth` is set as intended for the deployment. Leave it `false` whenever the NVCF API enforces authentication.
+| Endpoint | Purpose | Use as |
+| --- | --- | --- |
+| `GET /admin/health/liveness` | Always returns 200. Indicates the process is alive. | Kubernetes liveness probe. |
+| `GET /admin/health/readiness` | Returns 200 when all components are healthy, 503 otherwise. | Kubernetes readiness probe. |
+| `GET /health` | Returns per-component health for `cassandra_client` and `timeseries_db_client`. | Operator-facing detail and dashboards. |
 
-### Discovery is stalled
+Inspect the detailed endpoint through the service:
 
-Symptoms: the active function set in Cassandra stops growing despite traffic to new functions, `nvcf_autoscaler.distributed_lock.acquisition_failures_total` is rising across all replicas.
+```bash
+kubectl port-forward -n nvcf service/function-autoscaler 8181:8181
+curl http://127.0.0.1:8181/health
+```
 
-Checks:
+The liveness probe does not check Cassandra or the metrics backend. Dependency
+failures change readiness instead.
 
-- Inspect the `locks` table for the discovery lock row and its TTL. If the row never expires, the previous leader may have stopped refreshing without releasing it.
-- Confirm at least one replica's `nvcf_autoscaler.distributed_lock` gauge reports the leader state.
-- Restart the holding replica if the cluster is otherwise healthy. The lock expires within `discovery_lock_duration_seconds`.
+## Troubleshooting
 
-See [Architecture](./architecture.md#cassandra-lightweight-transactions-lwts) for the lock state machine.
+| Symptom | Check |
+| --- | --- |
+| Function Autoscaler is not installed | Use the `control` or `all` profile. Keep `stateMetrics.enabled: true`. |
+| `cassandra_client` is unhealthy | Check contact-point DNS, credentials, and the configured TLS files. |
+| `timeseries_db_client` is unhealthy | Check the resolved PromQL endpoint, authentication mode, credentials, and backend retention. |
+| Scaling decisions are not applied | Check NVCF API authentication and function status. |
+| Discovery does not find active functions | Confirm the backend contains the request and worker metrics listed in [Architecture](./architecture.md#metrics-backend). Check the discovery lock metrics and TTL. |
 
 ## See also
 
 - [Function Autoscaler Observability](./observability.md) for the metrics and traces referenced in the symptoms above.
 - [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds and policy via the NVCF API.
 - [Architecture](./architecture.md) for the component layout these symptoms map to.
+- [Observability Configuration](../observability.md) for shared stack settings.