Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,10 @@ kubectl port-forward -n ate-system svc/atenet-router 8000:80

3. In a **separate terminal**, send an HTTP request to increment the counter:
```shell
curl -X POST -H "Host: my-counter-1.ate-demo-counter.actors.resources.substrate.ate.dev" -i http://localhost:8000/
curl -X POST \
-H "X-Ate-Actor-Name: my-counter-1" \
-H "X-Ate-Atespace: ate-demo-counter" \
-i http://localhost:8000/
```

Worker capacity is versioned: the dataplane (the atelet DaemonSet and the
Expand Down Expand Up @@ -234,7 +237,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit
* `cmd/ateapi`: The core control plane API server exposing gRPC endpoints to manage actor and worker lifecycles.
* `cmd/atelet`: A node-level DaemonSet that supervises physical worker pods, coordinates snapshotting, and manages state transfers.
* `cmd/atecontroller`: A Kubernetes controller that reconciles WorkerPool custom resources.
* `cmd/atenet`: A combined networking controller providing DNS, Envoy routing, and proxy sidecars.
* `cmd/atenet`: A combined networking controller providing Envoy routing and proxy sidecars.
* `cmd/ateom-gvisor`: An interior-pod helper running inside sandboxed worker pods to execute `runsc` checkpoint and restore commands.
* `cmd/ateom-microvm`: The micro-VM peer of `ateom-gvisor`, running actors as cloud-hypervisor VMs.
* `cmd/podcertcontroller`: A "polyfill" that provides Pod Certificate signers that
Expand Down
2 changes: 1 addition & 1 deletion benchmarking/automation/testtypes/nighthawk_ingress.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def validate(test: dict[str, Any]) -> None:
if not re.fullmatch(r"[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?", nh["atespace"]):
raise ValueError(
f"nighthawk-ingress test {name!r} atespace {nh['atespace']!r} "
f"must be a DNS label (it is routed as a Host-header subdomain)"
f"must be a DNS label"
)


Expand Down
16 changes: 7 additions & 9 deletions benchmarking/locust/tests/counter_demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,9 @@
tracer = get_tracer(__name__)


# Atenet router fronts all actor traffic. Actors are addressed by setting
# the HTTP Host header to <actor-name>.<atespace>.actors.resources.substrate.ate.dev;
# the router resolves that to the actor's current worker pod.
# Atenet router fronts all actor traffic. The actor routing headers select
# the actor whose current worker pod the router resolves.
ROUTER_URL = "http://atenet-router.ate-system.svc.cluster.local"
ACTOR_DOMAIN = "actors.resources.substrate.ate.dev"


class CounterUser(User):
Expand Down Expand Up @@ -99,12 +97,9 @@ def on_start(self) -> None:
except Exception as e:
logger.error(f"Failed to create actor {self.actor_name}: {e}")

# One HTTP session per user, talking to the router. The Host header
# pins each request to this actor regardless of which worker pod
# hosts it after a resume.
# One HTTP session per user, talking to the router.
self.http_session = requests.Session()
self.run_url = f"{ROUTER_URL}/"
self.host_header = f"{self.actor_name}.{ATESPACE}.{ACTOR_DOMAIN}"

def on_stop(self) -> None:
update_user_count(-1, self.__class__.__name__)
Expand Down Expand Up @@ -139,7 +134,10 @@ def run_and_suspend(self) -> None:
# 2. Run/Increment (HTTP via atenet-router)
start_time = time.time()
with tracer.start_as_current_span("RunCounter") as span:
headers = {"Host": self.host_header}
headers = {
"X-Ate-Actor-Name": self.actor_name,
"X-Ate-Atespace": ATESPACE,
}
inject(headers)
try:
response = self.http_session.post(self.run_url, headers=headers)
Expand Down
12 changes: 6 additions & 6 deletions benchmarking/nighthawk-ingress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ just don't steer the search.*
## What it measures

Every request exercises the **full production routing path** — created and
warmed sandboxed actors, Host-header routing, the ext_proc routing
warmed sandboxed actors, identity-header routing, the ext_proc routing
decision, and the mTLS hop to the worker:

```mermaid
flowchart LR
subgraph job["nighthawk runner Job"]
alc["nighthawk_adaptive_load_client<br/>exponential ramp + binary search"]
svc["nighthawk_service<br/>16 event loops, Host header<br/>rotated across all actors"]
svc["nighthawk_service<br/>16 event loops, actor headers<br/>rotated across all actors"]
alc -->|gRPC| svc
end

Expand All @@ -54,7 +54,7 @@ flowchart LR
atunnel["atunnel :443"] --> glutton["glutton actor<br/>POST /ping :80"]
end

svc -->|"HTTP :80, Host:<br/>actor-N.benchmark.actors..."| envoy
svc -->|"HTTP :80<br/>X-Ate-Actor-Name + X-Ate-Atespace"| envoy
extproc -->|ResumeActor| ateapi
envoy -->|"mTLS :443"| atunnel
```
Expand Down Expand Up @@ -84,9 +84,9 @@ One Kubernetes Job per `type: nighthawk-ingress` tests.yaml entry, driven by
down afterwards, so nothing leaks.
2. **Create + warm actors.** The runner creates one glutton actor per
WorkerPool worker (the entry's `workerCount`) via ateapi and POSTs
`/ping` through the router with each actor's Host header until it
`/ping` through the router with each actor's routing headers until it
answers 200.
3. **Adaptive search.** Open-loop traffic with the Host header rotated
3. **Adaptive search.** Open-loop traffic with the actor routing headers rotated
across all actors; `clientConcurrency` event loops (default 16,
decoupled from `envoyCpu`) and large per-loop pools so the harness is
never the bottleneck. Exponential ramp → binary search → a 60s
Expand Down Expand Up @@ -174,7 +174,7 @@ Start with `capacity.json`, drill into `stats.jsonl`:

The fleet size is the entry's top-level `workerCount` (required): the
benchmark warms one actor per worker, so it is also the number of glutton
actors receiving rotated-Host traffic. Everything else lives in the
actors receiving rotated identity-header traffic. Everything else lives in the
`nighthawk-ingress:` block:

| Knob | Default | Meaning |
Expand Down
8 changes: 5 additions & 3 deletions benchmarking/nighthawk-ingress/actors.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,11 +110,10 @@ def _warm_actor(
) -> None:
"""Bring one created actor to serving, so the load ladder never
measures a cold start: resume the actor, then poll POST /ping through
the router (addressed by the actor's Host header) until it answers
the router until it answers
200 or the deadline expires. Resume errors are retried: ateapi
returns FailedPrecondition/Unavailable until a worker frees up."""
ref = ateapi_pb2.ObjectRef(atespace=atespace, name=name)
host = spec_mod.actor_host(name, atespace)
session = requests.Session()
last_err: str = "not attempted"
while time.time() < deadline:
Expand All @@ -125,7 +124,10 @@ def _warm_actor(
try:
resp = session.post(
f"{router_url.rstrip('/')}/ping",
headers={"Host": host},
headers={
"X-Ate-Actor-Name": name,
"X-Ate-Atespace": atespace,
},
data=b"",
timeout=10,
)
Expand Down
6 changes: 2 additions & 4 deletions benchmarking/nighthawk-ingress/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -268,12 +268,10 @@ def main() -> None:
log=lambda m: tee(logs, m),
)

hosts = [
spec_mod.actor_host(n, args.atespace) for n in actor_names
]
spec_dict = spec_mod.build_spec_dict(
uri=f"{args.router_url.rstrip('/')}/ping",
hosts=hosts,
actor_names=actor_names,
atespace=args.atespace,
client_concurrency=args.client_concurrency,
connections=args.connections,
max_pending_requests=args.max_pending,
Expand Down
18 changes: 9 additions & 9 deletions benchmarking/nighthawk-ingress/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,6 @@
)


def actor_host(actor: str, atespace: str) -> str:
"""Host header the router routes on (internal/resources/actor.go)."""
return f"{actor}.{atespace}.actors.resources.substrate.ate.dev"


def _metric_spec(name: str) -> dict:
return {"metric_name": name, "metrics_plugin_name": BUILTIN_METRICS_PLUGIN}

Expand Down Expand Up @@ -91,7 +86,8 @@ def _binary_threshold(
def build_spec_dict(
*,
uri: str,
hosts: list[str],
actor_names: list[str],
atespace: str,
client_concurrency: int,
connections: int,
max_pending_requests: int,
Expand All @@ -117,12 +113,16 @@ def build_spec_dict(
"request_method": "POST",
"request_headers": [
{
"header": {"key": "host", "value": host},
"header": {"key": "x-ate-actor-name", "value": actor_name},
"append_action": "OVERWRITE_IF_EXISTS_OR_ADD",
},
{
"header": {"key": "x-ate-atespace", "value": atespace},
"append_action": "OVERWRITE_IF_EXISTS_OR_ADD",
}
},
],
}
for host in hosts
for actor_name in actor_names
]
# Threshold roles: tail latency (mean+2stdev, ~p95 proxy — no true
# percentiles in the builtin adaptive metrics) is the SLO bound;
Expand Down
31 changes: 16 additions & 15 deletions benchmarking/nighthawk-ingress/tests/test_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,8 @@
def build(**overrides):
kwargs = dict(
uri="http://atenet-router.ate-system.svc.cluster.local:80/ping",
hosts=[
spec_mod.actor_host(f"sb-{i}", "benchmark") for i in range(3)
],
actor_names=[f"sb-{i}" for i in range(3)],
atespace="benchmark",
client_concurrency=4,
connections=1000,
max_pending_requests=10000,
Expand Down Expand Up @@ -67,14 +66,23 @@ def test_traffic_template_shape():
}


def test_host_rotation_covers_all_actors():
hosts = [spec_mod.actor_host(f"sb-{i}", "benchmark") for i in range(5)]
spec = build(hosts=hosts)
def test_identity_rotation_covers_all_actors():
actor_names = [f"sb-{i}" for i in range(5)]
spec = build(actor_names=actor_names)
plugin = spec["nighthawk_traffic_template"]["request_source_plugin_config"]
assert plugin["name"] == spec_mod.REQUEST_SOURCE_PLUGIN
options = plugin["typed_config"]["options_list"]["options"]
got = [o["request_headers"][0]["header"]["value"] for o in options]
assert got == hosts
got = [
{header["header"]["key"]: header["header"]["value"] for header in o["request_headers"]}
for o in options
]
assert got == [
{
"x-ate-actor-name": actor_name,
"x-ate-atespace": "benchmark",
}
for actor_name in actor_names
]
assert all(o["request_method"] == "POST" for o in options)
# 0 = loop the list indefinitely.
assert plugin["typed_config"]["num_requests"] == 0
Expand Down Expand Up @@ -116,13 +124,6 @@ def test_tail_latency_slo_threshold():
assert len(build(tail_latency_slo_ms=None)["metric_thresholds"]) == 2


def test_actor_host_format():
assert (
spec_mod.actor_host("sb-1", "benchmark")
== "sb-1.benchmark.actors.resources.substrate.ate.dev"
)


def test_spec_round_trips_through_real_protos():
"""Round-trip through the real Nighthawk protos; self-skips outside the
runner image (no FileDescriptorSet)."""
Expand Down
2 changes: 1 addition & 1 deletion cmd/ate-setup/internal/cmd/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ var deployControllerCmd = &cobra.Command{

var deployAtenetCmd = &cobra.Command{
Use: "atenet",
Short: "Deploy the atenet dataplane only: router, egress, and DNS",
Short: "Deploy the atenet dataplane only: router and egress",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return env.DeployAtenet(cmd.Context())
Expand Down
1 change: 0 additions & 1 deletion cmd/ate-setup/internal/steps/delete.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ func (e *Env) DeleteAtenet(ctx context.Context) error {
// other does not.
{"atenet-egress.yaml"},
{"atenet-egress-with-sdsmint.yaml"},
{"atenet-dns.yaml"},
} {
if err := e.Kube.DeletePath(ctx, e.Cfg.Manifest(path...)); err != nil {
return err
Expand Down
7 changes: 2 additions & 5 deletions cmd/ate-setup/internal/steps/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ func (e *Env) DeployAtelet(ctx context.Context) error {
return e.Kube.RolloutStatus(ctx, kube.KindDaemonSet, NamespaceAteSystem, ateletName, e.Cfg.RolloutTimeout)
}

// DeployAtenet redeploys the atenet dataplane: router, egress, and DNS.
// DeployAtenet redeploys the atenet dataplane: router and egress.
func (e *Env) DeployAtenet(ctx context.Context) error {
log.Step("deploy_atenet")

Expand All @@ -316,11 +316,8 @@ func (e *Env) DeployAtenet(ctx context.Context) error {
if err := e.applyAtenetEgress(ctx); err != nil {
return err
}
if err := e.ResolveAndApply(ctx, e.Cfg.Manifest("atenet-dns.yaml")); err != nil {
return err
}

for _, name := range []string{"atenet-router", "atenet-egress", "dns"} {
for _, name := range []string{"atenet-router", "atenet-egress"} {
if err := e.Kube.RolloutStatus(ctx, kube.KindDeployment, NamespaceAteSystem, name, e.Cfg.RolloutTimeout); err != nil {
return err
}
Expand Down
14 changes: 2 additions & 12 deletions cmd/atenet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

atenet is a combined daemon for all networking functionality.

* DNS server for ATE Actor resolution: `atenet dns`
* Envoy control plane for programming ATE resolution. `atenet router`

This is built as a single binary for convenience in the prototyping.
Expand Down Expand Up @@ -31,8 +30,8 @@ likely be split in the future for better scalability.)
a separate process on the worker pod, not part of Envoy -- so the port to
reach on the actor itself (its default port, or an arbitrary one for
CONNECT) still travels as a real header, `atunnel.TargetPortHeader`.
`:authority`/`Host` reaches atunnel unmodified either way, so it authorizes
the actor by its own DNS name.
`X-Ate-Actor-Name` and `X-Ate-Atespace` identify the Actor independently of
`:authority`/`Host`.
* Termination: the router drains gracefully on SIGTERM (readiness flip →
endpoint propagation → Envoy admin-API drain → ext_proc drain), and the
Envoy container's `preStop` hook waits for the router's drain-complete
Expand All @@ -45,15 +44,6 @@ likely be split in the future for better scalability.)
RBAC permissions:
* get, list, watch on ate-system EndpointSlices

### dns

* `atenet dns` will be deployed as:
* Deployment
* Service exposing tcp and udp 53

* read, list on kube-system services
* read, list on ate-system services

## testing

Run the package tests with `go test ./cmd/atenet/...`. Cluster e2e
Expand Down
Loading
Loading