ateapi.proto: Actor tags 4–7 were re-issued with new types and no reserved, so a pre-18-Aug client silently mis-decodes every actor and one RUNNING actor fails ListActors for the entire atespace

[mayawang]

Reported on 2026-09-01 by a partner running an external integration against a private build, and
diagnosed and reproduced with them. Byte dumps below are our own reproduction of the reported
failure, with a synthetic atespace name and UUID; the structure and lengths are unchanged.

Expected Behavior

Within a 0.x line, a client generated from an older ateapi.proto and pointed at a newer control
plane should either work, or fail with an error that names the problem. Specifically:

1. Field tags that are removed should be reserved, so no later change can re-issue them with a
   different meaning.
2. A client too old for the server it is talking to should say so, rather than decoding the wire
   format into plausible-looking nonsense.
3. One malformed or unexpected row should not deny an entire ListActors response. Losing the view
   of every actor in an atespace because of one actor's field is disproportionate.

Actual Behavior

None of the three.

message Actor had tags 4, 5, 6 and 7 freed by one change and re-issued with different types by a
later one, and ateapi.proto contains no reserved statement anywhere. A client generated
before 2026-08-18 therefore mis-decodes every actor it sees, and the failure is silent until it
suddenly is not:

kubectl-ate get actors -a <atespace>
Error: failed to list actors: rpc error: code = Internal desc = grpc:
failed to unmarshal the received message: string field contains invalid UTF-8

Not one actor — the whole list, so every actor in the atespace becomes invisible at once. Before
that point, the same client reports actors as STATUS_UNSPECIFIED (i.e. "not running") with no
error at all, and prints serialized protobuf in an address-shaped field:

ateomPodIp: '\x08\x04".\n\x06agents\x12$f0e1d2c3-4b5a-4697-8c9d-0a1b2c3d4e5f'

Nothing anywhere reports a version problem. The router logs Route ok throughout, and an external
gateway using the same generated client swallows the error, so agents simply stop being answerable.
The reporter found this by decoding the bytes by hand.

Steps to Reproduce the Problem

1. Build or obtain a client generated from ateapi.proto before 4c1bd9d3 (Add status fields to Substrate resources #1025, 2026-08-18)
   — any kubectl-ate, or any generated Go/Python client, built before that date. This is not an
   exotic setup: it is what anyone gets by building tooling once and leaving it alone.
2. Point it at a control plane built from 4c1b37d0 or later (2026-08-24, the base of
   release-0.1-rc, which contains Add status fields to Substrate resources #1025) — or from current main.
3. Create an actor and leave it SUSPENDED. Run kubectl-ate get actors -a <atespace> with the
   old client. It succeeds, and is wrong: status reads STATUS_UNSPECIFIED, and
   ateomPodIp contains serialized protobuf. Note that nothing errors — this is the dangerous
   state, because a fleet can sit in it indefinitely.
4. Resume the actor so it is RUNNING and has a worker_assignment (namespace, pool, pod name,
   36-character pod UID, pod IP, ObjectRef — any real cluster clears the threshold; see below).
5. Run kubectl-ate get actors -a <atespace> again. It now fails with
   string field contains invalid UTF-8, for every actor in the atespace, not just the running
   one. Any other caller of ListActors — including a gateway using the same generated client —
   fails identically.
6. Recovery: suspend the running actor. Listing works again immediately, with no data lost.
   Deleting it also works but is unnecessary and destroys state — see "Recovery" below, because the
   obvious inference from the symptom is the destructive one.
7. Without an old binary: encode
   ActorStatus{state: 4, latest_snapshot: ObjectRef{atespace: "…", name: "<uuid>"}} and read it
   back as a string field. It round-trips as valid UTF-8, which is the whole problem. Then do the
   same with a populated worker_assignment and watch the length prefix cross 0x80.

Specifications

• Version: control plane at 4c1b37d0 (2026-08-24) or later; client generated before
  4c1bd9d3 (2026-08-18). Confirmed still present on main at ea3bdc32 (2026-09-02): see
  "The pattern is still running" — the situation on main is worse, not better.

• Platform: any. This is a wire-format issue, not environment-dependent. Observed on GKE.

• Evidence: the kubectl-ate failure and the ateomPodIp blob above. Hand-encoding
  ActorStatus{state: 4, latest_snapshot: ObjectRef{atespace, name}} reproduces the reported blob
  byte for byte, which is how the tag mapping was confirmed:

  b'\x08\x04".\n\x06agents\x12$f0e1d2c3-4b5a-4697-8c9d-0a1b2c3d4e5f'
    \x08\x04   field 1, varint 4     -> ActorState = ACTOR_STATE_SUSPENDED
    \x22\x2e   field 4, 46 bytes     -> ObjectRef
               \n\x06agents            atespace = "agents"
               \x12\x24<36 chars>      name     = "f0e1d2c3-…"
  

---

Root cause

Two schema changes freed tags that a later change re-issued with different types.

Before 4f2fd491 — Wrap worker pod metadata into a WorkerAssignment message (#737, 2026-08-04):

message Actor {
  ResourceMetadata metadata = 1;
  string actor_template_namespace = 2;
  string actor_template_name = 3;
  Status status = 4;                  // nested enum, wiretype 0
  string ateom_pod_namespace = 5;     // wiretype 2
  string ateom_pod_name = 6;          // wiretype 2
  string ateom_pod_ip = 7;            // wiretype 2
}

After 4c1bd9d3 — Add status fields to Substrate resources (#1025, 2026-08-18):

message Actor {
  ResourceMetadata metadata = 1;
  string actor_template_namespace = 2;
  string actor_template_name = 3;
  ObjectRef actor_template = 4;       // wiretype 2  (was varint)
  Selector worker_selector = 5;       // wiretype 2
  ObjectRef source_snapshot_tag = 6;  // wiretype 2
  ActorStatus status = 7;             // wiretype 2
}

#737 freed 5, 6 and 7; #1025 freed 4 and re-issued all four. No reserved was added at either step.

Why there is no clean error. The old reader expects string at 5, 6 and 7. A string and an
embedded message are both wiretype 2, so the wire types match, the decoder is satisfied, and the
serialized submessage lands directly in the string. The only remaining check is proto3's UTF-8
validation on string fields — which is a coincidence, not a compatibility mechanism.

Tag 4 is the one case where the wiretype genuinely differs (varint against message). Go's runtime
files that away as an unknown field, without complaint, which is why the old client reports
STATUS_UNSPECIFIED — "not running" — for actors that are running fine.

What decides whether it is silent or fatal, and it is not luck

Every byte of the SUSPENDED-actor blob above is below 0x80, so it is valid UTF-8 and decodes
silently to garbage. The fatal case needs one byte >= 0x80 inside the serialized ActorStatus, and
the reliable source of one is worker_assignment:

• A SUSPENDED actor has no worker_assignment — the field is unset whenever the actor has no
  worker. The message stays short and all-ASCII. Silent.
• A RUNNING actor carries a WorkerAssignment with namespace, pool, pod name, pod UID, pod IP
  and an ObjectRef. With realistic values that is ~130 bytes, so its own length varint becomes
  two bytes — \x82\x01 — and 0x82 is not a valid UTF-8 start byte. The entire ListActors
  response fails.

The threshold is a worker_assignment of 128 bytes or more. Pod names, pool names and a
36-character pod UID clear it comfortably in any real cluster. This is ordinary data, not a corner
case: the failure is dormant in every cold fleet and arrives the first time enough actors are
running.

Blast radius

ListActors unmarshals as a single message, so one actor's bytes deny every caller's view of
every actor in the atespace. There is no per-row isolation and no error naming the offending
actor. And it is silent all the way up: the router logs Route ok, an external gateway using the
generated client swallows the error, and agents just stop being answerable with nothing in any log a
person would notice.

Recovery, for anyone hitting this today

Suspend the offending actor. Do not delete it. Suspending clears worker_assignment, which
drops the message back under the 128-byte threshold and restores listing, with no data lost. The
reporter's inferred recovery was suspend and delete, which also works and destroys state — an
understandable inference from the symptom, and the wrong one for any actor carrying something you
care about. This is worth correcting wherever recovery guidance is written down.

The real fix on the client side is to rebuild the client from the same commit as the control plane.

The pattern is still running

Since 4c1b37d0, 120 commits have landed on main, twenty of them touching ateapi.proto. None
addresses this, and the same discipline is still missing in changes going in now:

• There is still not one reserved statement in ateapi.proto. (The single textual match for
  "reserved" in the file is prose in a comment on GoldenSnapshotStatus.)
• fe9013a4 (Full cutover: Drop the k8s CRD ActorTemplate fields in the ate apiserver, and update e2e tests #1353, Full cutover: drop the k8s CRD ActorTemplate fields) deleted
  Actor.actor_template_namespace = 2 and actor_template_name = 3 without reserving them.
  message Actor on main now runs 1, 4, 5, 6, 7 — tags 2 and 3 are free and loaded for
  whoever re-issues them next.
• e1adb331 (api: drop ActorTemplateStatus.sandbox_assets for now #1300, 2026-08-29) dropped ActorTemplateStatus.sandbox_assets = 2, also
  unreserved.

Both are cutover work, i.e. the pattern that caused this is being repeated by the workstream moving
fastest through the API. Whatever else is decided here, reserved on everything #737, #1025, #1300
and #1353 have freed is five minutes and stops the next instance.

What to ask for — and what reserved does not buy

To be honest about the limit first: reserved does not fix an already-shipped client. The
reporter's binary has string ateom_pod_ip = 7 compiled into it and no server-side change alters
that. reserved prevents the next instance; it does nothing for the one in front of us. Ranked by
what actually protects a pinned client against a moving control plane:

1. A client/server version check. The one that matters. An old client must say it is old, rather
   than reporting STATUS_UNSPECIFIED and an address-shaped field full of protobuf. Bigger than a
   one-liner and worth scoping now rather than rushing — a 0.1.x follow-up is fine, silence is
   not.
2. Per-row isolation in ListActors, or at minimum an error naming the offending actor instead
   of failing the whole page. Small, independently useful, and filed separately (see below) so it
   does not wait on the discussion above.
3. reserved 2, 3; in message Actor, plus a sweep across every tag freed by Wrap worker pod metadata into a WorkerAssignment message. #737, Add status fields to Substrate resources #1025,
   api: drop ActorTemplateStatus.sandbox_assets for now #1300 and Full cutover: Drop the k8s CRD ActorTemplate fields in the ate apiserver, and update e2e tests #1353. Five minutes; should be in the 4 Sep cut.
4. Correct the recovery guidance wherever it appears: suspend, do not delete.
5. A release note: within 0.x, the client and the control plane must be built from the same tag.

3, 4 and 5 can realistically make the cut. 2 is small. 1 is the one to schedule.

Release-policy implication

The 0.x compatibility policy covers a server upgrade — no supported upgrade across a minor,
incompatible API changes at the minor boundary. It does not cover a client older than the
server, which is what happened here and which any consumer produces routinely by building tooling
once and leaving it alone. Worth a line in the stability-window proposal: within 0.x, client and
control plane must be built from the same tag, and reserved is what makes the failure legible when
they are not.

[BenTheElder]

This is expected. We don't support compatibility yet.

[BenTheElder]

We've been compacting ahead of the first release.

[mayawang]

Makes sense — compacting before the first release is a good reason not to reserve. Dropping that ask
and taking this off P0.

One thing I'd still like, and it's a release-note line rather than code: no wire compatibility before
the first release, rebuild clients against the server's protos. Worth stating because skew is
currently silent rather than loud — tag 4 changed wiretype, so an old Go client files it as unknown
and reads STATUS_UNSPECIFIED for every actor with no error, and tag 7 kept its wiretype, so it
decodes into a plausible string. Happy for this to close on that.

I'll file the ListActors per-row isolation separately as promised above — worth having whatever we
do with tags.