The finding
Found while building the k3d harness (#18). The operator's startup gate calls STS with no endpoint override, so AWS_MICROVM_ENDPOINT never reaches it:
// operator-controller/.../health/AwsConnectivityStartup.java
void onStart(@Observes StartupEvent ev) {
try (StsClient sts = StsClient.builder().region(Region.of(region)).build()) {
var identity = sts.getCallerIdentity();
...
AwsConnectivityHealthCheck.setAwsConnectivityConfirmed(true);With no reachable STS the health check reports awsConnectivity: false permanently. Readiness never passes, the webhook Service gets no endpoints, and every CR create fails:
Internal error occurred: failed calling webhook "validate.microvms.lambda.aws.amazon.com":
no endpoints available for service "kube-microvm-operator"
The pod then CrashLoopBackOffs. Nothing in the logs points at STS — the operator reports all four reconcilers started successfully first, so the failure reads as a webhook or networking problem rather than a credentials one.
This is wider than m80: the operator cannot be brought up against any emulator, or any environment without reachable real STS. It is the same class of finding as #17, which m80 filed and upstream had already solved.
The workaround the harness uses
AWS_ENDPOINT_URL_STS is honoured by the SDK, so pointing it at something that answers GetCallerIdentity is enough. The harness points it at floci, which emulates STS:
AWS_ENDPOINT_URL_STS=http://floci.kube-microvm.svc.cluster.local:4566
The operator then logs AWS connectivity confirmed: account=000000000000 arn=arn:aws:iam::000000000000:root and comes up. m80 does not emulate STS and should not — it models one service.
What to propose upstream
Give the STS client the same endpoint-override treatment the MicroVMs client already has. Either honour AWS_ENDPOINT_URL_STS/AWS_ENDPOINT_URL explicitly, or let AWS_MICROVM_ENDPOINT cover it, or make the connectivity gate non-fatal when an override is configured — a startup probe that hard-fails on an unrelated service is harsh regardless.
Tasks
Gate: filing upstream is outward-facing and needs explicit approval.
The finding
Found while building the k3d harness (#18). The operator's startup gate calls STS with no endpoint override, so
AWS_MICROVM_ENDPOINTnever reaches it:With no reachable STS the health check reports
awsConnectivity: falsepermanently. Readiness never passes, the webhook Service gets no endpoints, and every CR create fails:The pod then CrashLoopBackOffs. Nothing in the logs points at STS — the operator reports all four reconcilers started successfully first, so the failure reads as a webhook or networking problem rather than a credentials one.
This is wider than m80: the operator cannot be brought up against any emulator, or any environment without reachable real STS. It is the same class of finding as #17, which m80 filed and upstream had already solved.
The workaround the harness uses
AWS_ENDPOINT_URL_STSis honoured by the SDK, so pointing it at something that answersGetCallerIdentityis enough. The harness points it at floci, which emulates STS:The operator then logs
AWS connectivity confirmed: account=000000000000 arn=arn:aws:iam::000000000000:rootand comes up. m80 does not emulate STS and should not — it models one service.What to propose upstream
Give the STS client the same endpoint-override treatment the MicroVMs client already has. Either honour
AWS_ENDPOINT_URL_STS/AWS_ENDPOINT_URLexplicitly, or letAWS_MICROVM_ENDPOINTcover it, or make the connectivity gate non-fatal when an override is configured — a startup probe that hard-fails on an unrelated service is harsh regardless.Tasks
Gate: filing upstream is outward-facing and needs explicit approval.