RFC: workload identity as a platform feature

[stone-z]

No description provided.

[QuentinBisson]

As the RFC notes, topology changes (cluster renames, region moves, MC migrations) invalidate all existing authorization policies. And it leaks internal architecture to anyone who can read the cert.

How about we instead use an hybrid approach spiffe://wid.giantswarm.com/v1/app/platform/falco/v1-2-3/loc/customer-x/eu-west-1/cluster-abc where the path has two explicit, versioned segments:

/app/... stable semantic identity (what it is)
/loc/... structural location (where it runs)

This lets relying parties write policies at any granularity:

• UC1: match on /app/platform/falco/* — any Falco, anywhere
• UC2: match on /app/platform/falco/*/loc/customer-x/eu-west-1/* — Falco in EU for customer X
  UC4: match on /app/platform/prometheus/*/loc/*/*/cluster-abc — Prometheus in this specific cluster

The /app/ and /loc/ separators also make it explicit which part of the ID is "what" vs "where" — and in theory, you could evolve each independently (e.g. bump to /app/v2/... or /loc/v2/...) without breaking the other.

The tradeoff is that location info still leaks cluster names/regions to anyone reading certs, but that seems unavoidable if you want scoped authorization to actually work.

[stone-z]

Yes, including semantics and then structure in the ID is currently my working idea. But it has tradeoffs.

• wildcards are not universally supported (prefix would have more support, but still not 100%)
• examining any ID discloses architecture information, and we'd need to anonymize customer in the IDs, which makes it less human-friendly

Plus some others. However, so far, my suggestion would still be to use that rough structure (semantic identity, then structural qualifiers) because I do think it's very important to be able to write policies that target both customers and regions, and I don't see a way to offer that semi-universally without having it encoded in the ID.

I would love if a reviewer did have an idea for hiding the structure information from the ID but still making it publicly available for policy writing.

[QuentinBisson]

I think having one per WC would definitely reduce the risks similar to zot being deployed on WCs but maybe we could provide this as an opt-in?

[stone-z]

My tendency is to make it optional, but opt-out. To support air-gapped clusters, we would need to have WC SPIRE anyway. So I would make MC + WC the default, since it is most resilient and ensures we're good at running WC SPIREs, but let customers choose to consolidate to using only MC SPIREs if they want to reduce the footprint and they accept the risk. I don't think it would be difficult to reconfigure a running setup, but I'd leave that to after the initial implementation.

[QuentinBisson]

Sure that makes a lot of sense

[QuentinBisson]

I think it would be easier for offboarding if they own it right?

[stone-z]

Definitely, and that's closer to what we already do with their domains in other contexts. It just means they have limited identity options until they set it up, and if they mess up DNS the blast radius is larger.

[puja108]

I would also default to them owning it

[QuentinBisson]

I think it would be interesting to know how many of those use cases we have for this right. Can this be fixed by adding a sidecar that presents the correct svid depending on what workload is coming (based on the other workload identity)?

[stone-z]

That actually isn't an issue for the relying party. The issue here is when a given client (say, alloy) has multiple identities/certs available, it needs to support giving a different one to different services (like, "hello Grafana Cloud, I'm a Giant Swarm Alloy!" vs "hello Customer X Victoria Metrics, I'm a Customer X Alloy!"). IMO the alloy example is a great use case for multiple identities, but client support for it is relatively unknown to me.

[pipo02mix]

Why not use global identities always? That way, they can be validated in a local environment or globally for external services. Maybe I don't understand it, but I suppose the difference lies in the SVID format, am I right?

[stone-z]

The global vs local is only one example of a more generic dual identity question (also see Open question 4). I would expect the vast majority to be global (simply meaning the trust/CA bundle is public) identities, except perhaps for air-gapped clusters or a few unusual use cases.

The challenge still remains if a workload needs multiple global identities or multiple local ones, as in the alloy example. We end up in cases where we need to choose whether a workload is "ours" or "theirs" when it might serve use cases for both (or we need to run separate "ours" and "theirs" instances).

[QuentinBisson]

If each customer MC runs its own GS regional SPIRE instance (not shared across customers), a customer extracting the intermediate CA can only forge GS identities scoped to their own path so the blast radius it limited to the customer (having access to the MC) sabotaging his own installation right?

It is the same trust model as the teleport agent running on the MC though is it not?

[stone-z]

My suggestion is that customers don't run or have any access to GS regional SPIREs.

a customer extracting the intermediate CA can only forge GS identities scoped to their own path

If we run a GS SPIRE in a customer MC, currently the intermediate cert could be used for signing any path, not limited to the customer's own. Support for only signing certain paths is part of the spec, but not yet implemented.

It is the same trust model as the teleport agent running on the MC though is it not?

They are different mechanisms and risks. During MC creation, we generate a token in the Teleport cluster which gets passed to the new MC. This token is used by teleport-operator to issue fresh tokens for all the attached WCs. But the initial MC-level token is still subject to Teleport RBAC and controlled centrally in the Teleport cluster. There are still some risk tradeoffs (see here if you're curious), but if a customer were to abuse their Teleport token it would be a much smaller risk than abusing an intermediate signing cert.

[QuentinBisson]

Alright that makes sense. But then I do wonder if the initial implementation could not start with this approach and then try to find a more secure solution later on. it could also change mid implementation as well unless named path is something that could be pushed upstream? Do you have infos on that feature?

[stone-z]

The X509-SVID standard says that the issue is most libraries doing the actual validation don't support the field, even though it's part of the X509 spec. If SPIRE issued a cert with a permittedSubtrees field (that's what it's called in the spec), and a library used by some server tried to check it, it would just fail. So in order to use that feature, many client libraries need to be updated to support it (and the SPIFFE folks are apparently already working on that). In theory it could show up any time, but I don't think we can influence the timeline.

[QuentinBisson]

There are tradeoffs to having it running in a customer WC though

[stone-z]

Absolutely 💯

I think the best case is the customer already has a SPIRE root and we just plug into that. However, the vast majority won't have one, so from a customer experience perspective, I think us providing a root in a special WC is the second best overall for them and us.

WDYT?

[QuentinBisson]

You're probably right but I would hope cloud providers could provide something for this and then this would be an onprem only thing :D

[stone-z]

Interesting, I was thinking of going the other way and making SPIRE the default everywhere, and only using provider-specific options in cases where there was a clear indication for it. I'll add building default orchestration of each provider's thing as an alternative

[QuentinBisson]

Do you think it would make sense to rationalize all the workload identity mechanism we use today (IRSA, the one on Azure, Teleport and maybe others) in this RFC or is it fine to check this out later? There might be some overlap at some point

[stone-z]

would make sense to rationalize all the workload identity mechanism we use today (IRSA, the one on Azure, Teleport and maybe others)

I hadn't planned on trying to survey all of the other methods specifically in this, however I would love to hear from users of those other methods about their usage.

It would be great to replace the provider-specific identity mechanisms with a single common one for all the use cases where that's possible and makes sense. We probably still end up needing some provider-specific stuff for early infrastructure lifecycle stuff, but that's fine.

Teleport is also a good callout. In theory, an alternative to this RFC would be to invest in building identity around Teleport Enterprise. However, that is a lot of lock-in, plus additional cost. But it would be an option.

[QuentinBisson]

The direction of the RFC is really good. There are still some opened questions but it should definitely not block the implementation of an MVP on giantswarm clusters to learn more about spiffe/spire and try to find solutions to the opened questions. Nice work @stone-z

[pipo02mix]

Is that possible? How can an S3 bucket verify a container credential? I am interested to learn about it

[QuentinBisson]

AWS trust policies are for this I think

[stone-z]

AWS can be configured to recognize the public trust bundle as an OIDC identity provider (example/tutorial here). Once that is set up, SPIFFE ids can be bound to IAM Roles and assume other roles just like human users.

[pipo02mix]

Why local should be different from global? I mean, can we use global to probe identity in local envs? If it is the case, we don't need that distinction, and every identity could be globally validated, and we have the benefit that can work in both cases

[pipo02mix]

It means the agent needs to pull identities frequently so new fresh container can authN using the new identity? I have no idea how this part works (agent <-> container <-> API)

[stone-z]

Right, there is an agent in every cluster (MC and WC) that identifies each workload and contacts the (nested) SPIRE server API to request / renew the certificates.

[pipo02mix]

Why OIDC discovery provider and federation bundle has to run in a WC instead of MC?

[stone-z]

Let's distinguish between the bundle and the SPIRE server itself.

The trust bundle and OIDC discovery document are static, and not synchronous requests to the SPIRE server. We could (and likely will) serve these statically from e.g. S3 in order to serve them regionally. They update in the order of every few hours by default.

The root SPIRE will serve synchronous API requests from the nested SPIRE servers, and nested SPIRE servers will serve synchronous API requests from the agents.

The SPIRE root shouldn't run in an MC to reduce both the security and operations blast radii. If the cluster running the root SPIRE were to be compromised, the cryptographic material there could be used to impersonate any workload in the fleet or create entirely new ones. Plus, customers with multiple MCs would need network routes to the "central" MC. It also means the lifecycle of the root server is tied to that of an MC. With our normal backups, it's much easier to recover a deleted WC than it is a deleted MC.

[teemow]

I'm not yet sold on the idea to use dedicated WCs for management functionality. And I would like to discuss the pros and cons before we make such a decision.

Cons are:

• Even more costs and overhead for a GS installation
• East West traffic between WCs
• Even more complex bootstrapping of installations for us. And more management overhead.

Maybe it is also good to discuss with customers?

[stone-z]

The actual location of all SPIRE roots will definitely get more discussion. We also need to figure out where to run ours. It may turn out that all SPIRE roots are some entirely different thing outside of our normal platform (like Teleport).

Ideally, customers would run their own SPIRE root however/wherever they run their most heavily protected infrastructure. This should be a discussion with them during onboarding.

Putting it in a WC would be an attempt to enable ourselves and customers who don't/won't run their own. If it's a goal to use strong cryptographic identities for auth from day 0 (also for things like authenticating our tools with their clusters), then we need to give them something on day 0. But I agree it might not be a WC, it could be something different we don't have a name for yet.

It shouldn't be on a management cluster because we've opened those up to usage by arbitrary users, and are running other workloads there that greatly increase the likelihood of compromise.

Root signing keys would typically included in the highest risk classification and protection level in a risk framework because compromise is so catastrophic. In organizations that already use X509 for anything else, even their cluster admins wouldn't have access to that material.

So, an MC wouldn't satisfy risk management frameworks, plus some more minor downsides of coupling the root with the lifecycle of a given region. This root would need to outlast all MCs' lifecycles. I haven't looked into the details of moving key material between regions, but it would in any case be a risky operation. In a WC we could at least heavily restrict access and move the cluster between management clusters in the future, but I agree it could be that something else is actually needed here.

[puja108]

as I mentioned in https://github.com/giantswarm/rfc/pull/144/changes#r3073434770 the problem of running secure/isolated workloads in an MC(-like) place is becoming more and more important and might deserve a special session in SIG arch

[pipo02mix]

Do we need to consider the effect of having hundreds of pods renewing this token on the API?

[stone-z]

SPIRE is used behind the scenes in some very large systems, but it's definitely something to be aware of. Workloads themselves don't request certs; the SPIRE agents do, so as long as the agents are well-behaved, we shouldn't have issues with e.g. crashlooping pods. However, that is one reason why it's useful to have WC-scoped nested SPIREs.

[pipo02mix]

Should we take that decision, or can each customer use a different approach?

[QuentinBisson]

If customers follow different approach then this will make it really hard to have shared/global policies. Also I think this is about Giant Swarm SVIDs and not customers :)

[stone-z]

Right, this question is primarily about how to design our own SVIDs. We should have a recommendation, but the decision is ultimately up to each customer (and, if we are re-using a SPIRE root that already exists, the decision was already made). It would be nice if every customer chose the same format so we could more easily automate default policies, but I don't think we can assume that will be the case.

[teemow]

This overall looks well designed with a lot of considerations. I haven't had the time to look into this deeper and I don't know if I have the time in the near future. I only skimmed over it. This is not because it is not valuable for giantswarm and not because I do not respect all your work here. This is a complex topic. I'll not block this. Please go ahead if things have been discussed enough from your pov.

I do like a system that is independent of cloud providers and it would fit Giantswarm very well but I also think that the solutions coming from the cloud providers are the ones that most customers want to use. So in an ideal scenario we are cloud provider native as much as possible. We had similar solutions in the past and this always leads to a lot of discussions, more complexity for us and the customer and often less functionality or value as the open source solutions lack integration with the cloud provider compared to the providers solution.

Eg what would be the benefit for a customer to use spiffe/spire if they can simply use the workload identity feature in EKS? Can we convince customers that our spiffe/spire solution is worth the extra complexity and effort?

[teemow]

Btw I am happy to connect you with amutable as they are working on the link between tpm, the OS and workload identities to make the whole chain verifiable. I anyway promised them that we'll give feedback as soon as they have things in place.

[QuentinBisson]

That's a good question. We do need workload identity for us but can our spire connect with whatever customers want to us instead of the root spire? If yes, maybe we should focus on our implémentation and provide a customer spire when needed? This is must have onprem though

[puja108]

this would currently mean with the current proposal, setting it up to trust each customer separately, right? which is not super bad IMHO

[stone-z]

Not specific to (G), but in general we would need one trust federation between our SPIRE and each customer, and any services without GS or customer identities would need to be configured to trust each SPIRE (us and each customer) individually.

[puja108]

I think the use case is anyway rare and if at all rather limited to we have "3 customers with Jenkins"

[puja108]

hybrid MCs are not really limited to a single region like with manufacturing customers where WCs are in other infras, but should not be a blocker AFAICS

[stone-z]

That's true. I don't think SPIRE makes the failure domain any worse in hybrid cases; they already have a dependency on the MC, and WC-local SPIREs would keep things running in temporary failure cases. We just might need to treat the WCs as semi air-gapped if customers have "frequently offline" WCs.

[puja108]

yes, that can and is already sometimes the case in some of our customers, so I guess that's fine

[puja108]

this is only exposed on actual usage, right? or can it be queried from outside?

[stone-z]

It is exposed when an application presents its own identity. There isn't an index or anything to look up all possible identities, as far as a I'm aware (that would be strange and concerning).

[puja108]

then I think this might be fine, customers that have a high privacy requirement here, would need to use some kind of code naming scheme in place, but I think that's overthinking it

[puja108]

but only limited to the MCs intermediary and the things it signs, right? as we kinda consider the MC to "belong to the customer" we do accept that as long as it is possible to ensure limited blast radius and permissions from MC components, which from how I understand Workload identity would be inherently possible, with the caveat of it actually needing to be somehow enforced/implemented.

[stone-z]

See also the note about Named Paths / Name Constraints. It isn't currently possible to limit the scope of an intermediate signing key to e.g. "only workloads under this MC". The signing key could be used to sign a certificate for any workload in the customer's fleet, even if it's in another cluster. This will change "someday" but it's the reality for now. So the nested SPIRE is in practice a mini root. For that reason, we definitely can't put our own there, and I guess customers also shouldn't, but this should be refined and will be also up to their risk tolerance.

[stone-z]

That's actually a really good point. Until Name Constraints are possible, we probably should have isolated root and nested SPIREs for both us and customers 🤔 that's potentially a lot of extra infrastructure.

[puja108]

hmm, right, and if it's only to limit blast radius

[stone-z]

Eg what would be the benefit for a customer to use spiffe/spire if they can simply use the workload identity feature in EKS? Can we convince customers that our spiffe/spire solution is worth the extra complexity and effort?

That's a good question. We do need workload identity for us but can our spire connect with whatever customers want to us instead of the root spire? If yes, maybe we should focus on our implémentation and provide a customer spire when needed? This is must have onprem though

One of the named alternatives is orchestrating the provider-native mechanisms instead. That could work too, it just requires far more work to productize. With SPIRE, we'd need to build very little in order to automatically issue every customer workload an identity, with little or no involvement from them, and that identity could be used across both our fleet and theirs. It also means auth itself is provider-independent; IRSA wouldn't be natively usable without extra work for authenticating with resources in other clouds, SaaS systems, or on-prem resources in hybrid setups. Orchestrating the provider-specific mechanisms would require us to build more integration, and we'd still need to run SPIRE for ourselves and for on-prem customers (most likely).

It isn't possible to use SPIRE to just "drive" a provider system directly, because one system has to control the signing keys. I haven't looked to see if it's possible to issue an intermediate signing cert from e.g. IRSA and then give that to SPIRE, but that wouldn't be using the provider tools anymore anyway (and I doubt it's possible).

All that said, customers could still choose to use an alternative on top of the identities we issue, this wouldn't limit them. They could use both in parallel, or for different purposes.

[puja108]

I'd like to gather use cases where per cluster/customer/region would make sense. have we asked customers about use cases? If we do not see any it's fine to omit I guess.

Question would also be if it would be possible to add later on if needed, or at least once a new customer would have a use case.

[puja108]

I'd prefer to keep it simple and always use the GS ID for GS components but I know that would increase off-boarding efforts. Question is how high those efforts would be?

[puja108]

This might be becoming a general issue, where we have things we might not want to expose to customers but would be on MCs. I'm thinking the future of MCs might need to be multi-cluster, which at least in the case of cloud would be fine as long as we use sth like EKS, but in case of onprem it would be too much overhead most probably. Or we'd need to look into other compartmentalization techniques. Or we do it like we used to do with Vault, where we had it running next to the MC in some VM IIRC. Might be a topic to discuss in SIG arch

[puja108]

With current customers we have at least the case where WCs might be offline for a few days.

[puja108]

I kinda like to keep per customer addition of auth for external use cases, it makes any addition or removal of a customer's app a conscious decision

[puja108]

for x-region we might also need to think about how much traffic it would incur

[puja108]

This would also be super hard to get through with all customers. There will always be customers blocking the use, especially if it is connected to the US, now with all the sovereignty hype going on

[lyind]

I read https://spiffe.io/docs/latest/spiffe-specs/spiffe-id/#2-spiffe-identity and didn't find any mention of the query URI part, in the SPIFFE docs.

Usage of the query part could allow attaching unordered attributes, as key-value pairs.

For example, would spiffe://giantswarm.io/v1/giantswarm/platform/klaus/v1-2-3?wc=operations&mc=gazelle&team=planeteers be a valid ID?

[lyind]

After reading the RFC and realizing the large scope of this, I wonder:

• how well this will mesh with classic identity systems (Entra, etc.).
• what UIs are going to be used, if those already exist and how mature they are. Since managing, displaying and analyzing the trust relationships and entities could get demanding (compare classic systems).
• why it is projected to demand that much ressources (CPU/RAM, will tuning be required?): https://spiffe.io/docs/latest/planning/scaling_spire/

[piontec]

Since this is forcing us to run our own CA root, should we also reconsider our assumption, that for artifact signing we want the "keyless" mode of cosign? Our main goal so far was to avoid managing the CA, but if we change that, does it make the situation any different? Is there any kind of integration possible between cosign and spiffe, like sharing common trust root? I don't think this is changing a lot, but maybe you've seen something related.

[stone-z]

for @lyind:

how well this will mesh with classic identity systems (Entra, etc.).

It parallels them, for machine identity use cases. We shouldn't e.g. create Google users for a machine, and we won't issue SVIDs for a human. However, a system (say, some API) can trust both Google and SPIRE as identity providers in order to authenticate humans or machines. Most large SaaS and infra providers also support this federation, which means we could authorize one of our workloads bearing a SPIFFE identity to access supported systems without pre-sharing any credentials.

what UIs are going to be used, if those already exist and how mature they are

There is at least an official UI. I haven't used it yet, but that would be a starting point if you have a particular use case in mind.

why it is projected to demand that much ressources

I don't really know the exact consumption patterns, but I would assume the largest computational work is cryptographic operations, and I'd assume the largest memory demand is [de]serialization of certificate document structures. Some tuning will definitely be required as we figure out how to size each level of the tree. JWTs also require more compute than X.509s, so the consumption will vary based on usage. The Deployment Sizing Considerations section gives an overview, but the nesting must be taken into account. We will certainly have 10,000 workloads on thousands of agents, but the load associated with managing them can be distributed among multiple nested SPIRE instances.

---

and for @piontec

should we also reconsider our assumption, that for artifact signing we want the "keyless" mode

Yeah good thinking. cosign does natively support SPIFFE identities. IMO we should stick to keyless at least for now, because although 1. all members of the trust domain can implicitly authenticate each other and 2. it would be possible to sign build artifacts with a SPIRE-provided cert, we would still need to handle key distribution to the signing points and clients would still need to be able to validate the signatures. We would likely still need to build some glue and end up needing to distribute public keys ourselves.

I'd suggest we use federation to trust e.g. the GitHub or CircleCI authority for the keyless signing case, at least to start. If we were to move builds into Tekton, or if SaaS build systems introduce native integration points for SPIFFE identities, it would make sense to revisit it.