A CustomResourceDefinition is a cluster-scoped singleton keyed on group and kind — there is exactly one cert-manager.io/Certificate per cluster, full stop, and a second team's apply of a conflicting definition either fails outright or silently overwrites the first team's schema and webhook wiring. That single fact is the root of most multi-tenant controller incidents: RBAC in vanilla Kubernetes has no built-in tenant boundary either, so a ClusterRole written with a wildcard verb on a wildcard group — a pattern kubebuilder and operator-sdk will happily generate from +kubebuilder:rbac markers if a developer copies them between controllers without pruning — can read or mutate every other operator's Secrets and custom resources in the same cluster. Add in shared informer caches that watch cluster-wide by default in controller-runtime unless a project explicitly sets cache.Options.DefaultNamespaces, default leader-election IDs that two deployments of the same base operator will happily contend over, and ValidatingWebhookConfiguration names with no namespaceSelector, and a cluster running five internal operators has five different ways to step on itself before anyone writes malicious code. This piece walks through each collision point and the concrete pattern that prevents it — API group namespacing, least-privilege RBAC splitting, cache scoping, and webhook isolation — for teams running multiple custom controllers in a shared cluster.
Why do two operators fight over the same CRD?
Two operators fight over a CRD because Kubernetes identifies a CustomResourceDefinition by its group, version, and kind triplet, and only one definition can own that triplet cluster-wide at a time — there is no per-namespace or per-tenant CRD scoping, even though the instances of a custom resource can be namespaced. If Team A ships a homegrown Certificate kind under a generic group like certs.example.com and Team B later installs cert-manager, which owns cert-manager.io/Certificate, there's no collision by name alone — but two teams both reaching for an unqualified group like example.com or internal.io absolutely will collide, and the second kubectl apply either errors on schema conflict or, worse, succeeds and quietly replaces validation rules and conversion webhooks the first CRD depended on. The Kubernetes documentation's own guidance on CRD versioning treats group ownership as effectively permanent once published, because every existing custom resource in etcd is tied to it. The fix is organizational, not technical: reserve a unique, product-prefixed API group per team (platform.acmecorp.io, billing.acmecorp.io) and enforce it with a CI admission check — a simple manifest linter that rejects any CRD apply whose group isn't on an allowed prefix list — before it ever reaches a shared cluster.
Why is RBAC not enough to separate tenants?
RBAC is not enough to separate tenants because ClusterRole and ClusterRoleBinding objects are, by design, cluster-wide grants with no notion of "this applies only to namespace X's controller." A ClusterRole with verbs: ["*"] on resources: ["*"] in apiGroups: ["*"] — which is exactly what a cluster-admin-adjacent operator ends up with if a developer widens RBAC markers to unblock a permission error rather than scoping them — grants that controller's service account read and write access to every other tenant's Secrets, ConfigMaps, and custom resources in the cluster, regardless of which namespace it's deployed in. The upstream Kubernetes RBAC documentation explicitly calls out escalate and bind as separate verbs precisely because a controller with broad roles/clusterroles permissions could otherwise grant itself more access later. The least-privilege pattern: give each controller a namespaced Role/RoleBinding for anything it only needs to touch in its own namespace, and reserve ClusterRole grants strictly for list/watch on the CRDs it must reconcile cluster-wide — never get/update/patch on core resources like Secrets or Nodes unless the controller's actual job requires writing them.
How does kubebuilder generate the RBAC that causes this?
Kubebuilder and operator-sdk generate RBAC automatically from +kubebuilder:rbac comment markers placed above a controller's Reconcile method, and the controller-gen tool scans every Go file in the project to aggregate those markers into a single ClusterRole manifest — by default named something generic like manager-role, as documented in the Kubebuilder book's RBAC marker reference. This is convenient for prototyping and a common source of real over-broad grants in production: because markers are just comments, it's trivial to copy a working marker block from one controller to a new one and forget to narrow the resources and verbs fields to what the new controller actually reconciles, so a project accumulates permissions no controller in it currently uses. Kubebuilder does support generating a namespaced Role instead of a ClusterRole via a namespace parameter on the marker when a controller only operates within one namespace — a straightforward win most teams don't use because it isn't the tutorial default. Treat generated config/rbac/role.yaml as a diff to review on every PR, not a build artifact to trust, and run kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> against a real cluster before promoting a new controller.
What risk does a shared informer cache introduce?
A shared informer cache introduces an information-disclosure and denial-of-service risk because controller-runtime's default cache watches every namespace cluster-wide, meaning a controller can receive — and potentially act on — create/update/delete events for objects belonging to every other tenant, even if its RBAC never grants it write access to write those objects back. controller-runtime's own cache design docs describe the DefaultNamespaces option (the successor to the older MultiNamespacedCacheBuilder) as the mechanism to restrict a manager's informers to an explicit set of namespaces instead of watching all of them; without it, a buggy reconcile loop that misidentifies ownership can trigger a "reconcile storm" across resources it was never meant to see, burning API server and controller CPU on objects outside its actual responsibility. Cluster-scoped resources are the one exception controller-runtime call out explicitly — those are served from an unrestricted cluster cache regardless of namespace settings, so any CRD your controller defines as cluster-scoped is watched everywhere no matter what. Setting DefaultNamespaces to the specific namespaces (or namespace prefix pattern) a controller actually owns is the direct mitigation, and it should be a default in any multi-tenant controller template, not an opt-in hardening step.
How do leader-election and webhook name collisions break tenants apart?
Leader-election and admission-webhook name collisions break multi-controller clusters apart because both mechanisms rely on a name being unique cluster-wide, and controller-runtime's scaffolded defaults are not unique across two deployments of similar operators. Leader election uses a Lease (or, in older setups, a ConfigMap/Endpoints) object keyed by a leader-election-id that kubebuilder scaffolds with a generic default; two independently-built operators that never changed that default will contend for the same lease object if deployed into the same cluster, with one silently sitting idle believing another instance already holds leadership. ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects are similarly cluster-wide and unscoped by default — if operator A's webhook matches on a broad resource rule with no namespaceSelector, it will intercept and can reject admission requests for operator B's tenant namespaces too, even though it has no reconcile logic for those objects. The fix on both fronts is the same discipline as the CRD problem: prefix leader-election IDs and webhook configuration names with the product name, and set an explicit namespaceSelector (or objectSelector) on every webhook rule so it only intercepts requests in namespaces that operator actually owns.
What should a pre-deployment checklist for a new cluster tenant look like?
A pre-deployment checklist should verify, before any new controller is merged into a shared cluster, that its API group is unique and product-prefixed, its RBAC has been reviewed line-by-line against config/rbac/role.yaml rather than accepted as generated, its cache is namespace-scoped via DefaultNamespaces unless it genuinely needs cluster-wide visibility, and its leader-election ID and webhook configuration names are unique and carry a namespaceSelector. None of these checks require exotic tooling — they're a code-review discipline plus a CI admission gate that statically inspects manifests before kubectl apply reaches the shared cluster, catching the group-collision and wildcard-RBAC classes of bug that would otherwise only surface once two teams' controllers are already running side by side and one starts mutating the other's objects. Multi-tenancy add-ons like OPA Gatekeeper or Kyverno can enforce these same rules as admission policies at the cluster level — rejecting any CRD apply outside an allowed group prefix, or any ClusterRole with a wildcard verb — turning a checklist a human might skip into a control the API server enforces on every request, which matters more as the number of independently-shipped controllers in one cluster grows past the handful a team can track by memory.