In our previous article on the topic, we discussed the challenges of implementing Dynamic Admission Control (DAC) in your Kubernetes cluster using Validating Admission Webhooks (VAW). We introduced the new Validation Admission Policy (VAP) resource and demonstrated a few simple use cases to its simplicity.
Today we’re going to focus a couple of more advanced features of VAP:
- Resource matching and filtering
Parameters in policies
Lastly, we will shortly discuss considerations for using VAPs over VAWs and some tips for migration.
All examples in the previous article and this one, can be found in a GitHub repository created for this purpose.
Resource matching and filtering
In the use case demonstrated in the previous article, we saw how we can apply a policy for only one type of resource (
Deployment), which makes sense for resource-specific validations.
But what if we wanted to apply a more generic policy in the cluster, such as enforcing a naming convention?
We have several options here. First, we can go with the explicit option and create a resource matching rule for each type of resource we would like to validate:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicy metadata: name: "demo-policy.example.com" spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments", "statefulsets"] - apiGroups: ["autoscaling"] apiVersions: ["v2"] operations: ["CREATE", "UPDATE"] resources: ["horizontalpodautoscalers"] validations: - expression: "!object.metadata.name.contains('demo') || object.metadata.namespace == 'demo'"
Sure, you can use the same rule for resources that are part of the same API group/version, but this list can grow very long very fast and will need to be maintained (especially when introducing new resource types into the cluster).
The alternative would be to match all resources and then selectively exclude those that we don’t want to validate. Resource filtering can be done on two levels:
- inside the
ValidatingAdmissionPolicyresource — using
- inside the
ValidatingAdmissionPolicyBindingresource — using
The first approach is documented, whereas the second one not so much.
.spec.matchConditions uses Common Expression Langauge (CEL) to filter requests that the rules have already matched (i.e. past the binding and the policy’s match constraints).
(.spec.matchResources.excludeResources)on the other hand, are performed before the resource even reaches the policy and use the same syntax used to match resources to create a resource exclusion list.
Here’s an example of a policy that will skip
Lease resources from the API group
coordination.k8s.io and any resource from the API group
rbac.authorization.k8s.io using CEL match conditions:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicy metadata: name: "demo-policy.example.com" spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: ["*"] apiVersions: ["*"] operations: ["CREATE", "UPDATE"] resources: ["*"] matchConditions: - name: 'exclude-leases' # Each match condition must have a unique name expression: '!(request.resource.group == "coordination.k8s.io" && request.resource.resource == "leases")' # Match non-lease resources. - name: 'rbac' # Skip RBAC requests. expression: 'request.resource.group != "rbac.authorization.k8s.io"' validations: - expression: "!object.metadata.name.contains('demo') || object.metadata.namespace == 'demo'"
And here’s a binding that will filter out similar requests using match exclusions:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicyBinding metadata: name: "demo-binding-test.example.com" spec: policyName: "demo-policy.example.com" validationActions: [Deny] matchResources: excludeResourceRules: # exclude all resources in the rbac.authorization.k8s.io API group - apiGroups: ["rbac.authorization.k8s.io"] apiVersions: ["*"] operations: ["*"] resources: ["*"] # exclude resource type leases from the coordination.k8s.io API group - apiGroups: ["coordination.k8s.io"] apiVersions: ["*"] operations: ["*"] resources: ["leases"] namespaceSelector: matchLabels: environment: demo
Since both achieve virtually the same, I would suggest going with
excludeResourceRules if possible, since the filtering happens earlier on and will avoid running CEL expressions for no reason (we discussed the cost budget of running CEL within the Kubernetes API server in the previous post). Additionally, the fact it uses the same syntax as matching resources, I believe that it’s a bit more clear and easier to understand.
matchConditions uses CEL expressions for filtering and has access to the same objects as the validation expressions (
request , and
params ), it also means that it has the power to filter for things that are not available with
excludeResourceRules , such as users/groups, or any other attribute on the available objects:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicy metadata: name: "demo-policy.example.com" spec: failurePolicy: Fail # match _all_ resources (for CREATE and UPDATE operations) matchConstraints: resourceRules: - apiGroups: ["*"] apiVersions: ["*"] operations: ["CREATE", "UPDATE"] resources: ["*"] matchConditions: - name: 'exclude-kubelet-requests' expression: '!("system:nodes" in request.userInfo.groups)' # Allow requests made by non-node users. validations: - expression: 'object.metadata.name.startsWith("demo-")'
In short, I would suggest a combined approach: use
excludeResourceRules when you’d like to filter out specific resources (by API group, version, operation, resource type) and use
matchConditions when you need more advanced filtering (requests from specific users, groups, etc).
To make policies even more powerful, you can parameterize them. This allows you to reuse a policy with different parameter values without having to redefine it for each permutation. Essentially separating the configuration from the policy itself.
Using parameters requires the cluster operator to create a
CustomResourceDefinition (CRD) to define and hold the parameters. This means that using parameters requires some extra work — but you don’t have to use them at all, so it’s up to you to decide.
Here’s a simple example of CRD that can be easily modified or extended if necessary:
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: parameters.example.com spec: group: example.com names: kind: Parameters plural: parameters versions: - name: v1 schema: openAPIV3Schema: type: object properties: maxReplicas: type: integer served: true storage: true scope: Cluster
The above CRD will allow us to create resources of the type
Parameters with a single supported custom field in the schema,
maxReplicas of type integer:
Once we have that, we configure our policy to take parameters (
.spec.paramKind ) by pointing it to the correct API version and kind (as defined in our CRD) and can use them in our validation expressions:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicy metadata: name: "demo-policy.example.com" spec: failurePolicy: Fail paramKind: apiVersion: example.com/v1 kind: Parameters matchConstraints: resourceRules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments"] validations: - expression: "object.spec.replicas <= params.maxReplicas" messageExpression: "'Deployments cannot have more than ' + string(params.maxReplicas) + ' replicas, this one has ' + string(object.spec.replicas)"
Note that the
messageExpression is also updated to use the parameter, allowing us to avoid hard-coding values anywhere in the policy.
Eventually, we’ll update the binding to reference the
Parameters resource we created by name:
apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicyBinding metadata: name: "demo-binding-test.example.com" spec: policyName: "demo-policy.example.com" validationActions: [Deny] paramRef: name: "demo-parameters" matchResources: namespaceSelector: matchLabels: environment: demo
Once we applied everything to our cluster, trying to create a deployment with more than the configured replicas will fail as expected:
$ k create deployment nginx — image=nginx — replicas=6 error: failed to create deployment: deployments.apps "nginx" is forbidden: ValidatingAdmissionPolicy 'demo-policy.example.com' with binding 'demo-binding-test.example.com' denied request: Deployments cannot have more than 3 replicas, this one has 6
This would allow us to create different
Parameters resources and bind them to different resources/namespaces so that we can reuse our parameterized policies.
VAW or VAP, which to choose
Since VAPs are so much simpler to implement than VAWs, it seems likely that it will eventually replace VAWs, or at least become the standard for starting up with in-cluster validation in Kubernetes.
CEL support in Kubernetes has opened many doors for potential quality of life improvements from a cluster operator perspective. A Mutating Admission Policy is another example for a feature that might become a reality thanks to CEL. It’s safe to assume that VAPs will continue to receive focus and development until reaching maturity and general availability.
That said, VAPs are still not a full-fledged replacement to VAWs. With VAWs you can still perform any kind of custom validation logic that you can put into code, whether it’s a simple request attribute validation, or even checking the state of an external system before admitting a request. Moreover, the fact that you have the write the validation server on your own, gives you the freedom to choose your own technology and language — you are not limited to using CEL for validation.
Also, considering that VAPs are still in Alpha (starting with v1.26), it’s not yet a valid alternative for anyone using managed Kubernetes offerings from the major clouds. EKS and AKS do not support alpha features, whereas GKE does but only for testing purposes (alpha clusters expire after 30 days). That means that even if you could, you probably shouldn’t fully adopt VAPs as your organization’s only mechanism of in-cluster validation just yet.
Tips for migrating from VAWs to VAPs
Are you reading this at a time where VAPs are already at beta or GA and considering using it instead of your current solution? Luckily, these 2 features are not mutually exclusive. This means you could run them side by side during the transitional period.
Since VAWs and VAPs match resources in a similar fashion, the process of creating VAPs should be fairly straightforward. Here are some tips to get you started:
- Create a
validatingAdmissionPolicythat has the same
.spec.matchConstraints.resourceRulesas the .webhooks.rules of your
- Migrate your validation expressions from the current langauge (likely Rego) to CEL. This is by far the most time consuming task, but I would suggest trying to use ChatGPT for this purpose, policy rules are usually straightforward enough that it can do a pretty decent job of it.
- Create a
.spec.validationActionsto Warn to return errors to the client, but not block any API requests.
- Decide how to bind a policy to resources using
.spec.matchResources. This can be a simple namespace label matcher, or using a more advanced match expression. Label your namespaces accordingly.
- Attempt to create valid and invalid resources in managed namespaces, evaluate any returned (or missing) warnings and refine your expressions throughout the process.
- Once you’re confident in your policy, change the binding
Denyto put the policy into full effect
- Remove any duplicate policies from the legacy system
Effortless In-Cluster Validation with Kubernetes: Introducing Validating Admission Policies
validating-admission-policy-playground GitHub repository containing full working examples for Validating Admission Policies
Validating Admission Policy in the official Kubernetes documentation
Common Expression Language in Kubernetes
Kubernetes Enhancement Proposal (KEP)-3488 CEL for Admission Control
The Path to Self Contained CRDs talk by Cici Huang