Karpenter is an open-source node provisioning project built for Kubernetes. Adding Karpenter to a Kubernetes cluster can dramatically improve the efficiency and cost of running workloads on that cluster. Karpenter works by:
- Watching for pods that the Kubernetes scheduler has marked as unschedulable
- Evaluating scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods
- Provisioning nodes that meet the requirements of the pods
- Removing the nodes when the nodes are no longer needed
In the scope of this post, i will guide you to install Karpenter on EKS Cluster, create a deployment to see how Karpenter spin up/down node to satisfy deployment resource.
Let assume that we already have a EKS cluster with OIDC enabled.
We use IRSA to provide some required permission for karpenter to create/remove node to/from cluster. Beside that each node will have an instance profile attached to them, we did that by creating an IAM role and an instance profile associated with this role.
Create Karpenter Node Role/Instance Profile
Karpenter spin up a node that have an instance profile associate with this. The node need to have enought permission to pull image also joining EKS cluster.
The role permission need to include the aws-managed policy
The role after creating is look like this.
So we had a role
arn:aws:iam::020213277421:role/copper-karpenter-node-role and an instance profile
Grant Karpenter Node Role to aws-auth Config Map
To make the karpenter node role has permission to join eks cluster, we need to update
aws-auth config map in
To update this i will use
kubectl edit command.
kubectl edit cm/aws-auth -n kube-system
From the above image, you can see that i added karpenter node role to
mapRoles array. After doing this step, we have done with
karpenter node role.
Create IRSA for Karpenter
Karpenter will be deployed into
karpenter namespace, the controller will have a service acccount name
karpenter. So with the irsa we need to set a
custom policy and
trust relationship like this. Remember to change your
The irsa role need to describe cluster and also need to have permission to create/delete EC2 instance, beside that this need to pass
Karpenter Node Role to EC2 instance.
For me after creating
IRSA i got irsa's arn equal to
Install Karpenter With Helm
Karpenter Node Role/Instance Profile so far. Time to install karpenter using helm with the script below. Remember to change your cluster name also the role arn, profile name.
After running this script. I successfully deploy Karpenter.
Verify the helm release and karpenter pods.
Deploy Karpenter Provisioner/Node Template.
When you first installed Karpenter, you set up a default Provisioner. The Provisioner sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes.
For me i will setup a
default provisoner like this.
And a default
NodeTemplate. For the karpenter node to be deployed, they will be deployed to
Private Subnet and for
ENI they will have a Security Group attached. This security is
additional security group that you crecated when creating cluster.
Deploy Test Deployment Resource
Currently i have 3 node on cluster.
It's fine to deploy a deployment with one pod like this.
After creating this deployment, the cluster already have enough resource to lauch this pod. But when we update the
replicate number to 10, there is no node to allocate pod.
Now, scale up replica to 10.
Let watch the node status
We can see that in just some seconds, the new node comming up and cluster placed the new pod.
Remove Deployment And See Karpenter Remove Node.
Let delete the deployment resource.
Watch the node status
Karpenter auto remove the node that did not placed any pod after some seconds.
As we can see, we can now scale workload just-in-time second. Karpenter can do more that that, you can visit the official documentation to learn more.
Thank for reading.