On the continued topic of Securing Kubernetes clusters today we will look at how to assign Kubernetes RBAC on Azure Kubernetes Clusters with Azure AD Objects.
By default, when you spin up an Azure Kubernetes Services cluster, to access and manage the cluster you need to assign your user the Azure IAM Role “Azure Kubernetes Service Cluster Admin”. This IAM Role essentially assigns cluster-admin role within Kubernetes giving you access to do everything any anything within the cluster. Following today’s zero trust and least privilege model this is not a good idea to be assigning our developers this level of access. Ideally you should be segregating your roles between three main groups. Depending on your organisation structure there may be more roles.
Developer Permissions to provision pods, services and manage secrets
Platform Engineer Permissions to create namespaces and manage the cluster
Monitor Permissions to read the logs and health check of running pods
To get started with our AKS Cluster we need to first enable Azure AD Integration. This is very simple to do this we update the existing AKS Cluster (or can assign to a new cluster as well) with the command --enable-aad Like below:
az aks update -g myResourceGroup -n myManagedCluster --enable-aad
What you will find once you enabled AAD integration, if you try connecting to your Kubernetes cluster now you will be prompted to authenticate against AAD:
$ az login –output none
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code D7YCDQ8YP to authenticate.
$ az aks get-credentials -n montytestaks-public -g p-montytest-dev-aue-rg --subscription ${Subscription-Id}
The behavior of this command has been altered by the following extension: aks-preview
A different object named clusterUser_p-montytest-dev-aue-rg_montytestaks-public already exists in your kubeconfig file.
Overwrite? (y/n): y
Merged "montytestaks-public" as current context in /home/admin/.kube/config
$ kubectl get nodes
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code D7MU743P to authenticate.
Error from server (Forbidden): nodes is forbidden: User "****" cannot list resource "nodes" in API group "" at the cluster scope
You will see when you authenticate in the browser you are actually authenticating to the Azure Kubernetes Service AAD Client, this is the service which is doing the AAD to K8s RBAC association.
Admin access to cluster
At this point you may be thinking oh no! I can’t get onto my cluster now. No worries, Microsoft have thought of that, provided your account is in the “Azure Kubernetes Service Cluster Admin” you can run bypass the AAD Client auth by changing your credentials and adding an --admin at the end.
az aks get-credentials --resource-group myResourceGroup --name myManagedCluster --admin
$ az aks get-credentials -n montytestaks-public -g p-montytest-dev-aue-rg --subscription ${Subscription} --admin
The behavior of this command has been altered by the following extension: aks-preview
Merged "montytestaks-public-admin" as current context in /home/admin/.kube/config
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
aks-nodepool1-27351289-vmss000000 Ready agent 252d v1.16.15
aks-nodepool1-27351289-vmss000001 Ready agent 226d v1.16.15
aks-nodepool1-27351289-vmss000002 Ready agent 226d v1.16.15
Creating your RBAC Role and Binding
Using this admin context, we are going to create a new role for developers and a role-binding to associate that role to an AD Group and also to an AD User.
To Create the role, we need to define what access we want to give the users. Kubernetes RBAC model enables very granular control. With the ability to define the level of access (known as verb) for every resource type available. To get a full list of all the resources you can control have a look at the list here.
For our Developers we want to limit their access to only manage deployments, pods, services, secrets and network policies and nothing else and only in the namespace dev.
For this our role will look like:
kubectl apply -f -<<EOF
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dev-user-deployment-access
namespace: dev
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["deployments","pods/*","services","secrets","networkpolicies.networking.k8s.io","pods"]
verbs: ["*"]
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs: ["*"]
EOF
role.rbac.authorization.k8s.io/dev-user-deployment-access created
Now to create our role binding, for this we are going to use an AAD Group, in which our developers are in. To associate the AD Group to the role-binding we need to find our Groups Object Id, using az cli we can do that easily
$ az ad group show --group AU_AZU_BusUserAdmin_app1_developer |grep -i objectId
"objectId": "93b40f18-b88d-4bc0-a90a-d539f0c67d40",
$ kubectl apply -f -<<EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: dev-user-access
namespace: dev
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: dev-user-deployment-access
subjects:
- kind: Group
name: 93b40f18-b88d-4bc0-a90a-d539f0c67d40
For this you work your group must also be assigned the Azure IAM Role “Azure Kubernetes Service Cluster User Role”
And that is it, lets login with a user in the group and test deploying a Pod in the default namespace and then in the Dev Namespace and creating a new namespace.
$ kubectl apply -f test-external.app
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "nginx", Namespace: "default"
from server for: "test-external.app": deployments.apps "nginx" is forbidden: User "***" cannot get resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=services", GroupVersionKind: "/v1, Kind=Service"
Name: "external-app", Namespace: "default"
from server for: "test-external.app": services "external-app" is forbidden: User "***" cannot get resource "services" in API group "" in the namespace "default"
$ kubectl apply -f test-external.app -n dev
deployment.apps/nginx created
service/external-app created
$ kubectl create namespace demo
Error from server (Forbidden): namespaces is forbidden: User "[email protected]" cannot create resource "namespaces" in API group "" at the cluster scope
Non-Interactive login
One key difference is non-interactive logins. If you are using pipelines or scripts to deploy resources into your cluster, you may realise that they may not work. Don’t worry there are solutions for this.
There are two options:
The first, is using Kubernetes Service Accounts. K8s service accounts are not subjected to the AAD Authentication flow hence if you are using service accounts in your pipeline it should work. For details on how to set this up check out my post on K8s Service accounts and using them in Azure DevOps here.
The Second, if you prefer to use an Azure AD SPN Account. There is a kubectl plugin is available to enable programmatic login via an SPN and continue to use your favourite kubectl commands to deploy. The plugin kubelogin , is a plugin which enables programmatic login to Kubernetes clusters that require authentication with SAML or OAUTH2.
By defining your SPN client ID and Key as well as details of your kubeconfig file in your environment variable you can authenticate to your cluster. Here is a sample of code I have created in my Azure DevOps pipeline.
- task: AzureCLI@2
inputs:
azureSubscription: 'ITS-DEV'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
curl -L https://github.com/Azure/kubelogin/releases/download/v0.0.7/kubelogin-linux-amd64.zip --output kubelogin.zip
unzip kubelogin.zip
chmod +x ./bin/linux_amd64/kubelogin
PATH=$PATH:./bin/linux_amd64
az aks get-credentials -n montytestaks-public -g p-montytest-dev-aue-rg --subscription ${subscriptionId};
export KUBECONFIG=~/.kube/config
export AAD_SERVICE_PRINCIPAL_CLIENT_ID=$servicePrincipalId
export AAD_SERVICE_PRINCIPAL_CLIENT_SECRET=$servicePrincipalKey
kubelogin convert-kubeconfig -l spn
kubectl get pods
addSpnToEnvironment: true
In my example I am utilizing the Azure CLI Task that is available this already defines my SPN details as variables I can use in my code. There are some other options such as using managed identities or storing your credentials in a KMS solution.
Also just a quick note I am using a MS hosted build server hence I am downloading and extracting kubelogin as well as defining my parameters to authenticate to my cluster.
I hope you enjoyed this blog, please be on the lookout of my next post which will be a look into Azure’s new IAM role managed RBAC.
