Today I spent some time setting up a Kubernetes Service Connection in Azure DevOps using a custom Kubernetes Service Account and insuring least privilege is provided. It is interesting to see how Kubernetes have setup its RBAC and the detail you may need to go to set it up.
In this blog we will look at the following:
- Setting up Custom Cluster Role for least privilege
- Creating and binding a new service account to the custom cluster role
- Testing the permissions of the service account
- Adding the service account into Kubernetes Service Connection
- Referencing the Service connection in a build pipeline
Creating Custom cluster role in Kubernetes.
Kubernetes has a few default roles, to get a list of all the roles you can run
Kubectl get clusterroles
In the vast list there is one role cluster-admin, this is the most commonly used role and what people tend to use by default as it provides full admin access to the cluster. We want to reduce this and insure we provide least privilege access to the cluster. Kubernetes RBAC model does allow very granular RBAC model, with the ability to define the level of access (known as verb) for every resource type available. To get a full list of all the resources you can control have a look at the list here.
There are mainly 5 main verb types available for each resource [get list watch create update patch delete].
To see what access, you get as a cluster admin run the command
$ kubectl describe clusterrole cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
[*] [] [*]
For an Azure-DevOps account we usually do not require access such as creating new accounts, or we may want to limit creating objects such as volumes or namespaces. In our example we are going to create a custom role for Azure DevOps to only be able to create , deployments pods, services, secrets and network policies and nothing else. To do this we are going to create a new ClusterRole as per below:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: create-deployments
rules:
- apiGroups: ["*"]
resources: ["deployments","pods/*","services","secrets","networkpolicies.networking.k8s.io","pods"]
verbs: ["get","list","watch","create","update","patch","apply"]
To check out the new cluster role you have created you can run
$ kubectl describe clusterrole create-deployments
Name: create-deployments
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"create-deployments"},"rules":[{"api...
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
deployments.* [] [] [get list watch create update patch]
networkpolicies.networking.k8s.io.* [] [] [get list watch create update patch]
pods.*/* [] [] [get list watch create update patch]
pods.* [] [] [get list watch create update patch]
secrets.* [] [] [get list watch create update patch]
services.* [] [] [get list watch create update patch]
Create Service Account and binding it to a cluster role
Now we have the Cluster Role created lets create the service account and bind it to a role
$ kubectl apply -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: azure-devops-svc
EOF
serviceaccount/azure-devops-svc created
$ kubectl create clusterrolebinding azure-devops-role-binding-svc --clusterrole=create-deployments --serviceaccount=default:azure-devops-svc clusterrolebinding.rbac.authorization.k8s.io/azure-devops-role-binding-svc created
Testing the RBAC Access of the service account
Kubernetes has a feature to allow you to impersonate an account and see what level of permissions it may have, to do this we will run
kubectl auth can-i <verb> <resource>-n <namespace> --as=system:serviceaccount:<ServiceAccountName>:<Namespace>
This will return a yes or no for the action you are trying to complete.
An example of this:
$ kubectl auth can-i create pods --as=system:serviceaccount:default:azure-devops-svc
yes
$ kubectl auth can-i create pods -n montytest--as=system:serviceaccount default:azure-devops-svc
yes
$ kubectl auth can-i create namspaces --as=system:serviceaccount default:azure-devops-svc
Warning: the server doesn't have a resource type 'namspaces'
no
Creating Service Connection in Azure DevOps
Now we have the service account created and tested the permissions are as per our standard we can go ahead and create the service connection, for this logging into our Azure DevOps instance and selecting “Create Service Connection > Kubernetes Service Connection”
Selecting Service Account, we require 2 details from our cluster
1) The API Server URL
$ kubectl config view --minify -o jsonpath={.clusters[0].cluster.server}
https://montytesta-p-montytest-dev--a5c4ea-***.hcp.australiaeast.azmk8s.io:443
2) The Service Account Secret
kubectl get serviceAccounts <service-account-name> -n <namespace> -o=jsonpath={.secrets[*].name}
kubectl get secret <service-account-secret-name (Output from previous line> -n <namespace> -o json
This will create a JSON Output you will need to copy and paste it into your Azure DevOps service connection.
Save this and you are now ready to deploy your application from Azure DevOps into your K8s cluster
To reference this in your pipeline you there is a task you can create for kubectl
OR as a YAML Deployment
- task: Kubernetes@1
inputs:
connectionType: 'Kubernetes Service Connection'
kubernetesServiceEndpoint: 'montytest-k8s-cluster'
namespace: 'default'
command: 'apply'
arguments: '-f nginx-testbuild-aks.yaml'
secretType: 'dockerRegistry'
containerRegistryType: 'Azure Container Registry'
Running this in the pipeline:
