Creating Kubernetes Service Accounts and Azure DevOps Service Connection



Today I spent some time setting up a Kubernetes Service Connection in Azure DevOps using a custom Kubernetes  Service Account and insuring least privilege is provided. It is interesting to see how Kubernetes have setup its RBAC and the detail you may need to go to set it up.

In this blog we will look at the following:

  • Setting up Custom Cluster Role for least privilege
  • Creating and binding a new service account to the custom cluster role
  • Testing the permissions of the service account
  • Adding the service account into Kubernetes Service Connection
  • Referencing the Service connection in a build pipeline

Creating Custom cluster role in Kubernetes.

Kubernetes has a few default roles, to get a list of all the roles you can run

Kubectl get clusterroles

In the vast list there is one role cluster-admin, this is the most commonly used role and what people tend to use by default as it provides full admin access to the cluster. We want to reduce this and insure we provide least privilege access to the cluster. Kubernetes RBAC model does allow very granular RBAC model, with the ability to define the level of access (known as verb)  for every resource type available. To get a full list of all the resources you can control have a look at the list here.

There are mainly 5 main verb types available for each resource [get list watch create update patch delete].

To see what access, you get as a cluster admin run the command

$ kubectl describe clusterrole cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]

For an Azure-DevOps account we usually do not require access such as creating new accounts, or we may want to limit creating objects such as volumes or namespaces. In our example we are going to create a custom role for Azure DevOps to only be able to create , deployments pods, services, secrets and network policies and nothing else. To do this we are going to create a new ClusterRole as per below:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name:  create-deployments
rules:
 - apiGroups: ["*"]
   resources: ["deployments","pods/*","services","secrets","networkpolicies.networking.k8s.io","pods"]
   verbs: ["get","list","watch","create","update","patch","apply"]

To check out the new cluster role you have created you can run

$ kubectl describe clusterrole create-deployments
Name:         create-deployments
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"create-deployments"},"rules":[{"api...
PolicyRule:
  Resources                            Non-Resource URLs  Resource Names  Verbs
  ---------                            -----------------  --------------  -----
  deployments.*                        []                 []              [get list watch create update patch]
  networkpolicies.networking.k8s.io.*  []                 []              [get list watch create update patch]
  pods.*/*                             []                 []              [get list watch create update patch]
  pods.*                               []                 []              [get list watch create update patch]
  secrets.*                            []                 []              [get list watch create update patch]
  services.*                           []                 []              [get list watch create update patch]

Create Service Account and binding it to a cluster role

Now we have the Cluster Role created lets create the service account and bind it to a role

$ kubectl apply -f -<<EOF
 apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: azure-devops-svc
EOF
serviceaccount/azure-devops-svc created

$ kubectl create clusterrolebinding azure-devops-role-binding-svc --clusterrole=create-deployments --serviceaccount=default:azure-devops-svc clusterrolebinding.rbac.authorization.k8s.io/azure-devops-role-binding-svc created

Testing the RBAC Access of the service account

Kubernetes has a feature to allow you to impersonate an account and see what level of permissions it may have, to do this we will run

kubectl auth can-i <verb> <resource>-n <namespace> --as=system:serviceaccount:<ServiceAccountName>:<Namespace>

This will return a yes or no for the action you are trying to complete.

An example of this:

$ kubectl auth can-i create pods --as=system:serviceaccount:default:azure-devops-svc
yes
$ kubectl auth can-i create pods -n montytest--as=system:serviceaccount default:azure-devops-svc
yes
$ kubectl auth can-i create namspaces --as=system:serviceaccount default:azure-devops-svc
Warning: the server doesn't have a resource type 'namspaces'
no

Creating Service Connection in Azure DevOps

Now we have the service account created and tested the permissions are as per our standard we can go ahead and create the service connection, for this logging into our Azure DevOps instance and selecting “Create Service Connection > Kubernetes Service Connection”

Selecting Service Account, we require 2 details from our cluster

1) The API Server URL

$ kubectl config view --minify -o jsonpath={.clusters[0].cluster.server}
https://montytesta-p-montytest-dev--a5c4ea-***.hcp.australiaeast.azmk8s.io:443

2) The Service Account Secret

kubectl get serviceAccounts <service-account-name> -n <namespace> -o=jsonpath={.secrets[*].name}

kubectl get secret <service-account-secret-name (Output from previous line> -n <namespace> -o json

This will create a JSON Output you will need to copy and paste it into your Azure DevOps service connection.

Save this and you are now ready to deploy your application from Azure DevOps into your K8s cluster

To reference this in your pipeline you there is a task you can create for kubectl

OR as a YAML Deployment

- task: Kubernetes@1
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: 'montytest-k8s-cluster'
    namespace: 'default'
    command: 'apply'
    arguments: '-f nginx-testbuild-aks.yaml'
    secretType: 'dockerRegistry'
    containerRegistryType: 'Azure Container Registry'

Running this in the pipeline: