Kubernetes Part 3: Deploying Kubernetes Services with AWS Secrets Store CSI Driver using Helm

August 15, 2024

by Thomas Tran

In this latest installment of my Kubernetes series, we’ll use Helm, a handy tool that makes deploying resources in Kubernetes easier and faster. We’ll start by deploying a basic NGINX service, a popular and straightforward choice for getting started with Kubernetes clusters. This example will guide you through using Helm to quickly set up a working service in your Kubernetes environment. Additionally, we’ll deploy the AWS Secrets Store CSI Driver, which allows your Kubernetes pods to securely retrieve secrets from AWS, adding an extra layer of functionality to your deployment.

Prerequisites

AWS CLI installed and configured
kubectl installed and configured
helm installed
eksctl installed (optional but recommended)

1. Create a Helm Chart for Nginx

1.1 Initialize a New Helm Chart

helm create nginx-test
cd nginx-test

1.2 Update `values.yaml`

Update values.yaml to include the necessary configuration:

# Default values for nginx-test.
replicaCount: 1

image:
  repository: nginx
  pullPolicy: IfNotPresent
  tag: "latest"

serviceAccount:
  create: true
  name: nginx-sa
  automount: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/your-iam-role-name

podAnnotations: {}
podLabels: {}

podSecurityContext: {}

securityContext: {}

service:
  type: ClusterIP
  port: 80

resources: {}

livenessProbe:
  httpGet:
    path: /
    port: http

readinessProbe:
  httpGet:
    path: /
    port: http

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80

volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: nginx-secret-provider-class

volumeMounts:
  - name: secrets-store-inline
    mountPath: /mnt/secrets-store
    readOnly: true

nodeSelector: {}

tolerations: []

affinity: {}

1.3 Update `templates/deployment.yaml`

Ensure deployment.yaml is configured correctly:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "nginx-test.fullname" . }}
  labels:
    {{- include "nginx-test.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "nginx-test.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "nginx-test.labels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ .Values.serviceAccount.name }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            {{- toYaml .Values.livenessProbe | nindent 12 }}
          readinessProbe:
            {{- toYaml .Values.readinessProbe | nindent 12 }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
            {{- toYaml .Values.volumeMounts | nindent 12 }}
      volumes:
        {{- toYaml .Values.volumes | nindent 8 }}

1.4 Update `templates/serviceaccount.yaml`

Ensure serviceaccount.yaml is configured correctly:

{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.serviceAccount.name }}
  labels:
    {{- include "nginx-test.labels" . | nindent 4 }}
  {{- with .Values.serviceAccount.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
{{- end }}

2. Deploy AWS Secrets Store CSI Driver

2.1 Add the Helm Repository

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update

2.2 Install the CSI Driver

helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

3. Create an AWS IAM Role and Policy

3.1 Create an IAM Policy

Create an IAM policy that grants permissions to access AWS Secrets Manager. Save the following JSON as nginx-sa-policy.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue"
            ],
            "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:SECRET_NAME"
        }
    ]
}

Replace REGION, ACCOUNT_ID, and SECRET_NAME with your specific values.

3.2 Create the IAM Policy in AWS

Use the AWS CLI to create the policy:

aws iam create-policy --policy-name nginx-sa-policy --policy-document file://nginx-sa-policy.json

3.3 Create an IAM Role with Trust Relationship

Create a trust relationship JSON file, trust-relationship.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_ID"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_ID:sub": "system:serviceaccount:default:nginx-sa"
                }
            }
        }
    ]
}

Replace ACCOUNT_ID, REGION, and EKS_CLUSTER_ID with your specific values.

3.4 Create the IAM Role

Use the AWS CLI to create the role:

aws iam create-role --role-name nginx-sa-role --assume-role-policy-document file://trust-relationship.json

3.5 Attach the Policy to the Role

Attach the previously created policy to the IAM role:

aws iam attach-role-policy --role-name nginx-sa-role --policy-arn arn:aws:iam::ACCOUNT_ID:policy/nginx-sa-policy

Replace ACCOUNT_ID with your specific value.

4. Annotate the Service Account

Ensure the service account nginx-sa in the default namespace is annotated with the IAM role ARN.

4.1 Annotate the Service Account

kubectl annotate serviceaccount nginx-sa \
  eks.amazonaws.com/role-arn=arn:aws:iam::820753594122:role/nginx-sa-role \
  --namespace default

5. Deploy the Helm Chart

5.1 Deploy the Helm Chart

Use the helm upgrade --install command to deploy the Helm chart:

helm upgrade --install nginx-test ./nginx-test --namespace default

6. Verify the Deployment

6.1 Check the Pods

Ensure that the pods are running:

kubectl get pods --namespace default

6.2 Describe a Pod

Get detailed information about a specific pod:

kubectl describe pod <pod-name> --namespace default

Replace <pod-name> with the name of your pod.

6.3 Exec into the Pod

Exec into the pod to verify that it can access the secret:

kubectl exec -it <pod-name> --namespace default -- /bin/sh

6.4 Navigate to the Mounted Directory

Navigate to the directory where the secrets are mounted:

cd /mnt/secrets-store

6.5 List the Contents

List the contents of the directory:

ls

6.6 View the Secret

Use cat to view the contents of the secret file:

cat my-secret-key

Conclusion

In this tutorial, we successfully deployed a basic NGINX service using Helm, demonstrating how this powerful tool can simplify and speed up the process of managing Kubernetes resources. We also took an important step in enhancing our Kubernetes environment by deploying the AWS Secrets Store CSI Driver, enabling secure access to AWS secrets directly from our Kubernetes pods.

By following these steps, you’ve gained hands-on experience with Helm and learned how to set up a foundational service in Kubernetes. You’ve also integrated essential security practices by managing secrets securely within your cluster. These skills will serve as a strong foundation as you continue to explore and deploy more complex applications in Kubernetes. Whether you’re just getting started or looking to refine your deployment process, this tutorial has provided you with the tools and knowledge to confidently manage your Kubernetes environment.