Manage multi-region EKS the Declarative Way - Part 3: Dev & Prod Clusters

Now that we have our central cluster from the previous article, we can start deploying our dev and prod clusters.

All the code for this article can be found on jacobhjkim/declarative-eks-tutorial.

Table of Contents

• Dev AWS Infra Setup
  • Dev VPC
  • Dev EKS
  • Dev EKS Addons
• Dev Helm Chart Installation
• Prod Cluster Deployment

move to section dev-aws-infra-setupDev AWS Infra Setup

We are going to use Terraform to create AWS infrastructure for the dev cluster. We will use Atlantis to apply Terraform changes with pull requests.

Dev VPC

Like we did for the central VPC, change directory to terraform/dev/vpc and update Terraform files as you did before. Unlike the central VPC, instead of using terraform apply command, we are going to use Atlantis to apply the Terraform changes. Create a pull request with the changes. Atlantis will create a plan and post it to the pull request. You can review the plan and approve the pull request. Atlantis will then apply the changes.

Once you and your teammates approve the pull request, comment atlantis apply to Atlantis. Atlantis will then apply the changes.

It can take a few minutes for the VPC to be created. Once the VPC is created, Atlantis will post a comment to the pull request. You can then merge the pull request.

You just deployed a VPC with Atlantis! How was it? I think Atlantis is a great tool that allows DevOps teams to automate Terraform workflow with pull requests.

Dev EKS

Hopefully, you are now familiar with the process of creating AWS infrastructure with Atlantis. Updateterraform/dev/eks Terraform files with appropriate values and create a pull request.

If the plan seems valid and you are ready to apply, comment atlantis apply to Atlantis. After a few minutes, you should be able to see a new EKS cluster in your AWS console.

Once you have created the EKS cluster, update the kubectl context with the following command:

$ aws eks --region ap-northeast-2 update-kubeconfig --name dev

$ kubectl config get-contexts
CURRENT   NAME                                                      CLUSTER                                                   AUTHINFO                                                  NAMESPACE
          arn:aws:eks:ap-northeast-2:765355018960:cluster/central   arn:aws:eks:ap-northeast-2:765355018960:cluster/central   arn:aws:eks:ap-northeast-2:765355018960:cluster/central
*         arn:aws:eks:ap-northeast-2:765355018960:cluster/dev       arn:aws:eks:ap-northeast-2:765355018960:cluster/dev       arn:aws:eks:ap-northeast-2:765355018960:cluster/dev

$ kubectl config use-context arn:aws:eks:ap-northeast-2:765355018960:cluster/dev
Switched to context "arn:aws:eks:ap-northeast-2:765355018960:cluster/dev".

$ kubectl get nodes
NAME                                             STATUS   ROLES    AGE     VERSION
ip-10-98-4-110.ap-northeast-2.compute.internal   Ready    <none>   4m14s   v1.23.9-eks-ba74326

Dev EKS Addons

EKS addons allow the dev cluster to access Vault and Argo CD in the central cluster. Update terraform/dev/eksfiles with appropriate values and create a pull request. Run atlantis plan and atlantis apply. Once everything is applied, check Vault to see if Kubernetes authentication is successfully created. There will be a secret created under cluster/dev/argocd

This allows central Argo CD to deploy applications to the dev cluster. But there is just one thing we need to do before deploying stuff to the dev cluster with Argo CD. Update kubernetes/charts/argocd/values-central.yaml like below:

1- clusters: []
2+ clusters: 
3+   - name: dev

Since Argo CD is managed via Helm, deploy the changes with the following command:

- clusters: []
+ clusters: 
+   - name: dev

You can check Argo CD UI to check whether the dev cluster is configured correctly.

move to section dev-helm-chart-installationDev Helm Chart Installation

Since we are familiar with Argo CD GitOps workflow, let’s deploy all the Kubernetes applications at once to the dev cluster. Update the following:

• charts/karpenter/values-dev.yaml
• charts/aws-ebs-csi-driver/values-dev.yaml
• charts/aws-load-balancer-controller/values-dev.yaml
• charts/external-dns/values-dev.yaml
• charts/echo-server/values-dev.yaml
• charts/external-secrets-operator/values-dev.yaml

Lastly, update charts/argocd-gitops/values-dev.yaml with appropriate values. Commit and push, and Argo CD will deploy all the applications to the dev cluster. Your charts/argocd-gitops/values-dev should something like this.

clusterName: dev
clusterRegion: ap-northeast-2
infraAccountId: "765355018960"
rootNamespace: argocd

stage: dev
defaultRepoURL: https://github.com/jacobhjkim/declarative-eks-tutorial
defaultTargetRevision: main
defaultProject: dev-apps
destinationServer: https://396881089D4D5CF595840E32C7FBF4E3.yl4.ap-northeast-2.eks.amazonaws.com

apps:
  - name: aws-ebs-csi-driver
    namespace: kube-system
    source:
      path: kubernetes/charts/aws-ebs-csi-driver
    valueFiles:
      - values-dev.yaml
  - name: aws-load-balancer-controller
    namespace: kube-system
    source:
      path: kubernetes/charts/aws-load-balancer-controller
    valueFiles:
      - values-dev.yaml
  - name: cert-manager
    namespace: kube-system
    source:
      path: kubernetes/charts/cert-manager
    valueFiles:
      - values-dev.yaml
  - name: echo-server
    namespace: default
    source:
      path: kubernetes/charts/echo-server
    valueFiles:
      - values-dev.yaml
  - name: external-dns
    namespace: kube-system
    source:
      path: kubernetes/charts/external-dns
    valueFiles:
      - values-dev.yaml
  - name: external-secrets-operator
    namespace: external-secrets-operator
    source:
      path: kubernetes/charts/external-secrets-operator
    valueFiles:
      - values-dev.yaml
  - name: karpenter
    namespace: karpenter
    source:
      path: kubernetes/charts/karpenter
    valueFiles:
      - values-dev.yaml

Check argocd.[YOUR_DOMAIN_NAME]/applications/argocd-dev-apps to see Argo CD deploying all the applications at once.

move to section the-power-of-gitops-with-argo-cdThe Power of GitOps with Argo CD

How was deploying the dev cluster compared to the central cluster? It was much easier, right? Once you have setup the GitOps pipeline with Argo CD, deploying another cluster is just a matter of updating Terraform files and Helm Chart values.

move to section prod-cluster-deploymentProd Cluster Deployment

Deploying the prod cluster is similar to the dev cluster. The differences are the following:

• Cluster Name
• Region (ap-northeast-2 -> us-east-1)
• Access Control

Since deploying the prod cluster is just a matter of updating the Terraform files and Helm Chart values, prod cluster deployment is left as an exercise for the reader. 😉

Other Declarative Multi-region EKS Series :

1. Manage multi-region EKS the Declarative Way - Part 0: Introduction
2. Manage multi-region EKS the Declarative Way - Part 1: Central Infrastructure
3. Manage multi-region EKS the Declarative Way - Part 2: Central Helm
4. Manage multi-region EKS the Declarative Way - Part 4: CI/CD