Manage multi-region EKS the Declarative Way
- Part 3: Dev & Prod Clusters

Now that we have our central cluster from the previous article, we can start deploying our dev and prod clusters.

All the code for this article can be found on jacobhjkim/declarative-eks-tutorial.

Table of Contents

move to section dev-aws-infra-setupDev AWS Infra Setup

We are going to use Terraform to create AWS infrastructure for the dev cluster. We will use Atlantis to apply Terraform changes with pull requests.


Like we did for the central VPC, change directory to terraform/dev/vpc and update Terraform files as you did before. Unlike the central VPC, instead of using terraform apply command, we are going to use Atlantis to apply the Terraform changes. Create a pull request with the changes. Atlantis will create a plan and post it to the pull request. You can review the plan and approve the pull request. Atlantis will then apply the changes.

A screenshot of the Atlantis planA screenshot of the Atlantis plan

Once you and your teammates approve the pull request, comment atlantis apply to Atlantis. Atlantis will then apply the changes.

A screenshot of the Atlantis apply

It can take a few minutes for the VPC to be created. Once the VPC is created, Atlantis will post a comment to the pull request. You can then merge the pull request.

A screenshot of the Atlantis apply completed

You just deployed a VPC with Atlantis! How was it? I think Atlantis is a great tool that allows DevOps teams to automate Terraform workflow with pull requests.


Hopefully, you are now familiar with the process of creating AWS infrastructure with Atlantis. Updateterraform/dev/eks Terraform files with appropriate values and create a pull request.

A screenshot of the Atlantis plan of dev/eks

If the plan seems valid and you are ready to apply, comment atlantis apply to Atlantis. After a few minutes, you should be able to see a new EKS cluster in your AWS console.

Terraform Lock File

There is a Terraform lockfile cross platofrm issue. You should create .terraform.lock.hcl with terraform providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64 -platform=linux_arm64 You can check issue #28041 for more information.

Once you have created the EKS cluster, update the kubectl context with the following command:

$ aws eks --region ap-northeast-2 update-kubeconfig --name dev

$ kubectl config get-contexts
CURRENT   NAME                                                      CLUSTER                                                   AUTHINFO                                                  NAMESPACE
          arn:aws:eks:ap-northeast-2:765355018960:cluster/central   arn:aws:eks:ap-northeast-2:765355018960:cluster/central   arn:aws:eks:ap-northeast-2:765355018960:cluster/central
*         arn:aws:eks:ap-northeast-2:765355018960:cluster/dev       arn:aws:eks:ap-northeast-2:765355018960:cluster/dev       arn:aws:eks:ap-northeast-2:765355018960:cluster/dev

$ kubectl config use-context arn:aws:eks:ap-northeast-2:765355018960:cluster/dev
Switched to context "arn:aws:eks:ap-northeast-2:765355018960:cluster/dev".

$ kubectl get nodes
NAME                                             STATUS   ROLES    AGE     VERSION
ip-10-98-4-110.ap-northeast-2.compute.internal   Ready    <none>   4m14s   v1.23.9-eks-ba74326

Dev EKS Addons

EKS addons allow the dev cluster to access Vault and Argo CD in the central cluster. Update terraform/dev/eksfiles with appropriate values and create a pull request. Run atlantis plan and atlantis apply. Once everything is applied, check Vault to see if Kubernetes authentication is successfully created. There will be a secret created under cluster/dev/argocd

A screenshot of the Vault Kubernetes authentication

This allows central Argo CD to deploy applications to the dev cluster. But there is just one thing we need to do before deploying stuff to the dev cluster with Argo CD. Update kubernetes/charts/argocd/values-central.yaml like below:

1- clusters: []
2+ clusters: 
3+   - name: dev

Since Argo CD is managed via Helm, deploy the changes with the following command:

- clusters: []
+ clusters: 
+   - name: dev

You can check Argo CD UI to check whether the dev cluster is configured correctly.

A screenshot of the Argo CD UI showing dev cluster connection

Manage Argo CD Using Argo CD

We are using Helm to deploy and manage Argo CD. However, Argo CD is able to manage itself. Read more in the official documentation.

move to section dev-helm-chart-installationDev Helm Chart Installation

Since we are familiar with Argo CD GitOps workflow, let’s deploy all the Kubernetes applications at once to the dev cluster. Update the following:

  • charts/karpenter/values-dev.yaml
  • charts/aws-ebs-csi-driver/values-dev.yaml
  • charts/aws-load-balancer-controller/values-dev.yaml
  • charts/external-dns/values-dev.yaml
  • charts/echo-server/values-dev.yaml
  • charts/external-secrets-operator/values-dev.yaml

Lastly, update charts/argocd-gitops/values-dev.yaml with appropriate values. Commit and push, and Argo CD will deploy all the applications to the dev cluster. Your charts/argocd-gitops/values-dev should something like this.

clusterName: dev
clusterRegion: ap-northeast-2
infraAccountId: "765355018960"
rootNamespace: argocd

stage: dev
defaultTargetRevision: main
defaultProject: dev-apps

  - name: aws-ebs-csi-driver
    namespace: kube-system
      path: kubernetes/charts/aws-ebs-csi-driver
      - values-dev.yaml
  - name: aws-load-balancer-controller
    namespace: kube-system
      path: kubernetes/charts/aws-load-balancer-controller
      - values-dev.yaml
  - name: cert-manager
    namespace: kube-system
      path: kubernetes/charts/cert-manager
      - values-dev.yaml
  - name: echo-server
    namespace: default
      path: kubernetes/charts/echo-server
      - values-dev.yaml
  - name: external-dns
    namespace: kube-system
      path: kubernetes/charts/external-dns
      - values-dev.yaml
  - name: external-secrets-operator
    namespace: external-secrets-operator
      path: kubernetes/charts/external-secrets-operator
      - values-dev.yaml
  - name: karpenter
    namespace: karpenter
      path: kubernetes/charts/karpenter
      - values-dev.yaml

Check argocd.[YOUR_DOMAIN_NAME]/applications/argocd-dev-apps to see Argo CD deploying all the applications at once.

A screenshot of the Argo CD UI showing dev cluster applications

move to section the-power-of-gitops-with-argo-cdThe Power of GitOps with Argo CD

Argo CD logo

How was deploying the dev cluster compared to the central cluster? It was much easier, right? Once you have setup the GitOps pipeline with Argo CD, deploying another cluster is just a matter of updating Terraform files and Helm Chart values.

move to section prod-cluster-deploymentProd Cluster Deployment

Deploying the prod cluster is similar to the dev cluster. The differences are the following:

  • Cluster Name
  • Region (ap-northeast-2 -> us-east-1)
  • Access Control

Since deploying the prod cluster is just a matter of updating the Terraform files and Helm Chart values, prod cluster deployment is left as an exercise for the reader. 😉

Other Declarative Multi-region EKS Series :

  1. Manage multi-region EKS the Declarative Way - Part 0: Introduction
  2. Manage multi-region EKS the Declarative Way - Part 1: Central Infrastructure
  3. Manage multi-region EKS the Declarative Way - Part 2: Central Helm
  4. Manage multi-region EKS the Declarative Way - Part 4: CI/CD