Manage multi-region EKS the Declarative Way - Part 0: Introduction

Managing EKS is difficult. Declaratively managing multiple EKS clusters in different regions can be even more difficult. This article will cover how to manage multi-region EKS clusters declaratively with GitOps.

All the code for this article can be found on jacobhjkim/declarative-eks-tutorial.

Table of Contents

• The Devops Learning Curve
• What You Will Learn
• Principles
• Tools

move to section the-devops-learning-curveThe DevOps Learning Curve

As the cloud native ecosystem grows, the number of tools and services available to developers also grows. While it is excellent for the ecosystem, it can be difficult for developers to keep up with the latest and greatest.

”I use EKS as a managed Kubernetes service, Use Terraform with Atlantis to deploy AWS infrastructures, Helm Charts for my Kubernetes applications, and Argo CD to deploy those Helm Charts. Karpetner is my Kubernetes autoscaler of choice. Prometheus and Grafana for monitoring. Oh, and Vault for secrets management.”

What the fuck does that mean? Kubernetes, EKS, Terraform, Atlantis, Helm Charts, Prometheus, and Vault may sound like some Pokemon names. But they are all real things used by many DevOps engineers.

It might be overwhelming to learn all these things. However, these modern DevOps principles and the tools that go with them are powerful. With the rise in popularity of cloud-native architectures, these tools are becoming increasingly important. So, let’s embark on a journey to learn how to connect all these tools and create a modern cloud-native infrastructure declaratively.

move to section what-you-will-learnWhat You Will Learn

This article will start by explaining the principles of declarative infrastructure. Then we will go over the tools that we will use to create our infrastructure and why we use them. We will then create full-blown production-ready, multi-region EKS clusters with CI/CD in a declarative way.

move to section principlesPrinciples

These principles are not set in stone, but they are the principles that I have personally found to be helpful.

Infrastructure as Code

Infrastructure as Code (IaC) is a way to manage infrastructure declaratively. IaC is a great way to manage infrastructure because it allows you to version control your infrastructure. You can easily see the changes that you have made to your infrastructure and revert changes if you need to. Since your infrastructure is version controlled, it allows you to share your infrastructure with others easily. Which is especially useful if you are working in a team.

GitOps

GitOps builds upon the idea of IaC by not only version controlling your infrastructure but also your infrastructure deployment. With GitOps, your team can be confident that the deployed infrastructure is the same as the infrastructure configuration in your version control. You can read more about GitOps here.

Now that we have established the basic principles of declarative infrastructure, let’s go over the tools we will use to create our infrastructure.

move to section toolsTools

These are the DevOps tools that we will be using. There are many alternatives to these tools, but these are the tools that I have found to be the most useful in my specific use case. It might be hard to wrap your head around how all these tools work together, but don’t worry. We will also go over how to connect all these tools later in this article.

Kubernetes & EKS

Kubernetes is an open-source container orchestration system for automating deployment, scaling, and management of containerized applications¹.

Amazon Elastic Kubernetes Service (EKS) helps AWS users to run Kubernetes on AWS. If you are not familiar with Kubernetes, I recommend reading the official Kubernetes documentation. This article will assume that you have some basic knowledge of Kubernetes.

Helm

Helm is a package manager for Kubernetes. It allows you to define Kubernetes applications with this format called Charts. Charts are YAML files that define Kubernetes resources like Deployments, Services, and Ingresses. Helm also allows you to define variables in the charts, and you can override the variables with values files. This allows you to reuse the same chart for different environments, which comes in handy when you have multiple clusters. (Like our use case here 😉)

Helm’s other superpower is that many charts are already available in the open-source community. You can find charts for popular applications like MySQL, Redis, and Prometheus. You don’t have to write the YAML files for these applications yourself. You can just install the charts, and you are good to go.

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. ²

Argo CD is the core tool enabling us to manage Kubernetes applications declaratively using Git. The Helm chart we discussed earlier is what Argo CD uses to deploy applications to the Kubernetes clusters. Argo CD monitors your Helm charts from your git repositories and applies changes to your cluster. This allows us to do GitOps which we covered in the previous section.

Another great feature of Argo CD is that it allows us to manage multiple clusters with a single Argo CD instance. Argo CD from our central cluster installs and updates the Kubernetes application in our dev and prod clusters. It also has a nice web UI allowing you to see your applications’ status.

Terraform

Terraform is an infrastructure as code tool that lets you define cloud resources in human-readable configuration files that you can version, reuse, and share. ³

Our cloud infrastructures like VPC, EKS clusters, and RDS instances are defined and managed with Terraform. Terraform uses a declarative language called HCL to define our infrastructure. Terraform has a nice ecosystem of providers allowing us to create resources in many different cloud providers. We will be using the AWS provider to build our infrastructure in AWS.

Atlantis

Atlantis is a tool that allows you to run Terraform commands from pull requests. It is a great tool for teams that use Terraform. It helps you to review Terraform changes with your team before applying them to your cloud infrastructures. Recently, Atlantis’ maintainer joined Hashicorp (the creators of Terraform), so I expect Atlantis to become even better in the future.

Vault

HashiCorp Vault is an identity-based secrets and encryption management system.⁴

A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated by authentication and authorization methods. Vault is a great tool for managing secrets for our Kubernetes applications. We use Vault to store secrets for not only our applications but also for our infrastructure. For example, we can use Vault to store the AWS credentials for our CI/CD pipeline to deploy our applications.

External Secrets

External Secrets is a Kubernetes controller that allows you to use secrets from external secret management tools like Vault. You might ask, why can’t we use Kubernetes’ built-in secret? Well, if you create my-secret.yaml and commit it to your git repository, anyone can see the secrets. The secret data has to be encoded in base64, but it is still not secure. External Secrets allows you to store secrets in external secret management tools like Vault, injecting them as Kubernetes secrets. So your precious secrets are not exposed in your codebase.

GitHub Actions

GitHub Actions is a CI/CD tool that allows you to automate your workflows within GitHub. We use GitHub Actions to run CI/CD tasks like running tests, building Docker images, and many more. Instead of using GitHub Actions runner provided by GitHub, we are going to use self-hosted runners on our central cluster.

That is a lot of tools, but don’t worry. We will go through each tool one by one and explain how we use them. Once you see them in action, it will be much easier to understand how they all fit together.

In the next sections, we will go through the process of setting up our infrastructure. Then use the tools we discussed to deploy our applications to our dev and prod clusters.

Other Declarative Multi-region EKS Series :

1. Manage multi-region EKS the Declarative Way - Part 1: Central Infrastructure
2. Manage multi-region EKS the Declarative Way - Part 2: Central Helm
3. Manage multi-region EKS the Declarative Way - Part 3: Dev & Prod Clusters
4. Manage multi-region EKS the Declarative Way - Part 4: CI/CD