In this three-part series, I will explain how to use Kubernetes (K8s) and Terraform (TF) together to set up a Kubernetes cluster, manage applications and install Kasten. We will of course keep data management best practices in mind for every step. Installing Kasten in the cluster is also a great example of how Terraform can be used when managing cloud resources outside the cluster.
This series is intended for people who already have a basic understanding of Kubernetes and are questioning how Terraform could be useful in the context of Kubernetes.
In the first part, we will discuss the concepts behind Terraform and Kubernetes, their similarities & differences, and how to use the two in harmony. The second part will be a hands-on example for setting up a Kubernetes cluster on AWS EKS with Terraform. And lastly, we will use Terraform to install Kasten and setup an S3 export location. You can also find all the code on GitHub.
What is Terraform?
Infrastructure as a Code
The core concept behind Terraform is Infrastructure as Code or IaC for short, the idea is that the resources that make up your infrastructure should be defined in code. What infrastructure means in this context is up to you. Terraform gives you the basic framework and language for defining your infrastructure and a number of default “providers” that, well, provideresources. The most popular providers are for public cloud platforms like AWS, Azure, and GCP but you can also find providers for Git hosting platforms like GitHub or generic HTTP or FPT providers that can be used to tie together all kinds of things. You can find a truly impressive catalog of providers for pretty much every imaginable use case. Of course, there is also a Kubernetes provider for Terraform.
Using code to define infrastructure has various advantages:
- You can use version control like Git to keep a history of changes
- Code comments can be used to explain why things are how they are
- It makes it much easier to recreate the same setup
- Pull requests workflows can be used for reviews
Platform Independence
Terraform is not tied to any specific cloud platform. As long as you can find a provider for a given platform you can use it together with Terraform.
However, the code you write with Terraform is strongly tied to the platform you choose. You will also need a good understanding of the platform you are using. This may seem like a downside at first but it also comes with major advantages:
It enables Terraform to be transparent about what is happening behind the scenes and you never need to worry about changes being made outside your control
You can use multiple platforms together: Want to connect your AWS application to your CDN on Akamai? Just install the AWS and Akamai providers and use the address or your app as input for the CDN origin configuration.
Similarities and Differences
Kubernetes also supports the IaC paradigm. You probably already have a directory with YAML files for various Kubernetes resources.
Convergence vs. Plan & Apply
The way IaC works in Kubernetes is different from the way it works in Terraform. In K8s the YAML files are uploaded to the API server and the K8s controllers continuously try to move the cluster state towards the desired state in the control loop.
Terraform also tracks the current state of managed resources and calculates a delta to move from the current state to the defined state. This process is called the plan phase. The plan is then applied in the apply phase. After the apply the definition should match the real state of the resources. The difference is that the apply is only executed occasionally; either locally, in a CI pipeline, or using Terraform enterprise.
Drift Detection
If managed resources are changed outside of Terraform, you will see this in the next plan. Unless you also update the code, Terraform will then try to undo these changes. This is called drift detection.
In Kubernetes when you change a resource without updating the original YAML, Kubernetes will not warn you the next time you run kubectl apply -f. Using –dry-run and kubectl diff allow you to work around this but there are also some additional changes that Terraform can detect that kubectl can’t. For example, when you remove a resource from your definition file, kubectl will not automatically delete this resource.
The Kubernetes Terraform Provider
The Kubernetes provider can be used to address this exact issue. The Kubernetes Provider manages the state of the API server so it is simply an alternative to uploading your configurations via kubectl apply -f (and an alternative to YAML). The Kubernetes control loop is still in charge of converging to the desired state (starting pods, claiming volumes, etc).
The advantage of using Kubernetes to manage your K8s resources instead of kubectl and YAML is that if someone changes the state of the API server (e.g. through the web dashboard) you will be notified before you override these changes in the next apply. To do this Terraform will always first refresh the current state before calculating the delta in the plan phase.
Disadvantages
While this is can be a significant advantage when managing complex projects with hundreds of resources, I’d generally advise against this approach. The reason is simple: You would be working against the rest of the K8s community. kubectl+ YAML is the community standard. You will have to translate most guides, examples, and tutorials into your Terraform world. Additionally, there is no real counterpart for deploying remote YAML files via HTTP (kubectl apply -f <https://example.com/deploy.yaml>). Technically you would have to download the YAML file and manually translate it into Terraform code. The Terraform way of doing this would be to create and publish a module on the Terraform registry.
Helm can be used together with Terraform with the Helm Provider but you will be switching back and forth between the YAML in your Helm Charts and Terraform code.
When to use it
While I generally recommend against managing Kubernetes resources in Terraform via the Kubernetes Provider there is one exception:
As I said earlier, one big selling point of Terraform is tying resources from different platforms together. If you have a workflow that includes creating resources in Kubernetes and then linking them together with other resources that you want to manage via Terraform, using Terraform homogeneously for everything might be your best advice.
Importantly Terraform also gives you “data sources” to access resources read-only. That way you can create resources using common K8s tools (i.e. helm or kubectl, and even something like Rancher) and then access them via data sources in Terraform.
The Kubernetes + Terraform Harmony
So far you might think that using Kubernetes with Terraform only has very limited application but there still is a nice sweet spot where the two can work together harmoniously:
Using TF for Deploying K8s
Most importantly Terraform can be employ one level deeper to manage the Kubernetes cluster itself. No matter if you are using public clouds like AWS, Azure, or GPC or deploying Kubernetes on-prem, Terraform can be an incredibly useful tool for managing the underlying resources like Nodes, Volumes, Load Balancers, etc.
Later we will go into in more detail with an example of using Terraform to deploy K8s on AWS using EKS.
Using TF for Deploying into K8s
As explained earlier using Terraform to manage resources in K8s should be carefully considered. If you still decide to go this route you should be careful about using Terraform to create the cluster itself and then accessing the cluster via a Terraform provider within the same TF project:
One limitation of Terraform is that TF wants to check the state and existence or non-existence of all resources up-front in the plan phase. If you have a TF project that spins up a cluster and then connects to that cluster to create resources, TF will fail because it can not calculate the plan for the K8s resources of the not yet existing cluster. There is an ongoing discussion about how to solve this on GitHub.
This issue is most commonly encountered when trying to manage different “levels” of infrastructure in the same project. In my opinion, the cleanest solution is to split your code into two projects and apply them one after the other. You can think about it as platform-level-infra and app-level-infra.
Using TF for Resources Outside of K8s
Another great use case for Terraform is creating resources outside of K8s. For example, an external RDS database or S3 bucket that your application can connect to. Later we will do exactly this to create an S3 export location for Kasten (I told you I would get to Kasten eventually). Some of this could also be archived via Custom Resource Definitions (CRD) in K8s but CRDs are far from where TF providers are in terms of their support for cloud resources.
And in the case of using a S3 bucket as export location, Terraform has another great advantage: The Terraform Plan, Apply, and State can be strictly separated from the K8s cluster. You can have a strict separation between your cluster and the backup location and still manage both with the same tool.
Related Resources
- Kubernetes and Terraform Part 2: Hands-on Deploying K8s with Terraform
- Working with Kubernetes and Terraform Part 3: Installing Kasten using Terraform
