Production grade IaC deployments using go testing framework
Introduction
When deploying Infrastructure as Code (IaC), provisioning is just the first step—ensuring that the infrastructure functions as expected is equally critical. A successful deployment doesn’t necessarily mean a fully operational system. Testing the outputs and interacting with the deployed resources is key to verifying real-world usability.
In this blog post, I’ll demonstrate how I use the Terratest Go library to automate the validation of an Azure Kubernetes Service (AKS) cluster deployed with OpenTofu. This approach has been invaluable across small, medium, and large-scale projects, saving me countless hours debugging and fixing infrastructure issues. By integrating testing directly into the deployment pipeline, I ensure that my infrastructure is not only successfully provisioned but also production-ready.
The code repository referenced in this blog post is accessible here.
Why Test IaC?
Without automated validation, infrastructure deployments can lead to hidden issues such as:
Misconfigured networking preventing application communication
Incorrect permissions breaking access to critical resources
Missing or improperly set environment variables
Services failing to start despite successful provisioning
Automated testing with Terratest allows us to catch these issues early by interacting with the deployed infrastructure and verifying its functionality.
Example: Validating an AKS Cluster Deployment
In the following example, I use Terratest to deploy an AKS cluster with OpenTofu and verify its functionality. The test:
Deploys the AKS cluster using OpenTofu.
Retrieves necessary outputs such as certificates and host information.
Uses these outputs to create a Kubernetes client.
Interacts with the cluster to ensure it is functional by listing namespaces.
This is an example for the main test function.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
package test
import (
"context"
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestAKSCluster(t *testing.T) {
t.Parallel()
// Define the OpenTofu options
terraformOptions := &terraform.Options{
// The path to where your OpenTofu code is located
TerraformDir: "../examples/aks",
// Variables to pass to our OpenTofu code using -var options
Vars: map[string]interface{}{
"resource_group_name": "test-rg",
"location": "uksouth",
"prefix": "test",
"labels": map[string]string{
"env": "test",
},
},
}
// Clean up resources with "tofu destroy" at the end of the test
defer terraform.Destroy(t, terraformOptions)
// Run "tofu init" and "tofu apply". Fail the test if there are any errors.
terraform.InitAndApply(t, terraformOptions)
// Fetch the outputs
caCert := fetchSensitiveOutput(t, terraformOptions, "ca_certificate")
clientKey := fetchSensitiveOutput(t, terraformOptions, "client_key")
clientCert := fetchSensitiveOutput(t, terraformOptions, "client_certificate")
host := fetchSensitiveOutput(t, terraformOptions, "host")
// Decode the base64 encoded strings
caCertDecoded := decodeBase64(t, caCert)
clientKeyDecoded := decodeBase64(t, clientKey)
clientCertDecoded := decodeBase64(t, clientCert)
// Create a new REST config using the outputs
restConfig := newRESTConfig(caCertDecoded, clientKeyDecoded, clientCertDecoded, host)
// Create a new Kubernetes client using the REST config
k8sClient, err := newK8sClient(restConfig)
if err != nil {
t.Fatalf("Failed to create Kubernetes client: %v", err)
}
// Use the Kubernetes client to interact with the cluster
_, err = k8sClient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
if err != nil {
t.Fatalf("Failed to list namespaces: %v", err)
}
}
Here is an example of the supporting test functions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
package test
import (
"encoding/base64"
"fmt"
"testing"
"github.com/gruntwork-io/terratest/modules/logger"
"github.com/gruntwork-io/terratest/modules/terraform"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
)
// fetch the sensitive output from OpenTofu
func fetchSensitiveOutput(t *testing.T, options *terraform.Options, name string) string {
defer func() {
options.Logger = nil
}()
options.Logger = logger.Discard
return terraform.Output(t, options, name)
}
// decode the base64 encoded string
func decodeBase64(t *testing.T, encoded string) string {
decodedBytes, err := base64.StdEncoding.DecodeString(encoded)
if err != nil {
t.Fatalf("Failed to decode base64 string: %v", err)
}
return string(decodedBytes)
}
// newK8sClient creates a new Kubernetes client using REST config
func newK8sClient(restConfig *rest.Config) (*kubernetes.Clientset, error) {
clientset, err := kubernetes.NewForConfig(restConfig)
if err != nil {
return nil, fmt.Errorf("failed to create Kubernetes client: %v", err)
}
return clientset, nil
}
// creates a new REST config using the provided options
func newRESTConfig(caCert, clientKey, clientCert, host string) *rest.Config {
return &rest.Config{
Host: host,
TLSClientConfig: rest.TLSClientConfig{
CAData: []byte(caCert),
CertData: []byte(clientCert),
KeyData: []byte(clientKey),
},
}
}
Breakdown of the Test
Provisioning the AKS Cluster: OpenTofu is used to deploy the cluster.
Retrieving Outputs: The test extracts sensitive outputs needed to authenticate with the cluster.
Creating a Kubernetes Client: These outputs are used to establish a connection with the cluster.
Validating Cluster Functionality: The test interacts with the Kubernetes API by listing namespaces, confirming that the cluster is accessible and functioning as expected.
Conclusions
Automating IaC testing with Terratest ensures that infrastructure deployments are not only successful but also functional and production-ready. By interacting with the cluster after provisioning, we can verify that it meets our expectations before proceeding with application deployment.
This approach has saved me significant time debugging and fixing infrastructure issues. If you’re deploying Kubernetes clusters or any cloud infrastructure, consider integrating automated testing into your workflow—it will pay off in the long run!
The code repository referenced in this blog post is accessible here.