Managing GKE (k8s) clusters and applications inside them has become a never ending battle for many of us. Managing various attributes like node-pools / add-ons / ingress controllers / SSL certificate manager / rollout of applications and its associated configuration accurately has become cumbersome for many. This has become more common given the rise of micro-services and event-driven architecture having many components.

Managing GKE ( K8S ) clusters

GKE / K8S cluster setup is a complicated process. A good approach is to automate the creation part as a IAC artefact. We will be using terraform as the IAC tool in this article.

IAC Setup

Creating modules with parameters / attributes for every resources is the key to maintain environment consistency (parity). We will have 3 modules in our repo. 1. GKE Cluster module having two node pools. 2. nginx ingress controller module using helm chart provisioner. 3. kcert letsencrypt SSL certificate provider module for your public endpoints.

The Code for all the below modules are provided in our repository at : https://github.com/agileguru/gke_nginx_kcert_quick_start

• Provisioning GKE Cluster with multiple node-pools
  In this module we have main.tf, variables.tf and outputs.tf for provisioning / parameterizing / resuable metadata for a GKE cluster respectively.

GKE Terraform IAC Module screenshot

• Installation of nginx ingress controller
  In this module we have main.tf, variables.tf and an optional (empty) outputs.tf for provisioning / parameterizing nignx controller respectively.

• Installation of kcert Letsencrypt SSL manager
  In this module we we have main.tf, variables.tf and an optional (empty) outputs.tf for provisioning / parameterizing kcert SSL controller respectively.

Kcert Terraform IAC provisioner main.tf screenshot

• Orchestrating Everything Together
  After finishing the 3 modules we will now create a “devops” k8s environment by creating “devops” module which is called from the main / root to club everything together.

Devops k8s Environment Module screenshot

Root orchestrate creating Devops Module screenshot

• Provision the cluster and controllers with terraform
  * Change the project / region / zone name in variables.tf
  * Change the bucket name in backend.tf after creating in gcp console
  * Execute the following commands

terraform init
terraform plan -var-file=sample.tfvars ( change sample.tfvars if needed)
terraform apply -var-file=sample.tfvars ( change sample.tfvars if needed)

1. After executing terraform commands, you will get the IP address of the LoadBalancer for domain registration.
2. You can get the kubectl config usig the command …

gcloud container clusters get-credentials <cluster name> — zone <cluster zone> — project <project id having the cluster>

Managing deployments using Kustomize

Cluster is now ready for workload deployment. We will be using Kustomize plugin to manage it in a easier way. We will use a simple use case for this article.

• We have 2 apps api-1 and api-2 based on tutum/hello-world image.
• We also have 2 k8s namespaces corresponding to DEV & SIT environments.
• We need to deploy the service and expose it via https (ssl ) with respective configurations / deployment / service and ingress mapping.
• We need to store this in a repository. For Demo it is stored in our repository at https://github.com/agileguru/kustomize_quickstart_demo

Step 1 : Creating the folder structure

Base Folders of components and configurations

Step 2 : Customise each environment using overlays

Overlays Folders For Environments using patch : merge config using kustomize yaml

Step 3 : Change Ingress Host Name Mapping

Change hostname in dev-ingress-patch.json & sit-ingress-patch.json to valid host / domain. It looks similar to code given below…

[
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "dev.agileguru.org"
    },
    {
        "op": "replace",
        "path": "/spec/tls/0/hosts/0",
        "value": "dev.agileguru.org"
    }
]

[
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "sit.agileguru.org"
    },
    {
        "op": "replace",
        "path": "/spec/tls/0/hosts/0",
        "value": "sit.agileguru.org"
    }
]

Step 4 : Deploying your Applications

$ kubectl apply -k overlays/dev
namespace/dev created
configmap/config-map-api-1 created
configmap/config-map-api-2 created
service/api-1-service created
service/api-2-service created
deployment.apps/api-1-deployment created
deployment.apps/api-2-deployment created

Step 5 : Un-deploying your Applications

$ kubectl delete -k overlays/dev
namespace "dev" deleted
configmap "config-map-api-1" deleted
configmap "config-map-api-2" deleted
service "api-1-service" deleted
service "api-2-service" deleted
deployment.apps "api-1-deployment" deleted
deployment.apps "api-2-deployment" deleted
ingress.networking.k8s.io "app-ingress" deleted

Kustomize Best Practices

1. Things To You Need To Do
   — Keep the Replica Count to 0 in the base configuration
   — Always Specify the Namespace in Overlay kustomization.yaml
   — Always Dry Run with yaml output to ensure accuracy
   — Good Naming Conventions for folders and manifest files
   — Keep Ingress Mapping in its own-folder
   — Always have override.yaml in for each component.
2. Things You Should Not Do
   — Hard-Coding namespace in Base Configurations
   — Mix Configurations and Application Code In the Same Folder
   — Git branches for Environment configuration ( know as parity drift )

Conclusion

After completing the above steps we have 1. k8s which is easy to manage / upgrade with nginx and kcert SSL certificate manager without having to manage you SSL certificates ever for your public endpoints. 2. Mechanism / Framework to manage your secure web based endpoints following IAC / Devops / DRY principles.

Resources

1. GKE Nginx Kcert getting Started Repo : https://github.com/agileguru/gke_nginx_kcert_quick_start
2. Kustomize Quick Started Repo : https://github.com/agileguru/kustomize_quickstart_demo
3. Kustomize Docs : https://kustomize.io/
4. Kcert GitHub : https://github.com/nabsul/kcert
5. Nginx Controller : https://kubernetes.github.io/ingress-nginx/
6. LetsEncrypt : https://letsencrypt.org/