Multi-Cluster Istio 1.5 with Private GKE Clusters and Google Cloud Internal Load Balancer

This tutorial shows you how to build a private multicluster service mesh solution, with Istio 1.5 and Service Mesh Hub for mesh Federation. This is achieved by using an Internal Load Balancer (ILB) to connect Istio workloads running in multi-region Private Google Kubernetes Engine (GKE) clusters.

1 dpz3uewxwyunt6w2pil wq


Istio 1.5 was released on March 5th and with this major release, comes several important changes, however, support for Hashicorp Vault as External CA is still in progress. For this reason, I started to investigate what other options there are for managing the common root CA in a secure way. I then came across’s open-source project, Service Mesh Hub. Service mesh hub is a solution that can help you to unify the root identity between multiple service mesh installations. This means that any intermediates are signed by the same Root CA and end-to-end mTLS between clusters and services can be established correctly.

How it Works

In this tutorial:

we will build the following architecture:

1 5picphwle91w0 on7d0xkq

There are several reasons to isolate your Google Kubernetes Engine (GKE) clusters from internet access, the primary one being security. In a private cluster, nodes only have RFC 1918 private addresses and communicate with the master’s private endpoint via private networking.” This prevents any access authorized or no from the outside internet. “

The TCP/UDP Internal Load Balancer has a less well-known beta feature, called Global Access. This feature allows clients from any region in your VPC network to access the internal TCP/UDP load balancer. Without global access, traffic originating from clients in your VPC network must remain in the same region as the load balancer. Global access is enabled, in your private Kubernetes clusters, on a per-Service basis by using the following annotation:”true”.

In the example implementation, we will annotate the ingressGateway Kubernetes Service directly to provision a GCP ILB with global access. After Global access has been applied we now have network connectivity in place between our private clusters running in different regions.

Success! Notice we have the service A ( that needs to talk to service B ( in another mesh. If a service in one mesh requires a service from another, you must federate identity and trust between the two meshes. To do this you must exchange the trust bundle between the meshes.

Service mesh hub allows you to group together multiple meshes into an object called a VirtualMesh. Creating this resource will instruct the Service Mesh to establish a shared root identity across the clusters in the Virtual Mesh as well as federate the services.

Understanding the shared root process

1 2sof8fap ntgkzjj131era

With the creation of the VirtualMesh resource:

  1. Service Mesh Hub will kick off the process to unify the identity with a shared root.
  2. Service Mesh Hub will use a Certificate Signing Request (CSR) agent on each of the different clusters to create a new key/cert pair that will form an intermediate CA. Each cluster’s intermediate CA will be present and used by the mesh on that cluster.
  3. It will then create a Certificate Signing Request, represented by the VirtualMeshCertificateSigningRequest CR.
  4. Service Mesh Hub will then sign the certificate with the Root CA specified in the VirtualMesh.
  5. Once this process has completed, we need to bounce the istiod control plane. This is because the Istio control plane picks up the CA for Citadel and does not rotate it often enough. This is going to be improved in future versions of Istio.

Once trust has been established, Service Mesh Hub will start federating services, automatically generating service entries for you! so that they are accessible across clusters.

1 5gjw2undtt0z4okxdtyzma

For this tutorial, the enforceAccessControl flag is disabled and this will be the subject for one of my next articles. Where we will see how Istio can provide security features for a strong identity, powerful policy, and authentication, authorization, and audit (AAA) tools to protect your services and data.


In this tutorial, I have demonstrated the ease with which you can implement a service mesh solution with intra-mesh communication. Service to service communication can happen internally via secure mTLS traffic communication, without leaving the Google Cloud’s Global Network.

Tutorial Code

The step by step instructions, for this tutorial, can be found here:


  1. Implementation code —
  2. GCP — Private GKE Clusters —
  3. GCP — TCP/UDP Internal Load Balancer —
  4. Istio-Configure the example service —
  5. Service Mesh Hub — Concepts —

Subscribe to updates, news and more.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related blogs