Skip to content
  • Products
    • Portfolio overview >

      Flexsave™

      Automatically manage cloud compute for optimized costs and agility

      Cloud Analytics

      Make sense of spend and chargeback to align with your business

      google cloud msp

      BigQuery Lens

      Optimize BigQuery costs with actionable recommendations and usage insights

      Spot Scaling

      Maximize AWS Spot savings and minimize disruptions for optimized scaling

      Anomaly Detection

      Autonomously identify cost spikes early, with zero configuration

      Platform overview >

      Organize your billing data for better business decisions

  • Services
    • Services overview >

      How we work

      Learn how we’re redefining support with our customer reliability engineering

      Stats

      View our live support and customer satisfaction statistics in real-time

      Cloud solutions

      Proven solutions to cloud complexity

      Areas of expertise

      Cloud Architecture

      Ensure your cloud architecture is future-ready and built for success

      Cloud Cost Optimization

      Identify opportunities to optimize costs and target spend for added value

      Cloud Migration

      Realize greater efficiency and innovation with successful cloud migration

      Cloud Security

      Center security in your cloud strategy to ensure ongoing efficacy and growth

      Data and Analytics

      Harness the potential of big data and analytics to gain a competitive edge

      Data Management

      Build your data practice with expert guidance tailored to your business goals

      DevOps Jump Start

      Accelerate your AWS workloads & release pipelines while also increasing automation, monitoring & reliability

      Infrastructure

      Maximize the full suite capabilities from your cloud infrastructure

      Kubernetes

      Manage the complexity of Kubernetes to enable innovation and scalability

      Location-Based Services

      Transform geolocational data into real-world, real-time intelligence

      Machine Learning

      Level-up key data with ML capabilities that accelerate innovation

      Multicloud

      Create meaningful business value with a robust multicloud strategy

      Training

      Build skills and capability across teams with certified, expert-led training

  • Partners
    • Alliances

      Proud to be an award‒winning multicloud partner to top‒tier cloud providers

      doit-together

      DoiT Together

      Enabling cloud growth and unlocking revenue through expert partnership

      ISV Go-Global

      Accelerate new customer growth and Marketplace integration on AWS and GCP

  • Resources
    • Resources hub >

      Blog

      Read the latest insights, tips and perspectives from our team of cloud experts

      Case Studies

      See how we’ve helped thousands of public cloud customers achieve their goals

      Ebooks and Guides

      Discover foundational expertise and future-ready recommendations for the cloud

      Events and Webinars

      Tech talks and interactive expert sessions delivered both virtually and in person

      GCPInstances.info

      Google Cloud Compute Engine instance comparison

      Help center

      Read documentation, product updates, and more

      Newsroom

      See what's new from DoiT in our latest news and announcements

      Trust Center

      How we focus on security, compliance, and privacy

      Videos

      Watch product demos, interviews and more from our cloud experts

  • About
    • About DoiT >

      Careers

      Browse our open positions and learn more about what it takes to be a Do’er

      Leadership

      Meet the team leading DoiT and our customers on a journey of hypergrowth

      Newsroom

      See what's new from DoiT in our latest news and announcements

  • Pricing
  • Contact us
  • Sign In
  • Products
    • Flexsave ™
    • Cloud Analytics
    • Spot Scaling
    • BigQuery Lens
    • Anomaly Detection
    • DoiT Platform
  • Services
    • How We Work
    • Stats
    • Cloud Solutions
    • Areas of expertise
      • Cloud Architecture
      • Cloud Cost Optimization
      • Cloud Migration Consulting Services
      • Cloud Security
      • Data and Analytics
      • Data Management
      • DevOps with AWS & DoiT
      • Infrastructure
      • Kubernetes
      • Location Based Services
      • Machine Learning
      • Multicloud
      • Training
  • Partners
    • ISV Go-Global
    • Award-winning public cloud partner
    • DoiT Together
  • Resources
    • Blog
    • Case Studies
    • Ebooks and Guides
    • Events and Webinars
    • GCPInstances.info
    • Help center
    • Newsroom
    • Trust Center
    • Videos
  • Pricing
  • About
    • Careers
    • Leadership
    • Newsroom
  • Contact us
  • Sign In
Contact us
Sign in

Blog

Google Kubernetes Engine Cluster Migration with Heptio Velero

  • Eran Chetzroni Eran Chetzroni
  • Date: April 22, 2019

1 yyq8hwire9b5wgawnxivhq

From time to time, you may need to move your entire Kubernetes workload to a new cluster. It might be for testing purposes or to upgrade a major version and sometimes as a disaster recovery.

Recently, I had to migrate one of our customer’s GKE (Google Kubernetes Engine) cluster from Google Cloud Legacy VPC Network to a VPC Native Network and, unfortunately, there is no documented upgrade path for networks in Google Cloud.

After some research, I have found a couple of tools that together made this exercise extremely easy.

Cloning Existing Cluster

The Google Kubernetes Engine (GKE) has “create a new cluster” feature, allowing to clone an existing cluster:

1 nkdwsv7pbf4pv6bavvnbw

This will help with copying the cluster configurations, including:

  • Zones
  • Node Pools and related node pool configuration
  • Additional Configuration settings such as node labels and more

You can of course, edit everything before creating the new cluster, but overall this “clone” functionality helps a lot in making sure all nodes and labels are properly configured as easy as two clicks. 🚀

Please bare in mind that it will NOT copy your Kubernetes resources such as deployments, services, ingress, custom resource definitions and therefore you will need another tool to help with migrating of all of these.

Migrating Kubernetes Resources

Now that we have a cloned cluster, I had to migrate all Kubernetes resources such as:

  • Workloads
  • Services
  • Configs
  • Secrets
  • Storage
  • Custom Resource Definitions
  • more more more…

Just a quick run of :

$ kubectl api-resources

reveals 74 kinds of Kubernetes resources on my test cluster 😱. Yikes!

Luckily, I was able to find Velero by Heptio (formerly Heptio Ark) to help me with backing up and restoring my Kubernetes cluster resources as well as persistent volumes.

Velero helps you with:

  • Backup and restore of your Kubernetes cluster.
  • Copy cluster resources from one cluster to another
  • Replicate your production environment for development and testing environments. 🎸

Velero looks like a good candidate for replicating the Kubernetes cluster resources to a new cluster, so I decided to give it a try.

Velero Installation

Velero has 2 major components:

  1. velero-cli — a command-line client that runs locally
  2. velero deployment — a server that runs on your cluster

before getting started, it’s always a good habit to review the official release of Velero, the latest (stable) release is v0.11.0

I am installing Velero on my GKE cluster, so I was following the instructions here.

The basic installation flow is :

  1. Install the velero-cli
brew install velero

(you can also download it manually, it’s just one binary)

2. Create a Google Cloud Storage bucket

gsutil mb gs://<gke-cluster-migrate-velero-placeholder-name>

3. Create service account / permissions / policies

See instructions under “Create service account” here

4. Add credentials to your GKE Cluster

See instructions under “Credentials and configuration” here

make sure you replace <YOUR_BUCKET> in the 05-backupstoragelocation.yaml

5. Deploy the velero-server

  kubectl apply -f config/gcp/05-backupstoragelocation.yaml
  kubectl apply -f config/gcp/06-volumesnapshotlocation.yaml
  kubectl apply -f config/gcp/10-deployment.yaml

It’s time for backup!

To backup my entire cluster, I have used:

velero backup create <BACKUP-NAME>

When you run velero backup create <BACKUP-NAME>:

  1. The Velero client makes a call to the Kubernetes API server to create a Backup object.
  2. The BackupController notices the new Backup object and performs validation.
  3. The BackupController begins the backup process. It collects the data to back up by querying the API server for resources.
  4. The BackupController makes a call to the object storage service – for example, GCS Bucket – to upload the backup file.

Nice!

To see the status of your backup just run:

velero get backups

Migration time

Now that we have a full backup of our original cluster (cluster 1) I had to deploy Velero on the new cluster (cluster 2).

There are a few things I had to attention to:

  • On cluster 2, I had to add the --restore-only flag to the server spec in the Velero deployment YAML
  • I had to make sure the BackupStorageLocation matches the one from cluster 1, so that your new Velero server instance points to the same bucket.
  • Finally, I made sure the Velero resource on cluster 2 are synchronized with the backup files in cloud storage. Note the default sync interval is 1 minute, so make sure to wait before checking.
velero backup describe <BACKUP-NAME>

Once I had confirmed the right backup (<BACKUP-NAME>) is now present, I could restore everything with:

velero restore create --from-backup <BACKUP-NAME>

Now let’s verify everything on cluster 2:

velero restore get

and use the restore name from the previous command:

velero restore describe <RESTORE-NAME-FROM-GET-COMMAND>

That’s it!

With GKE “Clone Cluster” and Heptio Velero, I was able to successfully migrate the cluster, including cluster configuration and resources.

These two tools have saved me tons of hours as well as greatly simplified the overall process of mapping, backing up and restoring Kubernetes resources .

Want more stories? Check our blog, or follow Eran on Twitter.

Subscribe to updates, news and more.

Subscribe

Subscribe to updates, news and more.

Related blogs

Ramp Plans Resource Hub Header1

Monitor your cloud commit attainment with DoiT Ramp Plans

DoiT Ramp Plans help you visualize, manage, and track your commit attainment so you can make sure you spend what you committed to, and act proactively.

Keep reading 
Secure-access-to-GCP-services-in-GitLab-Pipelines-with-Workload-Identity-Federation-DoiT-International

Secure access to GCP services in GitLab Pipelines with Workload Identity Federation

Traditionally, when using services like Google Cloud in a non-GCP environment (e.g., a CI/CD environment like GitLab pipelines), developers

Keep reading 
August 2023 Product Updates Resource Grid

[August 2023] DoiT Product Release Notes

We’re excited to share some recent updates we made in August to DoiT’s product portfolio. If you prefer watching

Keep reading 
View all blogs
Let’s do it

From cost optimization to cloud migration, machine learning and CloudOps, we’re here to make the public cloud easy — without the costs.

Ready to get started?

Get in touch

Company

  • About us
  • Blog
  • Careers
  • MS-HT statement
  • Newsroom
  • Privacy policy
  • Terms

Offering

  • Compliance
  • Products
  • Services
  • Solutions
  • Resources

Support

  • Sign in
  • Help center
  • Open ticket
  • Contact us

Never miss an update.

Subscribe to our newsletter

Subscribe to updates, news and more.