Skip to content
  • Products
    • Portfolio overview >

      Flexsave™

      Automatically manage cloud compute for optimized costs and agility

      Cloud Analytics

      Make sense of spend and chargeback to align with your business

      google cloud msp

      BigQuery Lens

      Optimize BigQuery costs with actionable recommendations and usage insights

      Spot Scaling

      Maximize AWS Spot savings and minimize disruptions for optimized scaling

      Anomaly Detection

      Autonomously identify cost spikes early, with zero configuration

      Platform overview >

      Organize your billing data for better business decisions

  • Services
    • Services overview >

      How we work

      Learn how we’re redefining support with our customer reliability engineering

      Stats

      View our live support and customer satisfaction statistics in real-time

      Cloud solutions

      Proven solutions to cloud complexity

      FinOps

      Learn how DoiT enables critical FinOps capabilities

      Areas of expertise

      Cloud Architecture

      Ensure your cloud architecture is future-ready and built for success

      Cloud Cost Optimization

      Identify opportunities to optimize costs and target spend for added value

      Cloud Migration

      Realize greater efficiency and innovation with successful cloud migration

      Cloud Security

      Center security in your cloud strategy to ensure ongoing efficacy and growth

      Data and Analytics

      Harness the potential of big data and analytics to gain a competitive edge

      Data Management

      Build your data practice with expert guidance tailored to your business goals

      DevOps Jump Start

      Accelerate your AWS workloads & release pipelines while also increasing automation, monitoring & reliability

      Infrastructure

      Maximize the full suite capabilities from your cloud infrastructure

      Kubernetes

      Manage the complexity of Kubernetes to enable innovation and scalability

      Location-Based Services

      Transform geolocational data into real-world, real-time intelligence

      Machine Learning

      Level-up key data with ML capabilities that accelerate innovation

      Multicloud

      Create meaningful business value with a robust multicloud strategy

      Training

      Build skills and capability across teams with certified, expert-led training

  • Partners
    • Alliances

      Proud to be an award‒winning multicloud partner to top‒tier cloud providers

      doit-together

      DoiT Together

      Enabling cloud growth and unlocking revenue through expert partnership

      ISV Go-Global

      Accelerate new customer growth and Marketplace integration on AWS and GCP

  • Resources
    • Resources hub >

      Blog

      Read the latest insights, tips and perspectives from our team of cloud experts

      Case Studies

      See how we’ve helped thousands of public cloud customers achieve their goals

      Cloud Masters Podcast

      Listen to our experts and customers share tangible tips for navigating the cloud.

      Ebooks and Guides

      Discover foundational expertise and future-ready recommendations for the cloud

      Events and Webinars

      Tech talks and interactive expert sessions delivered both virtually and in person

      GCPInstances.info

      Google Cloud Compute Engine instance comparison

      Help center

      Read documentation, product updates, and more

      Newsroom

      See what's new from DoiT in our latest news and announcements

      Trust Center

      How we focus on security, compliance, and privacy

      Videos

      Watch product demos, interviews and more from our cloud experts

  • About
    • About DoiT >

      Careers

      Browse our open positions and learn more about what it takes to be a Do’er

      Leadership

      Meet the team leading DoiT and our customers on a journey of hypergrowth

      Newsroom

      See what's new from DoiT in our latest news and announcements

  • Pricing
  • Contact us
  • Sign In
  • Products
    • Flexsave ™
    • Cloud Analytics
    • Spot Scaling
    • BigQuery Lens
    • Anomaly Detection
    • DoiT Platform
  • Services
    • How We Work
    • Stats
    • Cloud Solutions
    • FinOps
    • Areas of expertise
      • Cloud Architecture
      • Cloud Cost Optimization
      • Cloud Migration Consulting Services
      • Cloud Security
      • Data and Analytics
      • Data Management
      • DevOps with AWS & DoiT
      • Infrastructure
      • Kubernetes
      • Location Based Services
      • Machine Learning
      • Multicloud
      • Training
  • Partners
    • ISV Go-Global
    • Award-winning public cloud partner
    • DoiT Together
  • Resources
    • Blog
    • Case Studies
    • Cloud Masters Podcast
    • Ebooks and Guides
    • Events and Webinars
    • GCPInstances.info
    • Help center
    • Newsroom
    • Trust Center
    • Videos
  • Pricing
  • About
    • Careers
    • Leadership
    • Newsroom
  • Contact us
  • Sign In
Contact us
Sign in

Blog

How to Protect Your Statefulset Data with Backup for GKE

  • Felipe Amaral Felipe Amaral
  • Date: July 25, 2023
How-to-Protect-Your-Statefulset-Data-with-Backup-for-GKE

Introduction

Statefulsets are a powerful Kubernetes resource for managing stateful applications. However, StatefulSets can also be complex and difficult to back up. This can lead to data loss in the event of a failure.

Last year (2022), Google announced a new add-on feature for GKE called Backup For GKE. This is a simple and cloud-native way to protect, manage, and restore all containerized applications and data.

In this article we will go through the Backup For GKE components and how to backup volumes from a MySQL StatefulSet deployed in a GKE cluster.

Architecture

Backup for GKE consists of two main components:

  • A service that runs in Google Cloud and supports a resource-based REST API. This service serves as the control plane for Backup for GKE. The service includes Google Cloud console UI elements that interact with this API.
  • An agent that runs in every cluster where backups or restores are performed. The agent runs backup and restore operations in these clusters by interacting with the Backup for GKE API.

The Backup for GKE service integrates with the GKE UI, Google Cloud CLI, and REST APIs, providing consistent workflows for development and operations. Two forms of data are captured in a backup:

  • Config backup: a set of Kubernetes resource manifests extracted from the API server of the cluster undergoing backup, capturing the cluster state.
  • Volume backups: a set of volume backups that correspond to  PersistentVolumeClaim resources found in the config backup.

You can choose which workloads you want to back up or restore, or you can back up or restore all workloads. You can back up workloads from one cluster and restore them to another cluster. You can schedule your backups to automatically run so that you can respond quickly to recover your workloads in the event of an incident.

How To

In the next steps, we will create a zonal GKE cluster, deploy a MySQL stateful to the cluster and test the backup creation as well as the restore for our DB.

Define the variables

To make things easier, let’s create some environment variables:

export PROJECT_ID=felipe-playground-378415
export REGION=us-east1
export LOCATION=us-east1-c
export CLUSTER=backup-for-gke
export BK_PLAN_ALL_NAMESPACES=mysql-bk-plan-all-ns
export RESTORE_PLAN_ALL_NAMESPACES=mysql-bk-plan-all-ns
export BK_ALL_NAMESPACES=mysql-bk-all-namespaces
export RESTORE_ALL_NAMESPACES=mysql-restore-all-namespaces

Create GKE Cluster

Let’s set some variables to make the commands easier. Make sure you add your own  PROJECT_ID. For this exercise, we will be creating a zonal GKE cluster on the  us-east1-c zone. A zonal cluster has a single control plane in a single zone. I recommend using regional clusters for production workloads though as they offer higher availability running multiple replicas of the control plane in multiple zones within a given region. We are also using all default values for our cluster, such as default VPC & subnets, number of nodes, default service account, etc. I don’t recommend using the default values for production clusters.

Make sure you add the  --addons=BackupRestore flag in your command so the backup controllers and all the CRDs are deployed as well.

gcloud container clusters create ${CLUSTER} \
    --release-channel stable \
    --zone ${LOCATION} \
    --node-locations ${LOCATION} \
    --project ${PROJECT_ID} \
    --addons=BackupRestore

Now let’s connect to the cluster

gcloud container clusters get-credentials ${CLUSTER} \
--zone ${LOCATION} \
--project ${PROJECT_ID}

Install a StatefulSet

Now let’s create our StatefulSet. For this exercise, I chose the MySQL operator which is pretty easy and straightforward to install using Helm.

helm repo add mysql-operator https://mysql.github.io/mysql-operator/
helm repo update
helm install my-mysql-operator mysql-operator/mysql-operator \
   --namespace mysql-operator --create-namespace

Install Mysql

Install MySQL with help, please make sure to change your password!

helm install mycluster mysql-operator/mysql-innodbcluster \
   --set tls.useSelfSigned=true \
   --set credentials.root.user=safeuser \
   --set credentials.root.password=notsupersafepassword \
   --set credentials.root.host="%"

With the command,  kubectl get pods,pvc you should be able to see something around those lines

NAME                                    READY   STATUS    RESTARTS   AGE
pod/mycluster-0                         2/2     Running   0          23m
pod/mycluster-1                         2/2     Running   0          23m
pod/mycluster-2                         2/2     Running   0          23m
pod/mycluster-router-67585969f6-m9nd7   1/1     Running   0          22m

NAME                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/datadir-mycluster-0   Bound    pvc-d5214172-9b81-4d29-9bf0-9e0dcdc8f691   2Gi        RWO            standard-rwo   23m
persistentvolumeclaim/datadir-mycluster-1   Bound    pvc-df8e188a-51aa-4c4f-acb1-73daa72bc721   2Gi        RWO            standard-rwo   23m
persistentvolumeclaim/datadir-mycluster-2   Bound    pvc-defb5e6e-5946-4183-99d2-c168b05a3d6c   2Gi        RWO            standard-rwo   23m

Perfect! As the next step, we will set up the Backup for GKE resources for our cluster.

Backup for GKE includes two configuration and control resource types:

  •  BackupPlan : a parent resource for Backup resources that represent a chain of backups. This resource contains a backup configuration including the source cluster, the selection of which workloads to back up, and the region in which Backup artifacts produced under this plan are stored.
  •  RestorePlan : provides a reusable restore template. This resource contains a restore configuration including the target cluster in which you want to restore the backup, the source backup plan, the scope of the restore, conflict handling, and substitution rules.

Create a backup Plan for all namespaces

gcloud beta container backup-restore backup-plans create ${BK_PLAN_ALL_NAMESPACES} \
    --project=${PROJECT_ID} \
    --location=${REGION} \
    --cluster=projects/${PROJECT_ID}/locations/${LOCATION}/clusters/${CLUSTER} \
    --all-namespaces \
    --include-secrets \
    --include-volume-data

Create a restore plan for all namespaces

gcloud beta container backup-restore restore-plans create ${RESTORE_PLAN_ALL_NAMESPACES}\
  --all-namespaces \
  --project=${PROJECT_ID} \
  --location=${REGION} \
  --backup-plan=projects/${PROJECT_ID}/locations/${REGION}/backupPlans/${BK_PLAN_ALL_NAMESPACES} \
  --cluster=projects/${PROJECT_ID}/locations/${LOCATION}/clusters/${CLUSTER} \
  --cluster-resource-conflict-policy=use-existing-version \
  --namespaced-resource-restore-mode=delete-and-restore \
  --volume-data-restore-policy=restore-volume-data-from-backup \
  --cluster-resource-restore-scope="storage.k8s.io/StorageClass","scheduling.k8s.io/PriorityClass"

If the commands above went well, we do have all the plans in place.

Create a backup for all namespaces

gcloud beta container backup-restore backups create ${BK_ALL_NAMESPACES} \
--project=${PROJECT_ID} \
--location=${REGION} \
--backup-plan=${BK_PLAN_ALL_NAMESPACES} \
--wait-for-completion

This can take some time depending on the size of your Volumes, but you should see the message below as soon as this is done.

Backup completed. Backup state: SUCCEEDED

Test the backup restore

gcloud beta container backup-restore restores create ${RESTORE_ALL_NAMESPACES} \
  --project=${PROJECT_ID} \
  --location=${REGION} \
  --restore-plan=${RESTORE_PLAN_ALL_NAMESPACES} \
  --backup=projects/$PROJECT_ID/locations/${REGION}/backupPlans/${BK_PLAN_ALL_NAMESPACES}/backups/${BK_ALL_NAMESPACES} \
  --wait-for-completion

Restore Completed. Restore stat: SUCCEEDED

It just works!

Backup for GKE also enables you to back up a single application bundle created through the CRD ProtectedApplication, but this is for another post.

IaC

Today (Jun/2023) we only have one terraform resource to create GKE Backup Plan called  google_gke_backup_backup_plan added on the Google provider version 4.5.0 as you can see here. Hopefully, we will have more resources soon.

Pricing

Backup for GKE accrues fees along two dimensions: first, there is a GKE backup management fee, based on the number of GKE pods protected, and second, there is a backup storage fee, based on the amount of data (GB) stored. Both fees are calculated on a monthly basis, similar to other GKE feature billing.

As an example, a customer with a single backup plan in Iowa (us-central1) that backs up an average of 20 pods during a month, storing 200GB of backup storage data in Iowa, would be charged $25.60 in fees. This $25.60 would include $20 for the month for GKE backup management (20 x $1.00 / pod-month) and $5.60 for backup storage (200 * $0.028 / GB-month).

Starting June 26th 2023, new network egress charges will be introduced for backups that are stored in a different region from their source GKE cluster. These charges will be based on the source and destination region and the number of bytes transferred for each such “cross-region” backup operation

Clean UP

Delete cluster

gcloud container clusters delete ${CLUSTER} --project $PROJECT_ID --zone ${LOCATION}

Don’t forget to delete the backups and plans as well.

# Delete backup
gcloud beta container backup-restore backups delete ${BK_ALL_NAMESPACES} \
  --location=${REGION} \
  --backup-plan=${BK_PLAN_ALL_NAMESPACES}
# Delete backup plan
gcloud beta container backup-restore backup-plans delete ${BK_PLAN_ALL_NAMESPACES} \
  --location=${REGION}
# Delete restore plan
gcloud beta container backup-restore restore-plans delete ${RESTORE_PLAN_ALL_NAMESPACES} \
  --location=${REGION}

I hope you enjoy the journey and please let me know if you have any questions!

Conclusion

Overall, Backup for GKE is a powerful tool that can help you to protect your StatefulSets data. If you are using StatefulSets in Kubernetes, then you should consider using Backup for GKE to protect your data.

Resources:

MySQL :: MySQL Operator for Kubernetes Manual :: 2.1 Install using Helm Charts
Helm is an optional package manager for Kubernetes that helps manage Kubernetes applications; Helm uses charts to…
https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-installation-helm.html

Release v4.50.0 Hashicorp/terraform-provider-google
FEATURES: New Data Source: google_compute_network_peering (#13476) New Data Source: google_compute_router_nat (#13475)…
https://github.com/hashicorp/terraform-provider-google/releases/tag/v4.50.0

Pricing | Google Kubernetes Engine (GKE) | Google Cloud
Review pricing for GKE
https://cloud.google.com/kubernetes-engine/pricing#backup-for-gke

Subscribe to updates, news and more.

Subscribe

Subscribe to updates, news and more.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Related blogs

Using predefined IAM roles for enhanced Google Maps Platform governance

Examining using predefined Google Cloud IAM roles dedicated to Google Maps usage to enhance the governance of these activities.

Keep reading 
Ramp Plans Resource Hub Header1

Monitor your cloud commit attainment with DoiT Ramp Plans

DoiT Ramp Plans help you visualize, manage, and track your commit attainment so you can make sure you spend what you committed to, and act proactively.

Keep reading 
DoiT-Google-Kubernetes-Engine-Troubleshooting-Made-Simple-with-Interactive-Playbooks

Google Kubernetes Engine Troubleshooting Made Simple with Interactive Playbooks

In modern application management, Kubernetes is the foundation of container orchestration. It automates software deployment, scaling, and management, revolutionising delivery. However, growing complexity and scale pose challenges in troubleshooting and maintaining dynamic ecosystems.

Keep reading 
View all blogs
Let’s do it

From cost optimization to cloud migration, machine learning and CloudOps, we’re here to make the public cloud easy — without the costs.

Ready to get started?

Get in touch

Company

  • About us
  • Blog
  • Careers
  • MS-HT statement
  • Newsroom
  • Privacy policy
  • Terms

Offering

  • Compliance
  • Products
  • Services
  • Solutions
  • Resources

Support

  • Sign in
  • Help center
  • Open ticket
  • Contact us

Never miss an update.

Subscribe to our newsletter

Subscribe to updates, news and more.