Don’t Lose Your GPUs During Maintenance on Google Cloud

The global demand for GPUs is astronomical. Securing NVIDIA L4, A100, or H100 resources in Google Cloud Platform (GCP) often feels like a big win for your engineering team. Yet you can lose that capacity when maintenance of underlying hardware occurs, or when you manually stop your instance to perform routine tasks.

Stopping your VM releases your resources, whether it is a scheduled Google Cloud maintenance event or a user-initiated stop to patch a driver, rotate an image, or modify instance scopes. This throws your GPU back into the public pool, leaving your critical ML training runs or inference workloads stranded. Here is how to lock down your GPUs and ensure they survive infrastructure updates.

Important Restrictions and Edge Cases

While reservations are the standard defense against capacity loss, you must account for specific GCP architectural rules:

• Restricted GPU Shapes: Due to extreme supply constraints, some newer or highly sought-after GPU shapes cannot be reserved using standard on-demand capacity reservations. For example, A4X Max, A3 Ultra, and A3 High configurations with fewer than eight GPUs currently have restrictions. Always verify reservation eligibility in the GCP documentation.
• Spot VMs vs. CUDs: Reservations protect standard VM workloads. They do not protect Spot VMs, which cannot consume reservations and are preemptible by design.
• CUDs: Committed Use Discounts affect pricing; they do not themselves reserve capacity. Note that CUDs are charged for VMs but also for inactive reservations.
• GKE Integration: Compute Engine reservations can protect GPU capacity used by Google Kubernetes Engine (GKE). In GKE Standard, configure the node pool with reservation affinity. In GKE Autopilot, you must explicitly select a reservation or configure a supported custom ComputeClass.

How to Implement GPU Reservations for User Maintenance

The safest approach to prevent configuration drift is to define the VM once in an instance template and create the reservation from that template. By default, reservations are set to "Consume automatically". Any VM created in your project that exactly matches the reservation's specifications will automatically claim it.

1. Create the Reservation

If you are performing rolling maintenance across a cluster (such as node-by-node OS patching), you do not need to reserve capacity for the entire fleet. Size your reservation to match the exact number of instances you plan to bring down simultaneously. For example, if you drain and update two nodes at a time, set your --vm-count to 2 to catch those specific instances as they bounce.

You can provision the reservation using an instance template to ensure exact matching using the command line:

gcloud compute reservations create gpu-maintenance-reservation \
    --project=my-project-id \
    --source-instance-template=projects/my-project-id/global/instanceTemplates/my-gpu-template \
    --vm-count=1 \
    --zone=us-central1-c

Alternatively, you can define the reservation directly:

gcloud compute reservations create reservation-v100 \
    --project=joshua-playground \
    --zone=us-east1-c \
    --machine-type=n1-standard-1 \
    --accelerator=count=1,type=nvidia-tesla-v100 \
    --vm-count=1 \
    --require-specific-reservation=false

And of course this can be done in the Google Cloud Console.

2. Deploy Your VM and Verify Consumption

Because --require-specific-reservation is set to false, you don't need to change your VM deployment scripts. As long as you launch an exact match in the same zone, GCP will automatically slot the VM into your reserved capacity.

To confirm that the expected number of instances are consuming the reserved slots, check the reservation metrics in the Google Cloud Console. You can view the current reservations in Compute Engine > Reservations, or see historical use in Google Cloud Monitoring, using metrics compute.googleapis.com/reservation/used and compute.googleapis.com/reservation/reserved.

3. Clean Up Your Reservation

Unlike standard VMs, capacity reservations must be explicitly deleted when you no longer need them. If you permanently retire your workload and delete the VM, but forget to delete the reservation, you will continue paying for the empty GPU capacity at the full on-demand rate; and CUD's will be billed.

When you are done with the reservation, remove it using the command line:

gcloud compute reservations delete gpu-maintenance-reservation \
    --project=my-project-id \
    --zone=us-central1-c \
    --quiet

The Financials

Reservations in GCP do not cost extra, though you pay for the reserved resources whether they are actively attached to a running VM or sitting idle. Because you are already paying for the active GPU VM, wrapping it in a reservation is cost-neutral. You only incur idle costs during the brief minutes the VM is rebooting during maintenance, making this a cheap insurance policy against losing your hardware.

To verify exactly how much you paid for the reserved capacity while your instances were stopped, you can use DoiT Cloud Analytics. Filter your reports by the compute.googleapis.com/reservation SKU to audit the exact cost of the guaranteed capacity during your downtime.

By utilizing GCP reservations, you transform hardware availability from a game of chance into a predictable guarantee. Because reservations cost nothing extra on top of your underlying compute, failing to wrap your critical GPU instances in a reservation is an unnecessary risk. Provision the exact capacity you need in your target zone, deploy from a matched instance template, and ensure that routine restarts or unavoidable Google maintenance events never leave your workloads stranded.