Like many public cloud providers, Google Cloud follows a network pricing model which doesn’t charge you for ingress traffic but will charge you for egress traffic to other zones, regions and internet.
It’s not an easy task to track, at scale, the amount of traffic going out to other zones, regions or internet. To avoid “cloud bill shock”, we suggest to start using a new feature (still in beta) called VPC Flow Logs.
The VPC Flow Logs records network traffic sent to or from your instances. These logs can be used for network monitoring, forensics, real-time security analysis, and cloud cost optimization.
You can access VPC Flow Logs in Stackdriver Logging and, as with any logs in Google Cloud, you can export them automatically to supported destinations such as Cloud Pub/Sub, BigQuery and Cloud Storage.
VPC Flow Logs are aggregated by connection, at 5-second intervals and exported in near real time. VPC Flow Logs are enabled per each subnet and they don’t have performance impact on your network:
gcloud beta compute networks subnets update [subnet-name] \ --no-enable-flow-logs
Now lets create an export of the logs. Navigate to https://console.cloud.google.com/logs/viewer and select advanced filter then paste :
resource.type="gce_subnetwork" logName="projects/{#project_id}/logs/compute.googleapis.com%2Fvpc_flows"
Then, setup an export to Google BigQuery:
Now all the logs are streamed into a Big Query table and we can visualize them. For this task, we have choosen Redash as our visualization tool. Redash is an open source tool to query databases, visualize the results, and create dashboards. It supports many data sources, including BigQuery.
The table produced by the export has fairly simple and self-explanatory schema, here is the query we have used to extract the data to track egress traffic:
With Redash, it was easy to build visualizations of our egress traffic to internet:
Egress traffic between regions:
Finally, the egress traffic between zones:
We can now monitor our egress traffic and even setup alarms if the egress traffic passes a threshold.
Want more stories? Check our blog, or follow Aviv on Twitter.