Reducing GKE Log Ingestion

3 min read

[Update (2023-12-20): You can now turn off workload logging in Autopilot. That is the recommended approach if you want to remove all workload logs. To disable workload logs for Autopilot (for example, if you use a third-party logging agent like DataDog), pass this value at cluster creation:

gcloud container clusters create-auto $CLUSTER_NAME \
    --logging=SYSTEM

Or update

gcloud container clusters update $CLUSTER_NAME \
    --logging=SYSTEM

The remainder of this blog post is as originally posted, and shows you how to fine-tune the logs that that you keep]

Cloud Logging is a useful tool for GKE users (check out these great blog posts on getting the most out of it). You can view all the logs from all your containers, over all clusters. I prefer it to using kubectl get logs for many tasks. For example, I can monitor all pods from a Deployment at once which especially useful since kubectl only natively allows you to stream logs from a single container at a time.

Cloud Logging comes with a decent free tier (50GiB/project/month of logs ingested) however if you exceed this quantity of logs ingested, you may start to be interested in exactly what is being logged.

In Autopilot mode, you can’t turn off cloud logging completely (update: now you can, see above), however you can exclude logs from being ingested (and thus billed). Assuming you find value in the logs, like I do, then you an also exclude just a particular set of logs that are of no value to you (or, that one particularly verbose container), while keeping the rest. Here’s how.

  1. Browse to your Log Router settings, and edit the _Default Sink.
    Log Router UI
  2. Create a new exclusion rule, and give it any name you like
  3. Add the query for the logs you wish to exclude (more on that in a moment)
    User interface for log exclusion
  4. Save the Sink

As for what to put in the query field, you can build a query using the Logs Explorer UI.

Building a query with the Logs Explorer UI

Here’s how I do it:

  1. Browse to the Logs Explorer
  2. Toggle “Log fields” if it’s not on, and select a resource type and other components to filter the logs (e.g. first select Resource Type “Kubernetes Container”, then Cluster Name “autopilot-cluster-1”, and so on) to see the logs for this query.
  3. Once you have a query displaying logs that you don’t need and wish to exclude, toggle “Show query”
  4. Copy the resulting query, and use for your log exclusion rule per the instructions above.

Here are some example queries, and their effect:

Filter a whole cluster

resource.labels.cluster_name="YOUR_CLUSTER_NAME"

Got a cluster you don’t need logs for? Filter it completely.

You can disable/enable the filter for that cluster using the “disable” button, and create multiple filters, one for each cluster you want to exclude.

Filter a specific container

resource.type="k8s_container"
resource.labels.container_name="YOUR_CONTAINER_NAME"
resource.labels.cluster_name="YOUR_CLUSTER_NAME"

Filter one specific container in a cluster, perhaps one that is generating a lot of logs. If you have a particularly noisy container, and don’t want to go and fix it, this is what I would recommend rather than filtering everything.

Filter all GKE logs

resource.type=("k8s_container" OR "k8s_cluster" OR "gke_cluster" OR "gke_nodepool" OR "k8s_node")

Filter all GKE logs, for all clusters. Effectively turns off Cloud Logging for GKE.

Filter GKE System Logs

resource.type=("k8s_cluster" OR "gke_cluster" OR "gke_nodepool" OR "k8s_node")

Filter GKE system logs (cluster, nodes) but keep the user container logs, all clusters. If you want to view your logs, but don’t care about the system (since hey, Google manages that anyway).

Filter a particular type of log for a single cluster

resource.type="k8s_container"
resource.labels.cluster_name="YOUR_CLUSTER_NAME"

Filter all logs of a particular type for a single cluster. Maybe you have a dev/test cluster that you don’t need logging for.

Filter container logs for the kube-system namespace

resource.type="k8s_container"
resource.labels.namespace_name="kube-system"

Filter out all logs from a particular namespace across all clusters. In this example, we’re filtering logs for kube-system components, which are managed by Google.

Published By
Categorized as GKE