In April 2024, a misconfigured admission controller let an attacker bypass pod security restrictions on a publicly exposed Kubernetes API server — a class of failure tracked under CVE-2024-3177 — and the breach wasn't caught by the security team but by an on-call engineer who noticed unusual kubectl exec activity in audit logs three days later. That gap between compromise and detection is the entire argument for Kubernetes monitoring. A cluster running 200 microservices across 40 nodes generates control-plane events, container metrics, network flows, and audit logs at a volume no human can read line by line. Without instrumentation that correlates those signals — resource usage, API server calls, pod lifecycle events, and runtime syscalls — teams find out about incidents from customers, not dashboards. This guide covers what to monitor, which tools do it, and where monitoring alone stops being enough for security.
What is Kubernetes monitoring, and why is it different from monitoring a VM fleet?
Kubernetes monitoring is the practice of collecting metrics, logs, and events from four distinct layers — nodes, the control plane, workloads, and the network — because a single "server is slow" alert in a VM world maps to a dozen possible root causes in Kubernetes. A pod can be evicted because a node hit its 80% memory eviction threshold, because a LimitRange capped it below what the JVM needed, or because the scheduler bin-packed it onto a node the cluster autoscaler is about to terminate. In a VM, you monitor the box. In Kubernetes, you monitor the scheduler's decisions — kube-scheduler events, kube-apiserver latency (Google's SRE guidance flags anything over 1 second for mutating calls as a red flag), etcd disk fsync duration (etcd's own docs recommend keeping this under 10ms), and kubelet-reported node conditions like DiskPressure and PIDPressure. CNCF's 2023 annual survey found 84% of respondents running Kubernetes in production, up from 58% in 2020, and support ticket volume tied to "silent" cluster degradation has scaled with it.
What should you actually monitor in a production cluster?
You should monitor four categories: node resource saturation, control-plane health, workload behavior, and audit/security events, because each one fails independently and produces different symptoms. Node-level metrics — CPU throttling (container_cpu_cfs_throttled_periods_total), memory pressure, and disk I/O — come from cAdvisor and kubelet and explain the majority of OOMKilled and Evicted pod states. Control-plane health means watching etcd leader elections (a cluster with more than one election per hour is unstable), API server error rates by verb and resource, and admission webhook latency, since a slow webhook can add 500ms+ to every pod creation cluster-wide. Workload behavior covers restart counts — a pod crossing 5 restarts in 10 minutes trips Kubernetes' own CrashLoopBackOff backoff cap at 5 minutes between attempts — and readiness/liveness probe failure rates. Audit and security events are the layer most teams skip: the Kubernetes audit log (enabled via --audit-log-path on the API server) records every create, exec, and delete call, including kubectl exec -it into a pod, which is exactly the signal that caught the Tesla Kubernetes dashboard cryptomining incident in 2018 after the fact rather than in real time.
Which tools handle Kubernetes monitoring in 2026?
Prometheus paired with kube-state-metrics remains the default metrics stack, because it graduated as a CNCF project in 2018 and every major managed Kubernetes offering (EKS, GKE, AKS) ships Prometheus-compatible metrics endpoints out of the box. Prometheus scrapes time-series data — pod counts, deployment replica drift, PVC usage — while kube-state-metrics translates the API server's object state into metrics Prometheus can query, answering questions like "how many pods have been Pending for over 5 minutes." For logs, Fluent Bit or Fluentd ship container stdout/stderr and kubelet logs to a backend like Loki or Elasticsearch; Fluent Bit's DaemonSet footprint is deliberately small — under 650KB binary size — because it runs on every node. For runtime security monitoring specifically, Falco (created by Sysdig in 2016, CNCF-graduated in April 2024) uses eBPF or a kernel module to alert on syscall patterns like a shell spawned inside a container or a write to /etc/shadow. Datadog, Grafana Cloud, and New Relic offer managed versions of this same stack for teams that don't want to run Thanos or Cortex for long-term Prometheus storage.
How does Kubernetes monitoring connect to detecting an active attack?
Kubernetes monitoring connects to attack detection through the audit log and runtime syscall monitoring, because most Kubernetes-specific attacks — privilege escalation via an over-permissioned ServiceAccount, container escape, or lateral movement through the CNI network — leave a signature in one of those two places before they show up as resource anomalies. The 2020 Tesla incident and the 2023 disclosure of exposed Kubernetes dashboards at multiple Fortune 500 companies both trace back to the same root cause: an unauthenticated or over-permissioned API server endpoint that standard resource monitoring (CPU, memory) never flags, because cryptomining containers are deliberately throttled to stay under the radar. Effective detection requires alerting on RBAC anomalies — a ServiceAccount that suddenly calls list secrets across namespaces it's never touched — and on Falco rule violations like Contact K8S API Server From Container, which fires when a pod tries to talk to the API server directly instead of through its intended service, a common sign of a compromised workload probing for lateral movement.
What are the most common Kubernetes monitoring gaps that lead to incidents?
The most common gap is monitoring workloads without monitoring the software running inside them, because a dashboard full of green CPU and memory graphs says nothing about whether a container is running a base image with a known-exploitable CVE. A 2023 Sysdig cloud threat report found that 87% of container images analyzed had high or critical vulnerabilities, yet standard Kubernetes monitoring stacks — Prometheus, Grafana, Fluent Bit — have no visibility into image contents at all; they monitor the container's behavior, not its composition. The second gap is alert fatigue from unscoped thresholds: teams that set a single CPU alert threshold across a 200-node cluster with mixed workload types generate hundreds of false positives a week and eventually mute the channel, which is how genuine anomalies get missed. The third is treating the audit log as a compliance checkbox instead of a live feed — many clusters enable audit logging to satisfy SOC 2 but never wire it into an alerting pipeline, so the data exists only for after-the-fact forensics.
How Safeguard Helps
Safeguard extends Kubernetes monitoring beyond cluster metrics into the software supply chain that actually runs inside your pods. Safeguard generates and ingests SBOMs for every container image deployed to a cluster, so when a new CVE drops you know within minutes which running workloads are affected — not just which images exist in a registry. Reachability analysis then filters that list down to the vulnerabilities your code paths can actually invoke, cutting the noise that causes teams to ignore CVE alerts in the first place. Griffin AI correlates that vulnerability and reachability data with runtime and RBAC signals to prioritize what's exploitable right now, and Safeguard's auto-fix PRs open a base-image or dependency bump directly against your manifest repo so the fix ships through the same CI/CD path your monitoring stack already watches.