How to set up centralized logging with the ELK stack

Security incidents rarely announce themselves in a single log file. A credential-stuffing attempt might leave traces in your load balancer logs, your application logs, and your VPN logs at the same time — and if those three systems aren't talking to each other, you'll miss the pattern until it's too late. That's the problem centralized logging solves, and the Elastic Stack (Elasticsearch, Logstash, Kibana — plus Beats for collection) remains one of the most battle-tested ways to do it without paying per-gigabyte SaaS pricing.

This guide walks through how to set up ELK stack centralized logging from a bare set of servers to a working pipeline you can actually investigate incidents with: installing the core components, shipping logs from your fleet, parsing them into structured fields, and building the dashboards and alerts a security team needs. By the end, you'll have a functioning security log aggregation pipeline and enough context to harden it for production.

Overview: What You Get When You Set Up ELK Stack Centralized Logging

Before touching a terminal, it helps to know what each piece does, because the ELK stack is really four separate products wired together:

• Elasticsearch — the search and storage engine that indexes your logs and makes them queryable in near real time.
• Logstash — the ingestion pipeline that parses, enriches, and normalizes raw log lines into structured JSON events.
• Kibana — the visualization and query layer your analysts will actually live in.
• Beats (Filebeat, Winlogbeat, Auditbeat) — lightweight shippers installed on source hosts that forward logs to Logstash or directly to Elasticsearch.

This elasticsearch logstash kibana tutorial assumes a single-node setup for a lab or small environment; for production you'll want a multi-node Elasticsearch cluster, but the pipeline logic below doesn't change.

Step 1: Install and Configure Elasticsearch

Elasticsearch stores every indexed event, so it's the foundation. On Ubuntu/Debian:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch

Edit /etc/elasticsearch/elasticsearch.yml:

cluster.name: security-logs
node.name: node-1
network.host: 0.0.0.0
discovery.type: single-node
xpack.security.enabled: true

Enable security (mandatory for anything touching log data with credentials or PII), then start the service and generate credentials:

sudo systemctl enable --now elasticsearch
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Verify it's up:

curl -k -u elastic:<password> https://localhost:9200

Step 2: Install Logstash and Build a Parsing Pipeline

Logstash is where raw log noise becomes structured, searchable events. Install it the same way as Elasticsearch (same repo), then define a pipeline in /etc/logstash/conf.d/security-logs.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  if [fileset][module] == "nginx" {
    grok {
      match => { "message" => "%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:status} %{NUMBER:bytes}" }
    }
  }
  if [event][module] == "auth" {
    grok {
      match => { "message" => "Failed password for %{USERNAME:username} from %{IPORHOST:src_ip} port %{NUMBER:port}" }
    }
    mutate { add_tag => ["auth_failure"] }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => "${ES_PASSWORD}"
    ssl_certificate_verification => false
    index => "security-logs-%{+YYYY.MM.dd}"
  }
}

Test the config before reloading production traffic through it:

sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/security-logs.conf

Step 3: Deploy Beats on Source Hosts

Rather than pointing every application at Logstash directly, install Filebeat (or Winlogbeat on Windows hosts, Auditbeat for host-level auditd/file-integrity events) on each server you want visibility into:

sudo apt install filebeat
sudo filebeat modules enable nginx system auth

Point it at Logstash in /etc/filebeat/filebeat.yml:

output.logstash:
  hosts: ["logstash-host:5044"]

filebeat.modules:
  - module: auth
    enabled: true
  - module: nginx
    access:
      enabled: true

Start it and confirm events are flowing:

sudo systemctl enable --now filebeat
sudo filebeat test output

Repeat this across every host category that matters for incident response — web servers, bastion hosts, CI runners, VPN gateways, and cloud audit logs (CloudTrail, GuardDuty) via the appropriate Beats modules or Logstash input plugins.

Step 4: Configure Kibana and Build Security Dashboards

Install Kibana, point it at your Elasticsearch cluster in /etc/kibana/kibana.yml, and start the service:

server.host: "0.0.0.0"
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "<password>"

sudo systemctl enable --now kibana

In Kibana, create a data view against security-logs-*, then build the dashboards analysts will actually use during an investigation: failed authentication attempts by source IP, top talkers by bytes transferred, HTTP 4xx/5xx spikes, and a timeline view for pivoting on a specific host or user during an incident.

Step 5: Add Detection Rules and Alerting

Raw dashboards are useful for hunting, but you also want the stack to page someone. Elastic's Security app (bundled with Kibana) ships with prebuilt detection rules, or you can write your own with Kibana Query Language:

event.module: "auth" and tags: "auth_failure"

Set a threshold rule — for example, more than 10 auth failures from a single source IP within 5 minutes — and wire the action to email, Slack, or a webhook into your SOAR/ticketing system. This is effectively how you turn ELK into a lightweight SIEM: this elk siem setup guide approach won't replace a dedicated SIEM's correlation engine at enterprise scale, but for small-to-mid-size teams it covers the same core loop of collect, detect, alert.

Step 6: Harden and Retain

Before calling the pipeline production-ready:

• Enable TLS between Beats, Logstash, and Elasticsearch, not just on the Kibana front end.
• Set an Index Lifecycle Management (ILM) policy so security-logs-* indices roll over and age into cold/frozen tiers instead of filling your disks.
• Restrict Kibana roles so analysts can query but not delete indices.
• Ship a copy of raw logs to cold storage (S3, GCS) independent of Elasticsearch, so a compromised or corrupted cluster doesn't mean losing your evidence trail.

Troubleshooting and Verification

No data appearing in Kibana. Confirm the chain link by link: filebeat test output on the source host, then check Logstash's own logs (/var/log/logstash/logstash-plain.log) for pipeline errors, then query Elasticsearch directly with curl -u elastic:<password> https://localhost:9200/security-logs-*/_search?size=1 to see if documents exist at all before assuming Kibana is misconfigured.

Grok parse failures. Events landing with a _grokparsefailure tag mean your pattern doesn't match the actual log format — a common cause is a log format that changed after an application or OS upgrade. Use the Kibana Dev Tools console or the Grok Debugger to test patterns against real sample lines before deploying.

Elasticsearch running out of disk / going read-only. Elasticsearch flips indices to read-only at 95% disk usage by default. Check with GET _cluster/settings and fix the underlying retention/ILM policy rather than just raising the watermark.

High cardinality fields killing performance. Fields like raw URLs or session IDs indexed as keyword type can bloat your mappings. Use index templates to explicitly type fields and avoid dynamic mapping explosions on high-volume security logs.

Clock skew breaking event ordering. If timelines look wrong during an investigation, check NTP sync across source hosts — Logstash's date filter trusts the timestamp in the log line, and skewed clocks will scramble your incident timeline exactly when you need it most.

How Safeguard Helps

An ELK pipeline gives you the raw material for incident response, but it only helps if the logs you're centralizing are trustworthy and complete in the first place. Safeguard focuses on the software supply chain layer that sits upstream of your log data: verifying that the build artifacts, CI/CD pipelines, and dependencies producing your applications haven't been tampered with, so the events flowing into your security-logs-* indices reflect real application behavior rather than the actions of a compromised build step or a malicious package. When your ELK stack surfaces an anomaly — an unexpected outbound connection, a new binary running in production, a CI job behaving unusually — Safeguard's provenance and integrity data gives your analysts the missing context to know whether that event traces back to a legitimate release or a supply chain compromise, cutting investigation time during exactly the incidents this pipeline is built to catch.