Homelab Monitoring Explained: Prometheus, Grafana, Alerts, Safe Exposure

Homelab monitoring means scraping metrics from every host, disk, UPS, switch and service into Prometheus, alerting on a short list of symptoms through Alertmanager, and reading it all in Grafana. Keep those endpoints on your private network, protect them with TLS and basic auth via the Prometheus web config file, and reach them from outside only over a VPN.

This post goes one level deeper than our tiny data center at home guide. It covers what to monitor, how Prometheus works under the hood, the exporters that cover typical homelab gear, alert routing without fatigue, Grafana data sources and plugin risk, a full Docker Compose build with a disk space alert and a certificate expiry check, logs and retention with worked storage estimates, and a comparison of pull metrics, push metrics, logs and uptime checks. For the exposure question on its own, see is a Prometheus style metrics endpoint safe to expose?, and for the alert pipeline in more depth, Alertmanager grouping, inhibition and silences explained. Every setting below comes from the official project documentation linked in the sources.

Key takeaways

• Prometheus collects time series "via a pull model over HTTP", according to its own overview. It scrapes targets on a schedule, stores samples locally and evaluates alerting rules. Pushing is supported only through an intermediary gateway, and the project recommends that gateway only in limited cases.
• A homelab needs signals at six layers: host health, disks (space and SMART), power (UPS), network, services and containers, and the things that fail quietly (certificates and backups).
• Five exporters cover most gear: node exporter (port 9100) for Linux hosts, SNMP exporter (9116) for switches and access points, blackbox exporter (9115) for HTTP, TCP, ICMP and DNS probes including TLS certificate expiry, smartctl exporter (9633) for drive health, and a community Network UPS Tools exporter for the UPS.
• Alertmanager groups, deduplicates, routes, inhibits and silences alerts. Its defaults are group_wait 30s, group_interval 5m and repeat_interval 4h. At home, the biggest wins against noise are a for duration on every rule, fewer and better rules, inhibition of downstream alerts and quiet hours through time_intervals.
• The Prometheus security model says the HTTP endpoints of Prometheus components "should not be exposed to publicly accessible networks like the internet". Alertmanager serves its v2 API with Access-Control-Allow-Origin: *, and the SNMP and blackbox exporters take their targets from URL parameters, so anyone who reaches them can make them send requests elsewhere.
• Prometheus and the official exporters support TLS and bcrypt hashed basic auth through a YAML file passed with --web.config.file. The docs still label this experimental. The minimum TLS version defaults to TLS 1.2.
• Grafana's default admin user is admin with password admin, set on first run. Change it, keep allow_sign_up off, set cookie_secure when you serve over HTTPS, and do not load unsigned plugins.
• Prometheus stores "an average of only 1-2 bytes per sample". Default retention is 15 days. A six host homelab with a switch and a handful of probes fits 90 days of 15 second metrics in under 9 GB.

What should a homelab monitor?

A homelab fails in predictable places. Disks fill or wear out, a UPS battery ages until it cannot hold the load for the shutdown sequence, a switch port flaps, a container restarts in a loop, a certificate expires on a Sunday, and a backup job stops running without telling anyone. The table maps each layer to the signal that matters and the component that collects it.

Two rows deserve emphasis because they are the ones people skip. Certificate expiry is easy to monitor and embarrassing to miss. Backups are the only row where the absence of an event is the failure, so the job itself has to write a success timestamp that Prometheus can check. The Prometheus alerting practices page suggests a batch job threshold of "at least enough time for 2 full runs", which for a nightly backup means roughly two days plus the job's own run time. The node exporter README describes the textfile collector for exactly this: a cron job writes a .prom file with a completion time, and the collector exposes it as a metric "tied to a machine".

If you already run Network UPS Tools as described in our homelab UPS guide, an exporter for it turns ups.status and battery runtime into time series you can alert on. If your storage is a ZFS box, as in our home NAS guide, pool health and scrub results belong on the same dashboard as SMART data.

How does Prometheus work?

Prometheus is an open source monitoring and alerting toolkit originally built at SoundCloud. Its overview page says the project began in 2012 and joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. It is now maintained independently of any company.

The pull model

The Prometheus server holds a list of targets, from static configuration or from service discovery. On every scrape interval it sends an HTTP GET to each target's metrics path, usually /metrics, reads the text response and stores each line as a sample with a timestamp. The overview lists the key features: a multi dimensional data model of time series identified by metric name and key value labels, PromQL, single server nodes with "no reliance on distributed storage", collection via pull over HTTP, push through an intermediary gateway, and target discovery through service discovery or static configuration.

Pull has a property that matters at home. Every scrape produces an up metric for the target, 1 if the scrape succeeded and 0 if it did not. The Pushgateway documentation points out that you lose this "automatic instance health monitoring via the up metric" when you push instead. With pull, a dead host is visible by default.

Exporters and the exposition format

Most homelab gear does not speak Prometheus natively, so an exporter sits beside it, talks to the device in its own language (sysfs and procfs, SNMP, smartctl JSON, the NUT protocol) and serves the result over HTTP. Prometheus' own text format and the OpenMetrics 1.0 specification define what that response looks like. A minimal response looks like this:

# HELP node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 4.2133504e+10

OpenMetrics 1.0 is published on prometheus.io with "Status: Published". It says that for pull based exposition over HTTP, version negotiation uses standard HTTP content type negotiation and must default to the oldest version, 1.0.0, if no newer one is requested.

Time series and labels

A time series is a metric name plus a unique set of labels. node_filesystem_avail_bytes with mountpoint="/" is one series and the same metric with mountpoint="/srv" is another. Prometheus adds job and instance labels at scrape time. Every new label value creates a new series, so labels with unbounded values (user IDs, full URLs, timestamps) multiply storage and memory. In a homelab the series count is modest, and the storage estimate later in this post shows how to measure it.

PromQL in five queries

PromQL is the query language for both dashboards and alerting rules. These five patterns cover most homelab needs:

# 1. Which targets are down right now?
up == 0

# 2. Percent of filesystem space free, ignoring pseudo filesystems
100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
    / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}

# 3. Will this filesystem run out within 24 hours at the 6 hour trend?
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0

# 4. Network receive rate per interface in bits per second
rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m]) * 8

# 5. Days until the earliest certificate on each probed endpoint expires
(probe_ssl_earliest_cert_expiry - time()) / 86400

rate() works on counters and handles counter resets after a restart. predict_linear() fits a linear regression over the range and projects it forward by the given number of seconds. Both are documented on the PromQL functions page.

Retention and storage

The storage documentation describes the local database. Ingested samples are grouped into two hour blocks, each a directory with a chunks subdirectory, an index and a metadata file. The current block lives in memory, protected by a write ahead log stored in 128 MB segments. Background compaction merges blocks into larger ones covering up to 10 percent of the retention time or 31 days, whichever is smaller.

The two retention flags are --storage.tsdb.retention.time and --storage.tsdb.retention.size. If neither is set, retention defaults to 15 days. The storage page is frank that local storage "is not clustered or replicated" and should be managed "like any other single node database". For backups it recommends snapshots, because a backup taken without one can miss recent data. Snapshots are an admin API call, and the admin API is disabled by default for good reason (see the exposure section), so plan backups knowingly.

Which exporters cover common homelab gear?

Each project documents its own defaults. The table summarises what each exporter's README or the Prometheus port allocation page says, plus the caution that matters at home.

node exporter

The node exporter README says it listens on HTTP port 9100 by default and is "designed to monitor the host system". Running it in a container "requires extra care in order to avoid monitoring the container itself". The documented container recipe uses host networking, the host PID namespace and a read only bind mount of the host root at /host, with --path.rootfs=/host. Collectors such as filesystem, meminfo, netdev and diskstats are enabled by default; others such as systemd are disabled by default and switched on with --collector.systemd. Disable any default collector with --no-collector.<name>.

SNMP exporter

The SNMP exporter README recommends starting with the if_mib module and the public_v2 auth module against switches, access points and routers, using a read only community. It adds that SNMP community strings "are not considered secrets", since v1 and v2c send them unencrypted, and that "for secure access, SNMP v3 is required". Prometheus scrapes it at /snmp?target=<device>&module=if_mib, so one exporter covers every device on your network.

blackbox exporter

The blackbox exporter probes endpoints "over HTTP, HTTPS, DNS, TCP, ICMP and gRPC". You define modules (for example http_2xx) in blackbox.yml and Prometheus calls /probe?target=<url>&module=http_2xx. Every successful HTTPS probe also reports probe_ssl_earliest_cert_expiry, a Unix timestamp for certificate expiry in the chain the server presented (the exporter's help text calls it the "SSL chain expiry in unixtime"), which is what makes certificate alerts a single PromQL line. Its configuration reference also shows fail_if_not_ssl (default false) and that preferred_ip_protocol defaults to ip6, which surprises people on IPv4 only home networks.

smartctl exporter

The smartctl exporter, under the prometheus-community organisation, needs smartmontools 7.0 or later because that release introduced JSON output. It listens on :9633 by default, scans for devices itself unless you list them with --smartctl.device, polls every 60 seconds by default and rescans for new devices every 10 minutes. Its README shows a privileged, root container example, which is the price of raw disk access. Run it as a native service on the storage host if you prefer to avoid a privileged container.

NUT exporter

The Prometheus exporters page lists a Network UPS Tools exporter among third party exporters, and the default port allocation wiki lists more than one NUT exporter project on different ports. None is maintained by the Prometheus project. Pick one, read its code and README, pin a release, and point it at your upsd with a read only NUT user.

How does Alertmanager route, group and silence alerts?

Prometheus evaluates alerting rules and sends firing alerts to Alertmanager. The Alertmanager docs say it handles "deduplicating, grouping, and routing them to the correct receiver integration such as email", and "also takes care of silencing and inhibition of alerts". All of it is configured in one YAML file, except silences, which you create in the web interface or with amtool.

Routing tree

The route block is a tree. The top level route catches everything and names a default receiver. Child routes match on labels (for example severity="critical") and send to a different receiver. The first matching child wins unless you set continue: true.

Grouping and its timers

Grouping "categorizes alerts of similar nature into a single notification". You choose the labels in group_by. Three timers control the pace, and the configuration reference documents their defaults and behaviour:

Two details from the reference are worth knowing. If an alert resolves before group_wait ends, no notification is sent for it, which "reduces noise of flapping alerts". And repeat_interval is checked after each group_interval, so it is rounded up to the next multiple of group_interval.

Inhibition

Inhibition suppresses notifications for some alerts while another alert is firing. The documented example mutes alerts for a cluster while an alert says the whole cluster is unreachable. The homelab version: when the UPS is on battery or the core switch is down, do not also page for every host that becomes unreachable.

Silences

A silence mutes alerts that match a set of label matchers for a fixed time. Use one before maintenance: amtool --alertmanager.url=http://127.0.0.1:9093 silence add instance="nas:9100" --duration=2h --comment="disk swap". Silences live in Alertmanager's own data, not in its configuration file.

How to avoid alert fatigue at home

A home setup has one on call engineer who also has a day job and a family. The Prometheus alerting practices page argues for keeping alerting simple, alerting on symptoms rather than every possible cause, and having good consoles to find causes once you are paged. Translated for a homelab:

1. Give every rule a for duration. The alerting rules docs explain that for keeps an alert in the pending state until the condition has held through each evaluation for that long. Rules without it fire on the first evaluation.
2. Use keep_firing_for on flappy signals. It keeps an alert firing for the stated time after the condition was last met, which the docs say helps prevent flapping and false resolutions.
3. Two severities, two channels. critical goes to your phone. warning goes to email or a chat channel you read once a day.
4. Quiet hours. time_intervals (which replaced the deprecated mute_time_intervals) lets a route stay silent overnight for warnings while critical alerts still get through.
5. Inhibit downstream noise. One "UPS on battery" alert should not bring ten "host down" alerts with it.
6. Watch the watcher. The alerting practices page asks for metamonitoring, alerts that confirm "Prometheus servers, Alertmanagers, PushGateways, and other monitoring infrastructure are available and running correctly". Add one rule that always fires and route it to a receiver that expects it every few minutes, such as an external heartbeat check you run on a different network. If the heartbeat stops, the monitoring stack itself is down.
7. Delete rules you ignore. If an alert fired three times last month and you did nothing each time, it is a dashboard panel, not an alert.

How do Grafana dashboards connect to data sources?

Grafana does not store your metrics. It stores dashboards, users and data source definitions, and it queries the data source each time a panel loads. For Prometheus, the data source is the Prometheus server URL, and queries go through the Grafana server (the data source proxy) rather than straight from your browser.

You can create data sources in the UI or provision them from YAML files. Grafana's provisioning docs say environment variable lookups work "in all provisioning configuration" with $ENV_VAR_NAME or ${ENV_VAR_NAME}, and they warn that a password containing $ should use the $ENV_VAR_NAME form to avoid double expansion. Secrets go under secureJsonData, which Grafana encrypts with its secret_key. The configuration reference says secret_key uses AES-256 in CFB mode and cannot be changed without re-encoding stored data source settings.

Authentication risks

The configuration reference lists the defaults to change on day one:

The Prometheus security model adds a subtle point for Grafana users: "dashboard permissions are not data source permissions". Anyone who can use a Prometheus data source in Grafana can run arbitrary PromQL through it, whatever the dashboard permissions say. At home that means a viewer account can read every metric you store.

The data source proxy as a network hop

Grafana's "Configure security" page warns that non Grafana web services on the Grafana server or its local network "might be vulnerable to exploitation through the Grafana data source proxy". The documented precautions are an allowlist of data source URLs (data_source_proxy_whitelist), firewall rules that stop Grafana from reaching sensitive internal services, and a proxy server. In a homelab, where Grafana often shares a flat network with a router admin page, a NAS and a hypervisor console, the firewall rule is the practical one.

Plugin risks

Grafana supports panel, data source and app plugins. Its plugin management docs describe signature verification as "a security measure to make sure plugins haven't been tampered with", and the configuration reference says allow_loading_unsigned_plugins takes a list of plugin IDs to load even if unsigned, that "plugins with modified signatures are never loaded", and that Grafana does "not recommend using this option". Grafana also documents a Plugin Frontend Sandbox that runs plugin frontend code in a separate JavaScript context. A plugin is third party code running inside the server that holds your data source credentials, so install few, prefer signed ones, pin versions and remove what you do not use.

What should never be exposed to the internet?

The Prometheus security model starts with a warning that its HTTP endpoints "should not be exposed to publicly accessible networks like the internet (unless you know what you are doing and have taken appropriate measures)". It names the /metrics endpoint of instrumented binaries, the API endpoints of server components and the /pprof endpoint of Go components, and adds that it is "easily possible to overload and ultimately DoS servers" through them. The table shows why each endpoint is sensitive, drawn from that page and the component docs.

TLS, basic auth and the web config file

Prometheus, Alertmanager and the official exporters share the exporter toolkit's web configuration file, passed with --web.config.file. The Prometheus HTTPS page says the file "is read upon every http request", so certificate and password changes take effect immediately, and it labels the feature experimental. A minimal file:

# web-config.yml
tls_server_config:
  cert_file: /etc/prometheus/tls/server.crt
  key_file: /etc/prometheus/tls/server.key
  min_version: TLS12
basic_auth_users:
  # bcrypt hash, never the plain password
  prometheus: $2y$10$REPLACE_WITH_YOUR_BCRYPT_HASH
http_server_config:
  headers:
    X-Content-Type-Options: nosniff
    X-Frame-Options: deny

Generate the hash with htpasswd -nBC 10 "" | tr -d ':\n', the command given in the exporter toolkit docs, which also note that a cost of 10 takes around 70 ms per check on typical hardware and that higher costs slow every request. The security model says basic auth passwords are stored as bcrypt hashes, that basic auth without TLS exposes credentials "in cleartext over the network", and that the toolkit defaults to TLS 1.2 as the minimum version. For client certificate authentication, the HTTPS page warns that RequireAndVerifyClientCert is the only client_auth_type value that is not insecure. The same file can also set a rate_limit on the endpoint.

On the Prometheus side, a scrape job that talks to a protected exporter uses scheme: https, basic_auth with a password_file, and tls_config with your CA file. Avoid insecure_skip_verify, which the security model calls out as an option that skips verification.

How to look at dashboards from outside

The pattern that keeps everything above private is simple:

1. Bind each service to localhost or a private address, not to all interfaces. In Compose, publish ports as 127.0.0.1:9090:9090.
2. Firewall exporter ports so only the Prometheus host can reach them.
3. Reach Grafana from your phone or laptop through a VPN into the home network. Nothing listens on the public address except the VPN.
4. If you must publish Grafana through a reverse proxy, put an authentication layer in front of it, serve it over HTTPS, set cookie_secure = true, and do not publish Prometheus, Alertmanager or any exporter the same way. The security model notes there is no built in CSRF protection on admin and mutating endpoints, so block those paths at the proxy.

Step by step: Prometheus, node exporter, Alertmanager and Grafana with Docker Compose

This build runs on one Linux host (call it mon) with Docker Engine and the Compose plugin. It monitors mon itself, probes two HTTPS services for status and certificate expiry, alerts on disk space, and sends notifications by email. Image tags are the latest releases at the time of writing (Prometheus v3.15.1, Alertmanager v0.34.1, node exporter v1.12.1, blackbox exporter v0.29.1 and Grafana 13.2.4); check each project's releases page and pin what you test.

Step 1: lay out the directory

mkdir -p ~/monitoring/{prometheus/rules,alertmanager,blackbox,grafana/provisioning/datasources,secrets}
cd ~/monitoring
chmod 700 secrets

Step 2: create secrets

# Grafana admin password and the password Grafana and Prometheus use for each other
openssl rand -base64 24 > secrets/grafana_admin_password
openssl rand -base64 24 > secrets/prometheus_password
openssl rand -base64 24 > secrets/node_password
chmod 600 secrets/*

# bcrypt hashes for the web config files (htpasswd is in apache2-utils or httpd-tools)
PROM_HASH=$(htpasswd -nbBC 10 "" "$(cat secrets/prometheus_password)" | tr -d ':\n')
NODE_HASH=$(htpasswd -nbBC 10 "" "$(cat secrets/node_password)" | tr -d ':\n')

Step 3: web config files for Prometheus and node exporter

Basic auth without TLS sends credentials in cleartext, so on a single host where traffic stays on the loopback or Docker bridge this is acceptable; across your LAN, add the tls_server_config block shown earlier.

cat > prometheus/web-config.yml <<EOF
basic_auth_users:
  grafana: '${PROM_HASH}'
EOF

cat > node-web-config.yml <<EOF
basic_auth_users:
  prometheus: '${NODE_HASH}'
EOF

Step 4: prometheus.yml

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    basic_auth:
      username: grafana
      password_file: /run/secrets/prometheus_password
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    basic_auth:
      username: prometheus
      password_file: /run/secrets/node_password
    static_configs:
      # node exporter runs in the host network namespace;
      # host.docker.internal is mapped to the host in compose.yml
      - targets: ["host.docker.internal:9100"]
        labels:
          host: mon

  - job_name: blackbox_https
    metrics_path: /probe
    params:
      module: [https_2xx]
    static_configs:
      - targets:
          - https://nas.home.example:8443
          - https://git.home.example
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

  - job_name: alertmanager
    static_configs:
      - targets: ["alertmanager:9093"]

The three relabel_configs lines are the standard blackbox pattern from its README: the listed URL becomes the target parameter and the instance label, and the scrape itself goes to the blackbox exporter.

Step 5: blackbox.yml

# blackbox/blackbox.yml
modules:
  https_2xx:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: ip4
      fail_if_not_ssl: true
      valid_status_codes: []   # empty means any 2xx
      tls_config:
        insecure_skip_verify: false

Setting preferred_ip_protocol: ip4 avoids failed probes on networks without IPv6, since the documented default is ip6. If your internal services use certificates from your own CA, add ca_file under tls_config and mount the CA into the container rather than skipping verification.

Step 6: the disk space and certificate rules

# prometheus/rules/homelab.yml
groups:
  - name: homelab
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} target {{ $labels.instance }} is down"

      - alert: FilesystemAlmostFull
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}
          ) < 0.10
          and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs"} == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} has under 10% free"

      - alert: FilesystemWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
          and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"} < 0.25
        for: 30m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will fill within 24h at the current rate"

      - alert: ProbeFailing
        expr: probe_success == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} probe failing"

      - alert: CertificateExpiresSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate for {{ $labels.instance }} expires in {{ $value | humanize }} days"

      - alert: Heartbeat
        expr: vector(1)
        labels:
          severity: heartbeat
        annotations:
          summary: "Always firing; proves the alert pipeline works"

The second disk rule adds a 25 percent free guard so a large, slowly filling volume does not page you a year in advance. The certificate rule warns at 14 days, which leaves time even when automatic renewal has silently broken.

Step 7: alertmanager.yml

# alertmanager/alertmanager.yml
global:
  smtp_smarthost: "smtp.example.net:587"
  smtp_from: "alerts@home.example"
  smtp_auth_username: "alerts@home.example"
  smtp_auth_password_file: /run/secrets/smtp_password
  smtp_require_tls: true

route:
  receiver: email
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: ['severity="heartbeat"']
      receiver: heartbeat
      repeat_interval: 5m
    - matchers: ['severity="critical"']
      receiver: email
      repeat_interval: 4h
    - matchers: ['severity="warning"']
      receiver: email
      active_time_intervals: [daytime]

time_intervals:
  - name: daytime
    time_intervals:
      - times:
          - start_time: "08:00"
            end_time: "22:00"
        location: "Local"

inhibit_rules:
  - source_matchers: ['alertname="TargetDown"', 'job="node"']
    target_matchers: ['severity="warning"']
    equal: ["instance"]

receivers:
  - name: email
    email_configs:
      - to: "you@home.example"
        send_resolved: true
  - name: heartbeat
    webhook_configs:
      # an external heartbeat endpoint you run outside the homelab
      - url: "https://heartbeat.example.net/ping/REPLACE"

The warning route only notifies between 08:00 and 22:00 local time. The inhibition rule suppresses warnings for a host whose node exporter is already down, since the warnings are noise once you know the host is gone. Replace Local with your IANA time zone name if the container's local time is UTC.

Step 8: Grafana data source provisioning

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: $PROMETHEUS_PASSWORD

Step 9: compose.yml

# compose.yml
services:
  prometheus:
    image: prom/prometheus:v3.15.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.config.file=/etc/prometheus/web-config.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=90d
      - --storage.tsdb.retention.size=20GB
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prometheus-data:/prometheus
    secrets: [prometheus_password, node_password]
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "127.0.0.1:9090:9090"
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.34.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager:/etc/alertmanager:ro
      - alertmanager-data:/alertmanager
    secrets: [smtp_password]
    ports:
      - "127.0.0.1:9093:9093"
    restart: unless-stopped

  blackbox:
    image: prom/blackbox-exporter:v0.29.1
    command:
      - --config.file=/etc/blackbox/blackbox.yml
    volumes:
      - ./blackbox:/etc/blackbox:ro
    restart: unless-stopped
    # no published ports: only Prometheus on the compose network can reach it

  node-exporter:
    image: prom/node-exporter:v1.12.1
    command:
      - --path.rootfs=/host
      - --web.config.file=/etc/node/web-config.yml
    network_mode: host
    pid: host
    volumes:
      - /:/host:ro,rslave
      - ./node-web-config.yml:/etc/node/web-config.yml:ro
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.2.4
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      PROMETHEUS_PASSWORD: ${PROMETHEUS_PASSWORD}
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "127.0.0.1:3000:3000"
    restart: unless-stopped

secrets:
  prometheus_password:
    file: ./secrets/prometheus_password
  node_password:
    file: ./secrets/node_password
  smtp_password:
    file: ./secrets/smtp_password

volumes:
  prometheus-data:
  alertmanager-data:
  grafana-data:

The node exporter block follows the README's documented Compose recipe (host network, host PID, read only rslave bind mount of /), with the web config file added. Because it uses the host network, it listens on port 9100 on every host interface. Restrict that with a host firewall so only the Docker bridge can reach it, for example with nftables or ufw rules that allow 9100 from your Docker subnet and drop it from everywhere else. Every other service publishes only on 127.0.0.1, and the blackbox exporter publishes nothing.

Step 10: validate and start

cd ~/monitoring
{
  echo "PROMETHEUS_PASSWORD=$(cat secrets/prometheus_password)"
  echo "GRAFANA_ADMIN_PASSWORD=$(cat secrets/grafana_admin_password)"
} > .env && chmod 600 .env
printf '%s' 'your-smtp-app-password' > secrets/smtp_password && chmod 600 secrets/smtp_password

# Validate configuration before starting anything
docker run --rm -v "$PWD/prometheus:/etc/prometheus:ro" -v "$PWD/secrets:/run/secrets:ro" \
  --entrypoint promtool prom/prometheus:v3.15.1 check config /etc/prometheus/prometheus.yml
docker run --rm -v "$PWD/prometheus:/etc/prometheus:ro" --entrypoint promtool \
  prom/prometheus:v3.15.1 check rules /etc/prometheus/rules/homelab.yml
docker run --rm -v "$PWD/alertmanager:/etc/alertmanager:ro" --entrypoint amtool \
  prom/alertmanager:v0.34.1 check-config /etc/alertmanager/alertmanager.yml

docker compose up -d
docker compose ps

promtool checks the main file and the rule files it references, and amtool check-config checks the Alertmanager file and its receivers. Fix every error they print before you start the stack; a Prometheus server that fails to parse its configuration at startup will not start, and a bad reload keeps the old configuration running without telling you by email.

Step 11: confirm it works

# Unauthenticated requests must fail with 401
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9090/api/v1/targets

# Authenticated request lists targets and their health
curl -s -u "grafana:$(cat secrets/prometheus_password)" \
  http://127.0.0.1:9090/api/v1/targets | python3 -m json.tool | grep -E '"health"|"scrapeUrl"'

# Days left on each probed certificate
curl -s -u "grafana:$(cat secrets/prometheus_password)" \
  --data-urlencode 'query=(probe_ssl_earliest_cert_expiry - time()) / 86400' \
  http://127.0.0.1:9090/api/v1/query | python3 -m json.tool

# Confirm the node exporter refuses anonymous scrapes from the LAN
curl -s -o /dev/null -w '%{http_code}\n' http://mon.home.example:9100/metrics

Step 12: test the alert path

Fill a scratch filesystem rather than a real one. Create a small loop device, mount it, fill it past 90 percent and watch the rule go pending, then firing. This block creates and writes to a new image file only; double check the paths before you run it.

# Check that /srv/alerttest does not exist yet and that /var/tmp has 200 MB free
ls -ld /srv/alerttest 2>/dev/null; df -h /var/tmp

sudo truncate -s 200M /var/tmp/alerttest.img
sudo mkfs.ext4 -q /var/tmp/alerttest.img       # formats the image file, not a disk
sudo mkdir -p /srv/alerttest
sudo mount -o loop /var/tmp/alerttest.img /srv/alerttest
sudo dd if=/dev/zero of=/srv/alerttest/fill bs=1M count=185 status=none

# After 15 minutes the alert fires; then clean up
sudo umount /srv/alerttest && sudo rm /var/tmp/alerttest.img && sudo rmdir /srv/alerttest

Open http://127.0.0.1:9090/alerts (through an SSH tunnel if you are on another machine) to see the pending and firing states, and check that one email arrives, not one per evaluation.

How do logs fit in?

Metrics tell you that something is wrong. Logs tell you why. A homelab needs both, but logs do not need to go into Prometheus.

syslog, RFC 5424

Network gear, firewalls and many appliances emit syslog. RFC 5424, published in March 2009 on the Standards Track, obsoletes RFC 3164 and defines the modern message format: a priority value calculated by multiplying the facility by 8 and adding the severity, severities from 0 (Emergency, "system is unusable") to 7 (Debug), a version, an RFC 3339 timestamp, hostname, app name, process ID, message ID and optional structured data. RFC 5425 defines the TLS transport. Our syslog protocol explainer covers both in detail. At home, point your switch, router and access points at one syslog receiver on the monitoring host and keep that receiver off the internet like everything else.

journald

On systemd Linux hosts, journald already holds the logs of every unit. The journald.conf man page documents the Storage= modes (volatile, persistent, auto, none) and the size limits. SystemMaxUse= and SystemKeepFree= default to 10 percent and 15 percent of the filesystem, each capped at 4G, and journald respects whichever is smaller. If you want logs to survive a reboot on a host whose distribution defaults to volatile storage, create /var/log/journal or set Storage=persistent.

A log store

Shipping logs from every host to one log store lets you search across machines and correlate a log line with a spike in a Grafana panel. Two broad categories exist: stores that index only labels and scan log text at query time, which are cheap on disk and need little memory, and stores that build a full text index, which answer arbitrary text searches faster but need more memory and disk. A homelab rarely needs the second. Whatever you choose, it is another HTTP endpoint with every log line in it, so it gets the same treatment: private network, authentication, no public exposure.

How much retention does a homelab need?

The Prometheus storage page gives the formula: needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample, with an average of 1 to 2 bytes per sample. Measure your real series count with the query prometheus_tsdb_head_series after a day of scraping. The examples below use assumed series counts; substitute your own.

Worked example 1: 90 days at a 15 second scrape interval.

• Assume 6 hosts at about 1,000 series each from node exporter (6,000), one managed switch through SNMP at 2,000 series, and 20 blackbox probes at about 20 series each (400). Total: 8,400 series.
• Samples per second: 8,400 / 15 = 560.
• Retention in seconds: 90 × 86,400 = 7,776,000.
• At 2 bytes per sample: 560 × 7,776,000 × 2 = 8,709,120,000 bytes, about 8.7 GB.
• At 1 byte per sample: about 4.4 GB.

Worked example 2: one year at a 60 second scrape interval.

• Same 8,400 series. Samples per second: 8,400 / 60 = 140.
• Retention in seconds: 365 × 86,400 = 31,536,000.
• At 2 bytes per sample: 140 × 31,536,000 × 2 = 8,830,080,000 bytes, about 8.8 GB.

The two examples cost almost the same disk. Four times the history at a quarter of the resolution is often a better trade at home, because you look at year over year trends (disk growth, UPS battery runtime decline) far more often than at 15 second detail from last spring. The storage page also notes that reducing the number of series is likely more effective than lengthening the interval, because samples within a series compress well. Add headroom for the write ahead log and compaction, which the docs say can briefly push on disk size above retention.size; the 20GB cap in the Compose file leaves room for that.

Worked example 3: logs. Assume 6 hosts produce 20,000 log lines a day each at an average of 200 bytes. That is 6 × 20,000 × 200 = 24,000,000 bytes, about 24 MB a day raw, or about 2.2 GB for 90 days before compression. Log volume varies far more than metrics, so measure a week of real output before you size anything. A chatty container in debug mode can produce more than the rest of the lab combined.

A reasonable homelab default: 90 days of metrics at 15 seconds, 30 to 90 days of logs, and longer retention only for the handful of trend metrics you actually review.

Pull metrics, push metrics, logs and uptime checks: what each one catches

The Pushgateway documentation lists its pitfalls plainly: it becomes a single point of failure and bottleneck when many instances push through it, you lose the up metric, and it "never forgets series pushed to it" unless you delete them through its API. For a homelab, the node exporter textfile collector is usually the better push style option, because the metric is tied to the machine and disappears with it.

Common mistakes

• Publishing ports on all interfaces. ports: "9090:9090" in Compose listens on every interface. Docker's packet filtering docs warn that traffic to published container ports "gets diverted before it goes through the ufw firewall settings", so a host firewall rule may not protect it. Publish on 127.0.0.1 or a specific private address.
• Port forwarding Grafana or Prometheus on the router "just for a weekend". The security model is clear that these endpoints do not belong on the internet. Use a VPN.
• Leaving Grafana on admin / admin. The default password is set once on first run. Set it through configuration before the first start.
• Enabling --web.enable-admin-api or --web.enable-lifecycle and forgetting. Both are off by default. If you enable them for snapshots or reloads, keep them behind authentication and block their paths at any proxy.
• Basic auth over plain HTTP across the LAN. The security model says this exposes usernames and passwords in cleartext. Add TLS whenever traffic leaves the host.
• Using insecure_skip_verify to make an error go away. Mount your own CA and set ca_file instead.
• Alerting on causes instead of symptoms. CPU at 90 percent for five minutes on a build box is not a problem. A failing probe is.
• No for duration. Rules without it fire on the first evaluation and flap.
• Forgetting the monitoring stack can die. Without a heartbeat to an external check, silence looks like health.
• Running a privileged exporter container you have not read. smartctl and some UPS exporters need raw device or host access. Prefer projects with a clear maintainer, pin versions and review updates.
• Ignoring retention until the disk fills. Set both retention.time and retention.size from day one.
• SNMP v2c with the community public on a device that also accepts writes. Use a read only community at minimum and SNMP v3 where the device supports it.

FAQ

What is the easiest way to monitor a homelab?

Run Prometheus, node exporter, Alertmanager and Grafana with Docker Compose on one host, add the blackbox exporter for HTTPS and certificate checks, and start with five alerts: target down, disk almost full, disk filling fast, probe failing and certificate expiring. Add exporters for SMART, SNMP and your UPS once those work.

Is Prometheus pull or push?

Pull. The overview says time series collection "happens via a pull model over HTTP", with pushing supported through an intermediary gateway. The Pushgateway is recommended only for limited cases such as service level batch jobs.

What port does node exporter use?

9100 by default, according to its README. The Prometheus port allocation wiki lists 9090 for Prometheus, 9091 for Pushgateway, 9093 for Alertmanager, 9115 for blackbox exporter and 9116 for SNMP exporter. The smartctl exporter defaults to 9633.

Is it safe to expose Grafana to the internet?

It is not recommended. Grafana ships with a well known default admin account, acts as a proxy into your network through its data sources, and runs plugin code. Reach it over a VPN. If you publish it anyway, use HTTPS, strong unique passwords, an authentication layer at the proxy, cookie_secure = true and no unsigned plugins.

How do I secure the Prometheus /metrics endpoint?

Use the web configuration file with --web.config.file to add TLS and bcrypt hashed basic auth, bind to a private address, and firewall the port so only your Prometheus server can reach it. The same file format works on node exporter and the other official exporters.

How do I monitor SSL certificate expiry with Prometheus?

Probe each HTTPS endpoint with the blackbox exporter and alert on (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14. The metric is a Unix timestamp for the expiry of the presented certificate chain, so subtracting time() and dividing by 86,400 gives days left.

How long should Prometheus keep data in a homelab?

The default is 15 days, which is short for trend questions such as disk growth or battery runtime decline. 90 days at a 15 second interval or a year at 60 seconds both fit in about 9 GB for a six host lab, using the storage page's 1 to 2 bytes per sample.

How do I stop getting too many alerts?

Add a for duration to every rule, keep warnings out of night hours with time_intervals, group by alertname and instance, inhibit downstream alerts when an upstream failure (UPS, switch) is firing, raise repeat_interval for warnings and delete alerts you never act on.

Do I need a log server for a homelab?

Not on day one. journald keeps local logs with sensible default limits. Add a central syslog receiver for network gear, and a log store once you find yourself SSHing into several machines to piece together one incident.

Can I monitor a UPS with Prometheus?

Yes, through Network UPS Tools and a community NUT exporter listed on the Prometheus exporters page. Alert on the UPS being on battery and on low runtime, and inhibit host down alerts while the UPS alert is firing.

Where Safeguard fits

Safeguard does not monitor hosts, collect metrics or replace any part of this stack. The fit is indirect: every component here is open source software you pull as container images or binaries, and many homelabs add community exporters, Grafana plugins and small scripts on top. Safeguard can scan those images and project directories (safeguard scan --image prom/prometheus:v3.15.1, safeguard scan --dir .) to produce an SBOM, list the packages and versions inside and match them against vulnerability databases, with reachability analysis to rank what your code actually calls. Results land as JSON under ./output, and the platform shows which of your projects contain a given component and version. If you write your own exporter or alert relay, autonomous remediation can open a tested fix pull request when a dependency needs an upgrade, and it ships switched off. It will not tell you a disk is failing, and it does not secure your network edge.

A monitoring stack is a bundle of open source components that you rebuild every time you bump an image tag, and community exporters and dashboard plugins often lag behind on their own dependencies. Knowing what is inside those images, and having a path to a fixed version when a maintainer is slow, is the supply chain side of keeping a homelab healthy.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

These demos show dependency, SBOM and remediation workflows, not metrics collection or alerting. You can download Safeguard and follow the CLI docs to try them on your own projects.

This demo sets up the Safeguard runner on your own machine, approves it with its device code and scans from the terminal. Run on the host that holds your Compose file, the same scan can list the packages inside the Prometheus, Alertmanager, Grafana and exporter images you pinned, so you know which ones to rebuild when a component needs an update.

Click through it at your own pace, or open it full screen.

This demo is a real CLI session: ask it to fix the lodash CVE in a repository and watch it read the finding and the advisory, ask before each command, move lodash to 4.17.21, run npm install and npm test with your approval, and push nothing. The same approve each step flow applies when you bump a dependency in a small exporter, webhook relay or dashboard tool you wrote for your lab.

Click through it at your own pace, or open it full screen.

This demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. You can then ask which of your projects ship a given package version and get the answer from your own scan data rather than from memory.

Click through it at your own pace, or open it full screen.

Quick check

Sources

• Prometheus: Overview
• Prometheus: Security model
• Prometheus: HTTPS and authentication (web configuration file)
• Prometheus: Storage
• Prometheus: Query functions
• Prometheus: Configuration reference
• Prometheus: Alerting rules
• Prometheus: Alerting practices
• Prometheus: When to use the Pushgateway
• Prometheus: Exporters and integrations
• Prometheus: OpenMetrics 1.0 specification
• Prometheus: Alertmanager
• Prometheus: Alertmanager configuration
• Prometheus wiki: Default port allocations
• Exporter toolkit: Web configuration
• node_exporter README
• blackbox_exporter README and configuration reference
• blackbox_exporter prober source (metric definitions)
• snmp_exporter README
• smartctl_exporter README
• Grafana: Configure security
• Grafana: Configure Grafana (configuration reference)
• Grafana: Anonymous authentication
• Grafana: Plugin management
• Grafana: Provisioning
• Grafana: Run Grafana Docker image
• Docker docs: Packet filtering and firewalls
• RFC 5424: The Syslog Protocol
• systemd: journald.conf

Last reviewed: 24 October 2026. Image tags reflect the latest releases on each project's GitHub releases page on that date; check for newer versions before you deploy.