Build pipeline logs are the most ignored telemetry in most engineering organizations. They are voluminous, they are repetitive, and the job that produced them is usually already forgotten by the time anyone thinks to ask a question of them. Yet when something breaks — a flaky release, a leaked secret, a supply chain attack that quietly modified a build step — those logs are often the only record of what happened.
Grafana Loki is well suited to this use case. It stores logs cheaply, scales horizontally, and integrates with the dashboards engineers already use for metrics and traces. What it does not do is decide your label strategy or enforce retention discipline, and getting those two decisions wrong is how teams end up with a Loki deployment that costs more than it should and answers fewer questions than it could.
Why Loki fits the CI/CD shape
The reason Loki's design fits build logs particularly well is its labeling model. Unlike a full-text search engine, Loki indexes only labels and streams, not the content of every log line. That means you can store enormous volumes of build output — full compiler logs, full test output, full dependency resolution traces — at a fraction of the cost of indexed storage, provided you keep label cardinality under control.
The second reason is Loki's integration with the rest of the Grafana stack. Metrics come from Prometheus, traces come from Tempo, and logs come from Loki. For pipeline observability, the ability to pivot from a metric spike to the underlying logs with a single click is how you move from "our deploy rate dropped" to "here is the specific pipeline failure" in seconds rather than minutes.
Label strategy
Label strategy is the single most important decision in a Loki deployment, and it is disproportionately important for CI/CD logs because the temptation to label everything is strong.
A workable labeling scheme for build pipeline logs uses six labels: pipeline (the name or ID of the pipeline definition), repo (the repository the pipeline runs against), stage (build, test, deploy, etc.), environment (dev, staging, prod), runner_type (self-hosted, hosted, ephemeral), and status (success, failure, cancelled). That is enough to slice the data in most operational queries and not so much that label cardinality blows up.
Things that should not be labels: commit SHA, pull request number, individual runner hostname for ephemeral runners, timestamp components, or anything else that is unique per-run. Those values belong in the log line content where LogQL can filter on them, not in the label index where they would multiply cardinality.
Enforce the labeling scheme through your shipper configuration, not through individual pipeline definitions. Promtail or Grafana Agent configured centrally is the single source of truth, and pipelines should have no ability to inject arbitrary labels.
Retention and tiering
Build logs have a sharply declining usefulness curve. Logs from the last 48 hours are critical; logs from the last 30 days are important for trend analysis and postmortems; logs from the last 12 months are occasionally relevant for audit and compliance; logs older than 12 months are almost never consulted.
Configure Loki retention to match that curve. Hot storage with fast indexing for 48 hours, warm storage with standard indexing for 30 days, and cold storage with compression for longer horizons. Use Loki's per-tenant retention configuration if your deployment serves multiple teams with different needs.
For supply chain forensics specifically, keep a longer retention tier for logs that include package installation events. Malicious dependencies can live undetected in a codebase for months, and when the indicator of compromise finally surfaces, the ability to answer "did we install this package at any point in the last year" is worth the marginal storage cost.
LogQL patterns for pipeline observability
LogQL is concise once you internalize the pattern of filtering by labels, then narrowing within streams, then optionally parsing and aggregating. Four query patterns cover the majority of day-to-day pipeline work.
The "find the failing pipeline" pattern filters by status="failure" and pipeline=<name> over the last hour and sorts by timestamp. Add a parser for the specific error signatures you care about, and the query becomes a living runbook for the most common failure modes.
The "compare two pipeline runs" pattern filters by run ID and diffs the log output. Teams that use this as part of their incident response regularly find root causes that individual run inspection would miss, because the shape of the diff tells the story.
The "count pipeline failures by reason" pattern uses LogQL's aggregation features to group by parsed error signature and count occurrences over time. The resulting dashboard panel surfaces systemic failure modes that no individual failure alert would expose.
The "find secrets in logs" pattern searches the entire stream for patterns that look like tokens, keys, or credentials, and filters down to logs produced by runs that completed successfully. Secrets in a failed run are bad; secrets in a successful run that was then published to a registry are worse.
Supply chain-specific patterns
Pipeline logs are one of the best forensic sources when a supply chain incident is suspected. Three patterns pay off repeatedly.
A query that finds every pipeline run in the last N days that installed a specific package version. Given a newly discovered compromise, this answers "did we ingest the malicious version" faster than any other source.
A query that finds every pipeline run whose logs contain an outbound connection to a specific host. When threat intelligence surfaces a new C2 destination, this shows whether any of your build infrastructure talked to it.
A query that finds every pipeline run that modified a file outside the expected output directories. This pattern catches pipelines that were modified to write backdoor material into build artifacts — the same pattern that showed up in several real incidents.
Cost discipline
Loki costs scale linearly with stream cardinality and logs volume, and supply chain and CI/CD logs are a major source of both. Three habits keep costs under control without sacrificing coverage.
First, drop logs that are structurally noisy before they hit Loki. Heartbeat logs, successful health checks, and routine cache hits can be sampled or dropped at the shipper. The shipper is the right place to make these decisions because pushing them further upstream makes the cost of a tuning error higher.
Second, revisit label cardinality quarterly. Loki exposes metrics about label cardinality per tenant. A label that has grown to tens of thousands of unique values was almost certainly not intended to be a label, and refactoring it back into log line content pays for itself within a billing cycle.
Third, measure query patterns. The queries that run every hour, every day, and every incident tell you what the label schema needs to support. Queries that people want to run but cannot — because label cardinality would make them too expensive — tell you where the schema needs to evolve.
Operating the pipeline
Loki is as much a social system as a technical one. Assign an owner, review label cardinality monthly, and treat query patterns as a first-class artifact. Teams that do this end up with a logging system their engineers actually use; teams that don't end up with a cost center nobody can justify.
How Safeguard Helps
Safeguard integrates with Grafana Loki to enrich build pipeline logs with software supply chain context. When a pipeline installs a package, publishes an artifact, or modifies a signing key, Safeguard correlates the event to SBOM changes, CVE impact, and provenance signatures, and exposes the result as a structured field Loki can index and query. Engineering teams running LogQL queries against their build logs can ask supply chain questions — "did we ship a compromised dependency," "which pipelines touched signing material last week" — without context-switching to a separate platform. The combination delivers a single forensic surface for both operations and supply chain security.