Skip to content

Loki Log Aggregation in Practice

I have run log pipelines for a small platform team for years, and for most of that time I ran ELK. Elasticsearch was comfortable: I could grep anything, anywhere, instantly. Then the bill arrived. My ingest went from ~20 GB/day to ~120 GB/day as the microservice count doubled, and the Elasticsearch cluster needed three more data nodes just to keep index latency under a second. The storage cost and the ops load were both growing faster than my headcount. I migrated to Loki.

1. Why ELK and I parted ways

The core difference is the index. Elasticsearch full-text indexes every log line. That is why search is instant — but it is also why storage explodes. A 1 KB JSON log line can consume 3–5 KB on disk once you count the inverted index, the doc store, and replica overhead. Loki does the opposite: it does not index content at all. It indexes labels and stores the log bytes as compressed chunks. Search across the body is a scan, not an index lookup.

The cost math is what convinced me. In my ELK cluster, a hot data node with 1 TB of NVMe held roughly 8 days of logs at 120 GB/day before the 15% shard watermark forced rebalancing. On Loki with the same volume, the same retention fit on a single 1 TB disk with room to spare. Ingest also got simpler: no shard tuning, no mapping conflicts, no force-merge windows. For a team that mostly filters by service and searches by error string, Loki was the better trade.

2. Picking a deploy mode

Loki ships three ways. The single binary runs everything in one process and is fine for a dev box, but it has no HA. The microservices mode splits compactor, distributor, ingester, and querier into separate deployments — that is what Grafana Cloud runs and what a large platform needs, but it is overkill for a team of four. The docker-compose cluster runs the Loki components as separate containers sharing object storage. That is the mode I chose: simple enough to debug, resilient enough for production, and I can lift it into Kubernetes later without rewriting configs.

3. The single-cluster docker-compose walkthrough

My stack is three containers: loki, alloy, and grafana, wired into a monitoring network. The Loki config keeps a single filesystem backend and a ring backed by Consul or, simpler for one node, a boltdb file.

# loki-config.yaml
auth_enabled: false
server:
  http_listen_port: 3100
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 720h
  ingestion_rate_mb: 8

The 30-day retention lives in limits_config.retention_period, and I set ingestion_rate_mb per-stream to keep a noisy app from flooding the ingester. The compose file maps those into a named volume.

services:
  loki:
    image: grafana/loki:3.1.0
    command: -config.file=/etc/loki/config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml:ro
      - loki-data:/loki
    ports: ["3100:3100"]
  alloy:
    image: grafana/alloy:v1.4.0
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
      - /var/log:/var/log:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
  grafana:
    image: grafana/grafana:11.2.0
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
    ports: ["3000:3000"]
volumes:
  loki-data:

I verify the cluster in three steps: check readiness, push a probe line, then confirm it lands.

curl -s localhost:3100/ready
echo '{"streams":[{"stream":{"job":"probe"},"values":[["'"$(date +%s%N)"'","hello from loki"]]}]}' \
  | curl -s -X POST localhost:3100/loki/api/v1/push -H 'Content-Type: application/json' --data-binary @-
# expect: {"status":"success"}

In Grafana I add a Loki data source at http://loki:3100, then run {job="probe"} in Explore. I learned the hard way that the push endpoint wants nanosecond timestamps; sending milliseconds silently drops the entry.

4. The collector decision: Alloy, Promtail, or Vector

I ran all three before committing. Promtail is the legacy Loki collector. It works, but Grafana moved development to Alloy and Promtail is in maintenance mode. Vector is a general-purpose agent from Datadog. It is the strongest choice when I fan out the same logs to Loki and a metrics backend and an S3 archive, because it handles many sinks natively. Alloy is the current Grafana default, and for a Loki-centric pipeline it is what I settled on. It replaces both Promtail and the old Grafana Agent, uses a declarative River syntax, and gives me service discovery and relabeling in one config.

For my single-cluster setup the choice was clear: Alloy, because my pipeline ends at Loki and Alloy has the tightest integration with it.

# alloy-config.alloy
loki.source.docker "default" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.docker.containers
  forward_to = [loki.process.default.receiver]
}
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}
loki.process "default" {
  stage.regex { expression = "^(?P<level>\\w+)\\s.*" }
  stage.labels { values = { level = "" } }
  forward_to = [loki.write.default.receiver]
}
loki.write "default" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

The stage.regex and stage.labels blocks parse the log level out of the first field and attach it as a label, which is exactly the kind of label I can filter on later.

5. Label design: the golden rules

Loki is a label-first database. Every unique set of label values creates a stream, and every stream is an index entry. That is powerful but it is also the trap. My rules, hard-won:

  • Budget cardinality. I keep the total number of unique label combinations under ~100,000 across all streams. Past that, query planning and ingestion both slow down.
  • Labels are low-cardinality only. job, level, app, cluster, env are safe. pod_name, container_id, trace_id, and request_id are never labels. Those belong in the log line body, not the label set.
  • Labels are filters, not metadata. If I never query on it, I do not label it. Metadata belongs in the structured log body where | json can extract it lazily.

The anti-patterns I actually hit are worth naming. I once relabeled the full container_id from Docker. Every container restart created a new stream, and within a week I had 400,000 streams and queries that timed out. I also labeled path from nginx access logs — that was hundreds of streams per app, all from a field I rarely filtered on. I fixed both by moving the high-cardinality value into the log body and keeping only job + level + app as labels. My stream count dropped from ~400k to ~2k and queries went from timing out to sub-second.

6. LogQL cheat sheet

LogQL is the query language, and the two functions that trip everyone up are rate and count_over_time.

# events per second — use for dashboards and alerts
rate({app="api"}[5m])

# raw line count in the window — a step counter, NOT a rate
count_over_time({app="api"}[5m])

rate() normalizes by the range, so it is comparable across windows and it is what I plot. count_over_time() returns the raw count, so a 5m window shows five times the value of a 1m window. If I alert on it I must set the threshold with the range in mind.

Structured logs are where I parse JSON with | json. The trap: a malformed JSON line in the stream aborts the query for that stream. I always test against a mixed sample.

{app="api"} | json | level = "error" | line_format "{{.ts}} {{.msg}}"

# unwrap extracts a numeric value for metric-style aggregation
sum by (route) (rate({app="api"} | json | unwrap bytes [5m]))

unwrap is what turns a numeric field inside the log body into a series I can rate or sum. It only works on numbers; passing a string returns an error, which is another parsing trap I hit in production.

7. Five real incidents

Incident 1: log volume spike filled the disk. A cron job flipped into a tight error loop and wrote 400 GB of stack traces in six hours. The chunk disk hit 100%. The fix was a global ingestion_rate_mb in the limits config, plus a per-tenant cap, and a Grafana alert on rate({app=~".+"}[5m]) crossing a threshold so I know before the disk does.

Incident 2: label cardinality explosion. The container_id relabeling I described above. Queries started timing out at the querier. I fixed it by dropping the high-cardinality label, and I added a Grafana dashboard to watch stream growth.

Incident 3: retention not applying, disk filled again. I set retention_period and waited, and nothing happened. Loki's retention is applied by the compactor, not the ingester, and my single-binary config had compaction scheduled far apart. I set compactor.compaction_interval and retention_period together, ran a manual compaction, and the disk came back down.

Incident 4: cross-cluster federation sync lag. I had a second cluster forwarding to the primary. The federated querier returned stale results for minutes because the store refresh interval was too long. I lowered the store refresh_interval so queries balanced across stores.

Incident 5: multi-tenant misconfig leaked logs between tenants. I enabled auth_enabled but left the push endpoint open with a default tenant. A tenant A service pushed into the default tenant, and tenant B could read it. I enforced a tenant header on every push and locked down the distributor to reject pushes without a valid X-Scope-OrgID.

8. The production checklist

Before I take a Loki stack live, I run eight checks:

  1. Retention is set in limits_config and the compactor interval is aligned with it.
  2. Cardinality is below the 100k budget; I verify stream count with sum(count_over_time({job=~".+"}[1m])).
  3. High-cardinality labels (pod, container, trace) are excluded from the label set.
  4. Ingestion rate limits are configured globally and per-stream.
  5. Auth and tenants are enforced; no anonymous push, valid X-Scope-OrgID required.
  6. Alerts exist for log rate, stream count, and disk usage.
  7. Storage has headroom for at least two full retention cycles.
  8. Backup of the config and the object store is tested, not assumed.

9. Loki HTTP API quick reference

curl localhost:3100/ready                       # health check
curl localhost:3100/metrics                     # internal metrics
curl -X POST localhost:3100/loki/api/v1/push    # push logs
curl 'localhost:3100/loki/api/v1/query_range?query={app="api"}&start=...&end=...&limit=100'  # range query
curl -G 'localhost:3100/loki/api/v1/query_range' --data-urlencode 'query=rate({app="api"}[5m])'  # metric query

The push endpoint is a POST with a JSON body; the query endpoints take a URL-encoded LogQL expression. I keep the push body small and batched, because many small pushes are slower than a few large ones.

10. FAQQ: How do I know when to alert on a log rate spike?

A: Plot rate({app="api"}[5m]) and alert when it crosses a multiple of the baseline — I use 3x for 10 minutes. Alert on the rate, not on count_over_time, so the threshold is window-independent.

Q: Should I keep structured logs as JSON or plain text? A: JSON. | json lets me filter and unwrap numeric fields without regex. Plain text is fine for a single app, but the moment I need a field for a query it has to be structured.

Q: My query is slow — what do I check first? A: Cardinality. Run sum(count_over_time({job=~".+"}[5m])) and look at bytes_processed in the query stats. If stream count is high, drop the high-cardinality label. Then check that I used a label matcher, not just a body filter.

Q: Is Loki a drop-in replacement for Elasticsearch? A: No. Loki is not a full-text search engine. It is a label-first log store. If I need substring grep across arbitrary bodies at scale, ELK still wins. Loki wins when the primary access path is label filtering plus occasional scanning.