Skip to content

Thanos in Production: Long-Term Prometheus Storage and Multi-Cluster Querying

1. The day I stopped trusting single-node Prometheus

I run observability for a fleet of Kubernetes clusters spread across two on-prem data centers and one cloud region. For two years every cluster carried its own Prometheus with a 15-day retention. That held up fine — until an SRE asked me "was our API latency worse six weeks ago?" and the answer did not exist anymore, because Prometheus had already deleted it. Retention was a wall. I had two choices: raise retention and watch memory explode, or push TSDB blocks somewhere cheap and query everything from one place. I chose Thanos.

2. The architecture I landed on

Thanos is not a Prometheus fork. It is a set of components that bolt onto an existing Prometheus deployment. These are the five I run in production:

  • Sidecar — a small container next to every Prometheus that uploads completed TSDB blocks to object storage and exposes a StoreAPI endpoint for live queries.
  • Query — a stateless, horizontally scalable PromQL engine that fans queries out to sidecars, store gateways, and receivers.
  • Store Gateway — serves long-term blocks back from object storage to Query.
  • Compactor — applies retention, compaction, and downsampling to uploaded blocks.
  • Receiver — an optional write-ahead path that accepts remote_write, buffers data in memory, and flushes blocks to storage, useful when you want a global write path instead of scraping every cluster separately.

In my setup each cluster's Prometheus statefulset has a sidecar, the primary DC runs a query pair behind a load balancer, two store gateways, one compactor, and a receiver pair. Query is the only component a user ever talks to.

3. Object storage: MinIO on-prem, S3 in the cloud

The storage backend is the biggest architectural decision. I use one logical bucket per environment and abstract it with Thanos's storage client, configured by YAML passed via --objstore.config-file.

type: S3
config:
  bucket: thanos-metrics
  endpoint: minio.ops.svc.cluster.local:9000
  access_key: ${MINIO_ACCESS_KEY}
  secret_key: ${MINIO_SECRET_KEY}
  insecure: true
  signature_version2: false

The same file works against AWS S3 by swapping the endpoint and adding a region. Thanos also speaks GCS and Azure, but S3-compatible keeps my on-prem DCs and the cloud region on one code path, so I can mirror data between them.

4. Wiring the sidecar into Prometheus

The sidecar is the quiet workhorse. It watches the TSDB, uploads every completed two-hour block, and keeps a replica of the newest block for instant queries. Here is my sidecar container:

- name: thanos-sidecar
  image: quay.io/thanos/thanos:v0.34.1
  args:
    - sidecar
    - --objstore.config-file=/etc/thanos/objstore.yml
    - --prometheus.url=http://localhost:9090
    - --tsdb.path=/prometheus
    - --shipper.upload-compacted
  ports:
    - containerPort: 10901

The one non-obvious trap: the sidecar must use the exact same --tsdb.path as Prometheus. Point it anywhere else and it uploads nothing and silently waits forever.

5. Compaction and downsampling — why they matter

The compactor is the piece people underestimate. It merges overlapping blocks, applies retention, and — critically — builds downsampled series. Raw blocks hold 15-second resolution. The compactor rolls those into 5-minute and 1-hour resolution series. Without downsampling, rate(http_requests_total[30d]) would scan millions of samples. With it, the same query reads a few thousand.

- name: thanos-compactor
  image: quay.io/thanos/thanos:v0.34.1
  args:
    - compact
    - --objstore.config-file=/etc/thanos/objstore.yml
    - --wait
    - --retention.resolution-raw=30d
    - --retention.resolution-5m=180d
    - --retention.resolution-1h=0

The same expression automatically lands on the right resolution: a 30-day window is served by the 5m downsampled blocks, while the last hour stays at raw 15-second resolution.

# long range -> 5m downsampled series, fast and cheap
sum(rate(http_requests_total{cluster="dc1"}[30d])) by (cluster)

6. Multi-cluster federation with Query

With blocks in storage and sidecars in every cluster, I exposed a single global query endpoint. The --store flag points at every component that speaks StoreAPI:

- name: thanos-query
  image: quay.io/thanos/thanos:v0.34.1
  args:
    - query
    - --store=dnssrv+_grpc._tcp.thanos-store-gateway
    - --store=dnssrv+_grpc._tcp.thanos-sidecar
    - --store=dnssrv+_grpc._tcp.thanos-receiver
    - --query.replica-label=prometheus_replica
    - --query.auto-downsampling
    - --query.timeout=2m

The replica-label flag deduplicates identical series across HA Prometheus pairs. auto-downsampling lets Query transparently fall back to 5m or 1h data for long ranges. Now sum(rate(http_requests_total[5m])) against the global endpoint returns one number spanning every cluster, and the same expression with [30d] returns a downsampled line instead of a timeout.

For clusters that cannot be scraped directly, I use the receiver as a global write path. Prometheus ships metrics via remote_write, the receiver buffers them, and later flushes blocks to the same bucket, so the store gateway serves them like any other block.

# prometheus.yml in a remote cluster
remote_write:
  - url: http://thanos-receiver:19291/api/v1/receive
    queue_config:
      capacity: 5000
      max_shards: 100
    write_relabel_configs:
      - source_labels: [cluster]
        regex: dc3
        action: keep

7. Incident 1 — backfill that never landed

I backfilled 90 days of data from a decommissioned cluster. Uploads finished, blocks appeared in MinIO, but queries returned nothing. The root cause: the backfilled series carried a different external_labels set — they lacked cluster. Query deduplicated them away, and the compactor treated them as orphaned. I re-ran the backfill with a consistent label set on the sidecar, then verified with:

thanos tools bucket inspect --objstore.config-file=/etc/thanos/objstore.yml

8. Incident 2 — query timeout after a retention change

I raised compactor retention and overnight every [30d] query timed out. Raw data was still present for the full window, so Query scanned millions of samples and blew past --query.timeout=2m; the compactor had not yet produced 5m downsampled blocks for the new window. The lesson: keep --retention.resolution-raw below your longest common query range and let downsampling carry the long-range queries.

9. Incident 3 — compactor stuck in a partial compaction loop

The compactor logged compaction is in progress and never finished. Two blocks had been uploaded out of order by a sidecar restarted mid-upload; the partial block was never closed, so the compactor tried to compact it forever. I deleted it manually and added --shipper.upload-compacted only after confirming block ordering was stable.

thanos tools bucket delete \
  --objstore.config-file=/etc/thanos/objstore.yml \
  01HX8Z9Q3V2Y7K4M6N0PQRSTUV

10. Incident 4 — store gateway OOM on label queries

A dashboard with a very wide label_values() call pushed the store gateway past 8 GB and got it OOM-killed. The gateway was loading entire block indexes into memory. I split the gateway into two replicas, set --store.grpc.series-sample-limit, and capped the container memory so the OOM killer targeted the gateway and not the node. The real fix was reducing the cardinality of the label I was querying.

- name: thanos-store
  image: quay.io/thanos/thanos:v0.34.1
  args:
    - store
    - --objstore.config-file=/etc/thanos/objstore.yml
    - --store.grpc.series-sample-limit=1000000
  resources:
    limits:
      memory: 4Gi

11. Incident 5 — retention misconfig deleted historical data

The scariest one. I set --retention.resolution-1h=0 expecting "keep forever," but 0 means delete immediately. The compactor deleted every 1-hour downsampled block. I restored from a MinIO bucket-level snapshot. The rule I follow now: never set a retention flag to 0 unless I explicitly intend to drop that data.

12. One complete production docker-compose template

This is the full template for my central site — Prometheus with sidecar, query, store gateway, compactor, receiver, and MinIO:

version: "3.9"
services:
  minio:
    image: minio/minio:RELEASE.2024-01-16T16-07-38Z
    command: server /data --console-address ":9001"
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"

  prometheus:
    image: prom/prometheus:v2.51.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
      - --web.enable-lifecycle
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus

  thanos-sidecar:
    image: quay.io/thanos/thanos:v0.34.1
    command:
      - sidecar
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --prometheus.url=http://prometheus:9090
      - --tsdb.path=/prometheus
      - --shipper.upload-compacted
    depends_on:
      - prometheus
    volumes:
      - ./objstore.yml:/etc/thanos/objstore.yml:ro
      - prom-data:/prometheus

  thanos-query:
    image: quay.io/thanos/thanos:v0.34.1
    command:
      - query
      - --store=dnssrv+_grpc._tcp.thanos-store
      - --store=dnssrv+_grpc._tcp.thanos-sidecar
      - --store=dnssrv+_grpc._tcp.thanos-receiver
      - --query.replica-label=prometheus_replica
      - --query.auto-downsampling
    ports:
      - "10902:10902"
    depends_on:
      - thanos-store
      - thanos-sidecar

  thanos-store:
    image: quay.io/thanos/thanos:v0.34.1
    command:
      - store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --store.grpc.series-sample-limit=1000000
    volumes:
      - ./objstore.yml:/etc/thanos/objstore.yml:ro

  thanos-compactor:
    image: quay.io/thanos/thanos:v0.34.1
    command:
      - compact
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --wait
      - --retention.resolution-raw=30d
      - --retention.resolution-5m=180d
      - --retention.resolution-1h=0
    volumes:
      - ./objstore.yml:/etc/thanos/objstore.yml:ro

  thanos-receiver:
    image: quay.io/thanos/thanos:v0.34.1
    command:
      - receive
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --tsdb.path=/receiver
      - --receive.default-tenancy-id=default
    volumes:
      - ./objstore.yml:/etc/thanos/objstore.yml:ro
      - receiver-data:/receiver

volumes:
  minio-data:
  prom-data:
  receiver-data:

networks:
  default:
    name: thanos-net

13. Commands I run every week

# verify every block against object storage integrity
thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore.yml

# list blocks and their time ranges, sorted
thanos tools bucket inspect --objstore.config-file=/etc/thanos/objstore.yml

# confirm retention boundaries are respected
thanos tools bucket ls \
  --objstore.config-file=/etc/thanos/objstore.yml \
  --min-time=2026-06-01T00:00:00Z \
  --max-time=2026-10-31T00:00:00Z

# query a single cluster through the federation endpoint
curl -G 'http://thanos-query:10902/api/v1/query' \
  --data-urlencode 'query=sum(rate(http_requests_total{cluster="dc1"}[5m]))'

# confirm downsampled blocks exist for the historical window
thanos tools bucket inspect \
  --objstore.config-file=/etc/thanos/objstore.yml \
  --selector='{__name__=~".+"}' \
  | grep -i 'resolution=5m'

14. Frequently asked questions

Q: How does Thanos store Prometheus data long-term? A: The sidecar uploads each completed TSDB block to object storage, and the store gateway serves those blocks back to Query on demand, so retention is no longer bound by the Prometheus disk.

Q: What is downsampling in Thanos? A: The compactor builds 5-minute and 1-hour resolution series from the raw 15-second data, letting long-range queries read far fewer samples and finish quickly.

Q: Can Thanos query across multiple clusters? A: Yes. The query component fans PromQL out to every sidecar, store gateway, and receiver in one global endpoint, deduplicating HA replicas via --query.replica-label.

Q: Which object storage does Thanos support? A: Any S3-compatible bucket — MinIO, AWS S3, Google Cloud Storage, or Azure Blob Storage — plus filesystem for local development.

Q: Is Thanos a fork of Prometheus? A: No. It is a set of sidecar components that work alongside an existing Prometheus, so you can adopt it incrementally without replacing your scrapers or alerting rules.