Best Monitoring Stacks for Self-Hosted Infrastructure

If you’re running self-hosted infrastructure, you’ve probably experienced that sinking feeling: your application goes down, and you don’t find out for hours because you’re not actively watching dashboards. Or worse, you’re monitoring with a mishmash of tools that don’t talk to each other, requiring you to jump between five different interfaces to understand what actually failed.

A solid monitoring stack for self-hosted infrastructure isn’t just nice-to-have anymore—it’s essential. But unlike SaaS monitoring where you write a check and someone else handles the complexity, building an effective monitoring stack for self-hosted systems means choosing the right components, understanding how they integrate, and actually maintaining them. You’re responsible for the full stack: collection, storage, alerting, visualization, and log aggregation.

This article covers the best monitoring stacks for self-hosted infrastructure, including what works, what doesn’t, and the real tradeoffs you’ll face when deciding between popular open-source solutions and proprietary options.

Why Self-Hosted Monitoring Matters

Before diving into specific tools, let’s be clear about why you might choose self-hosted monitoring:

• Data sovereignty: Your metrics, logs, and traces stay on your infrastructure
• Cost control: You’re not paying per metric, per host, or per GB ingested
• Customization: You can modify components to fit your exact workflow
• No egress costs: Sensitive data doesn’t leave your network
• Independence: You’re not locked into a vendor’s pricing or feature roadmap

But there’s a catch: you’re also responsible for uptime, scaling, backups, and upgrades. This matters when choosing a stack.

The Core Monitoring Stack Architecture

Every mature monitoring setup needs these components:

1. Metrics collection (Prometheus, InfluxDB, etc.)
2. Time-series database (TSDB) for storage
3. Visualization (Grafana, Kibana, etc.)
4. Alerting engine (Alertmanager, etc.)
5. Log aggregation (ELK, Loki, Splunk)
6. Distributed tracing (optional but increasingly important)

Most teams build around 1-2 central components and add supporting tools. Let’s look at the real-world stacks.

The Prometheus + Grafana + Alertmanager Stack

This is the most common self-hosted monitoring stack, and for good reason.

Architecture overview:
– Prometheus scrapes metrics from endpoints (pull-based)
– Time-series data stored in Prometheus’s local storage or remote systems
– Alertmanager handles routing and notifications
– Grafana visualizes everything

Real example: You’re running 15 microservices across 3 VMs. Each service exports Prometheus metrics on /metrics. Prometheus scrapes every 15 seconds, stores data, and evaluates alert rules. When CPU hits 85%, Alertmanager pages your on-call engineer.

Setup example:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api-server'
    static_configs:
      - targets: ['api-server.internal:9090']

  - job_name: 'database'
    static_configs:
      - targets: ['db-primary.internal:9104']

  - job_name: 'node-exporter'
    static_configs:
      - targets: 
        - 'web1.internal:9100'
        - 'web2.internal:9100'
        - 'db.internal:9100'

# Alert rules
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - '/etc/prometheus/rules.yml'

Pros:
– Lightweight and fast (Prometheus binary is ~50MB)
– Excellent developer experience
– Huge ecosystem of exporters (node_exporter, postgres_exporter, mysql_exporter, etc.)
– Pull-based model prevents accidental metric flooding
– Local time-series storage requires minimal resources

Cons:
– Local storage isn’t highly available (single Prometheus instance = single point of failure)
– Horizontal scaling requires federation or complex sharding
– No built-in clustering for redundancy
– 15GB of metrics storage with typical settings = roughly 1-2 weeks of data at high cardinality
– Querying across multiple Prometheus instances is painful

Real deployment consideration: If you have 50+ hosts and 500K+ active time series, you’ll likely hit Prometheus scalability limits and need solutions like Thanos or Mimir (more on those below).

The ELK Stack (Elasticsearch, Logstash, Kibana)

Originally for logs, ELK has evolved into a full monitoring solution, especially with Beats collectors.

Architecture:
– Beats or Logstash forward data to Elasticsearch
– Elasticsearch indexes and stores (works for metrics, logs, traces)
– Kibana provides visualization
– X-Pack (commercial) adds alerting and machine learning

Real deployment:

# Install Elasticsearch 8.x
docker run -d \
  --name elasticsearch \
  -e discovery.type=single-node \
  -e xpack.security.enabled=false \
  -p 9200:9200 \
  docker.elastic.co/elasticsearch/elasticsearch:8.9.0

# Install Kibana
docker run -d \
  --name kibana \
  -e ELASTICSEARCH_HOSTS=http://elasticsearch:9200 \
  -p 5601:5601 \
  docker.elastic.co/kibana/kibana:8.9.0

# Ship logs with Filebeat
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.9.0-linux-x86_64.tar.gz
tar xzf filebeat-8.9.0-linux-x86_64.tar.gz

Pros:
– Excellent for log aggregation and searching
– Full-text search capabilities (Prometheus can’t do this)
– Works well with Elastic’s monitoring agents
– Can ingest metrics, logs, and APM data in single platform
– Mature ecosystem and community

Cons:
– Heavy resource consumer (Elasticsearch with 3-node cluster = significant memory/disk)
– Steep learning curve for production tuning (sharding, ILM policies, heap settings)
– Licensing changed in recent years (moving toward Elastic Cloud, basic features free)
– Costs scale with data volume (storage is expensive at scale)
– Setup complexity higher than Prometheus

When to use ELK: When logs are your primary concern or you need full-text search across time-series data. If you’re primarily monitoring infrastructure metrics, Prometheus is simpler.

Grafana Loki for Log Aggregation

If Prometheus is for metrics, Loki is Grafana’s purpose-built log aggregation tool, and it’s gaining traction fast.

Why Loki is different:
– Label-based (like Prometheus) not full-text indexed
– Compressed log storage (~1-2GB per TB of original logs)
– Integrates natively with Grafana
– Promtail agent is lightweight

Quick setup:

# loki-config.yml
auth_enabled: false

ingester:
  chunk_idle_period: 3m
  max_chunk_age: 1h
  max_streams_per_user: 10000
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

server:
  http_listen_port: 3100
  log_level: info

# promtail-config.yml (agent)
clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

  - job_name: app
    static_configs:
      - targets:
          - localhost
        labels:
          job: app
          __path__: /var/log/app/*.log

Pros:
– Dramatically lower storage requirements than ELK
– Grafana integration is seamless
– Can correlate logs with metrics in same dashboard
– Simple to set up and scale horizontally
– Great for containerized environments

Cons:
– Not full-text indexed (searches are slower than Elasticsearch)
– Label cardinality explosion can cause problems
– Querying syntax different from typical log tools
– Less mature ecosystem than ELK

Real consideration: Many teams run Prometheus + Grafana + Loki. It’s a powerful trifecta that’s relatively lightweight and well-integrated.

Thanos: Scaling Prometheus Beyond Limits

When you outgrow a single Prometheus instance, Thanos provides:
– Long-term storage (object storage backends like S3, MinIO)
– Global querying across multiple Prometheus instances
– High availability
– Downsampling for efficient long-term retention

Architecture with Thanos:

Multiple Prometheus instances
         ↓ (scrape metrics)
    Thanos Sidecar (on each Prometheus)
         ↓ (uploads to object storage)
    MinIO / S3 bucket
         ↓ (queried by)
    Thanos Query
         ↓ (visualized in)
    Grafana

# Prometheus with Thanos sidecar (docker-compose)
version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-storage:/prometheus
    command:
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=24h'

  thanos-sidecar:
    image: quay.io/thanos/thanos:latest
    volumes:
      - prometheus-storage:/prometheus
    command:
      - 'sidecar'
      - '--tsdb.path=/prometheus'
      - '--objstore.config-file=/etc/thanos/objstore.yml'
      - '--grpc-address=0.0.0.0:10901'

  minio:
    image: minio/minio:latest
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    ports:
      - "9000:9000"
    command: server /data

  thanos-query:
    image: quay.io/thanos/thanos:latest
    ports:
      - "9090:10902"
    command:
      - 'query'
      - '--store=thanos-sidecar:10901'

When Thanos makes sense:
– You have multiple environments (dev, staging, prod) with separate Prometheus instances
– You need metrics retention beyond 2 weeks
– You want to query across regions or clusters
– You’re running Kubernetes across multiple zones

Cost consideration: Object storage costs are low (S3 is ~$0.023/GB/month), so Thanos typically pays for itself through reduced Prometheus resource requirements and better data retention.

Comparison Table: Self-Hosted Monitoring Stacks

Distributed Tracing: The Missing Piece

Most teams neglect tracing, but when you’re debugging distributed microservices, traces are invaluable.

Popular open-source options:
– Jaeger (CNCF, mature, complex)
– Zipkin (simpler, older)
– Tempo (Grafana’s tracing tool, integrates with Loki)

Tempo is gaining traction because it integrates seamlessly with Prometheus + Grafana + Loki:

# tempo-config.yml
server:
  http_listen_port: 3200

distributor:
  rate_limit_bytes: 5000000

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces

metrics_generator:
  registry:
    enabled: true
  storage:
    path: /var/tempo/wal

Your application sends spans to Tempo, Grafana shows traces alongside logs and metrics, and you can trace a request from frontend to database.

Deployment Strategies for Production

Containerized Stack (Docker Compose)

Best for small teams, single-server deployments:

docker-compose up -d

Pros: Simple, fast to iterate. Cons: Not highly available, no automatic failover.

Kubernetes-Native Stack

If you’re running Kubernetes, deploy via Prometheus Operator or Grafana Stack Helm charts:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack

Pros: Native HA, automatic restarts, rolling updates. Cons: Adds operational complexity.

Hybrid: Self-Hosted + Managed Alerting

Some teams run Prometheus + Grafana self-hosted but use a managed service like Datadog or Managed Grafana for critical alerting. This gives you data sovereignty while offloading reliability concerns.

Common Pitfalls and How to Avoid Them

1. Metric cardinality explosion
– Problem: Too many unique labels (e.g., user_id in metrics) = out of memory
– Solution: Use recording rules to reduce cardinality; keep labels finite

2. Retention vs. storage tradeoff
– Problem: Want 1 year of metrics but storage fills up
– Solution: Use Thanos for long-term storage with downsampling

3. Alert fatigue
– Problem: Too many alerts, no one listens
– Solution: Be ruthless about alert thresholds; use silence rules for maintenance windows

4. Silent alerting failures
– Problem: Alert rules evaluate but notifications never reach you
– Solution: Test alert routing; monitor Alertmanager itself

5. Grafana dashboards become unmaintained
– Problem: 200 dashboards, no one knows which are used
– Solution: Tag dashboards, document dashboards-as-code in git

Real-World Example: Monitoring a SaaS Platform

Here’s how we’d set up monitoring for a typical SaaS platform (API servers, databases, caches, worker queues):

# Stack: Prometheus + Grafana + Loki + Alertmanager
# Infrastructure: 3 VMs (monitoring VM, app VM, database VM)

# Monitoring VM:
- Prometheus (stores 2 weeks of metrics)
- Alertmanager (sends to Slack + PagerDuty)
- Grafana (visualizes everything)
- Loki (centralized logging)

# App VM + Database VM:
- Prometheus exporters:
  - node_exporter (system metrics)
  - postgres_exporter (if using Postgres)
  - redis_exporter (if using Redis)
  - Custom application metrics

# Alerting rules:
- API latency p99 > 500ms
- Database connections > 90% pool
- Disk usage > 80%
- App error rate > 1%
- Memory usage trending up (predictive)

With this stack:
– Cost: ~$100/month (3 modest VMs)
– Alerting latency: ~1 minute
– Data retention: 2 weeks for detailed, 1 year with Thanos (optional)
– Team setup time: 1-2 days

Compare to SaaS monitoring: Same functionality = $1500-5000/month depending on volume.

When to Consider SaaS Monitoring Instead

Even though this article is about self-hosted, be honest:

• High-touch enterprises: Need compliance features (HIPAA, SOC 2) that require managed infrastructure
• Small teams with no ops bandwidth: Managed services handle reliability
• Massive scale (1M+ metrics/sec): Infrastructure becomes your full-time job
• Multiple vendors requirement: Some enterprises require multi-vendor monitoring

There’s no shame in using Datadog or New Relic if it means better sleep at night.

Getting Started: The Practical Path

1. Start simple: Deploy Prometheus + Grafana on a single VM
2. Add structure: Set up basic alerting with Alertmanager
3. Aggregate logs: Add Loki and wire it into Grafana dashboards
4. Solve real problems: Only add Thanos, tracing, or other components when you hit actual limits
5. Document everything: Exporters, alert rules, and Grafana dashboards should be version-controlled

Most teams make this journey over 6-12 months, adding components as needed rather than building a perfect stack day one.

Conclusion

The best monitoring stack for self-hosted infrastructure is the one your team can actually maintain and that solves your specific problems. For most DevOps teams running 5-100 servers, Prometheus + Grafana + Loki + Alertmanager hits the sweet spot: lightweight, powerful, well-documented, and low operational overhead.

As you scale beyond 100 servers or need multi-cluster monitoring, introduce Thanos for long-term storage and cross-cluster querying. If logs become your bottleneck, evaluate whether Loki’s label-based approach meets your needs or whether full-text search (ELK) becomes necessary.

The key is starting with simplicity and evolving your stack based on real operational needs, not hypothetical ones. Your monitoring infrastructure should make your life easier, not become another thing to maintain.

Start with Prometheus + Grafana this week, measure your metrics volume and storage needs, then plan your evolution from there.