Why monitoring bills grow faster than infrastructure
Commercial observability platforms typically charge per host, per gigabyte ingested, per indexed event and per custom metric. Every new microservice, container and label multiplies those units. Teams respond by instrumenting less, which is the worst possible outcome.
There are two paths to lower cost: tune what you send to your current platform, or move to a self-hosted open-source stack. Most teams should do the first regardless, and many should consider the second.
1. Reduce log volume at the source
Rank log volume by service and message pattern. In nearly every review, a handful of patterns (health checks, verbose framework logs, duplicate request logs, debug output) produce most of the volume.
- Run production at info level, with debug enabled on demand
- Drop health-check and readiness-probe logs
- Sample high-volume success logs; keep all errors
- Remove duplicate logging between proxy and application
2. Tier retention
Most log queries touch the last 24–48 hours. Keep a short hot window (often seven days), move older data to inexpensive object storage, and rehydrate when an investigation needs it. This alone can cut log costs by a third.
3. Control metric cardinality
A single label with unbounded values, such as user ID, request ID or full URL, can create millions of time series. Remove such labels, bucket values and aggregate where detail is not needed.
4. Sample traces intelligently
Tail-based sampling keeps every error and slow trace while dropping most routine successful requests. It typically reduces trace volume by 80–90% without hurting debugging.
5. Consider a self-hosted open-source stack
Prometheus (or a compatible long-term store) for metrics, Loki for logs, Tempo for traces and Grafana for dashboards and alerting form a mature, widely used stack. Instrument with OpenTelemetry so you are never locked in again.
The cost moves from licences to infrastructure and a few hours of maintenance a month. For many teams of 30 engineers or more, that is a fraction of the commercial bill.
6. Migrate safely with a parallel run
Inventory every dashboard and alert, and keep only those that are used. Run both stacks in parallel for two to three weeks, prove alert parity through real incidents or game days, then cancel the commercial platform at renewal.
Tuning versus migrating: what to expect
| Approach | Typical saving | Effort |
|---|---|---|
| Log filtering and levels | 30–60% of log cost | Low |
| Tiered retention | 20–40% of log cost | Low |
| Cardinality control | 30–70% of metric cost | Medium |
| Trace sampling | 50–80% of APM volume | Medium |
| Self-hosted open-source stack | 70–90% of total monitoring cost | Higher, one-off |
Checklist
- Log volume ranked by service and pattern
- Production log level set to info
- Hot retention reduced; archive tier in place
- Unbounded metric labels removed
- Tail-based trace sampling configured
- Dashboards and alerts pruned to what is used
- Open-source stack evaluated with total cost of ownership
