Telemetry for the whole fleet, with nothing stored in the cluster
A monitoring stack has four jobs — collect, store, query, present — and only the first has to run on my own hardware. The numbers exist in the kernel and the container runtime, so a collector has to be there; storage and query can be anywhere, and they are the expensive half. Everything below follows from that split, and from one rule: a monitoring system that competes for the resources it is monitoring will lie to you exactly when you most need it to be honest.
| Job | Where | Why there |
|---|---|---|
| Collect | in the cluster | the numbers only exist next to the kernel and the containers |
| Store | Grafana Cloud | a managed TSDB whose retention I never operate |
| Query | Grafana Cloud | it arrives with the storage |
| Present | both | two audiences, two very different sets of rules |
"No TSDB here" is a placement decision, not a reduced build — the expensive half exists, it is just somebody else's operational problem.
| Budget | Ceiling | What it bought |
|---|---|---|
| Series | 10,000 active | the kubelet filter, and metrics before traces |
| Storage | no new volumes | WAL in an emptyDir |
| Public surface | no new listeners | a static file instead of a Grafana |
Every choice on this page that looks independent turns out to be downstream of one of these three.
| Break | Counters | Gauges |
|---|---|---|
| exporter or collector down | total intact | permanently lost |
| network to the cloud down | intact | backfilled from the WAL |
| public snapshot timer fails | intact | intact — public page only |
Scraping and shipping are decoupled, so where a break happens decides whether a gauge dies at all. The backfill guarantee holds as long as the collector's pod is not recreated during the outage.
Every source is passive: it exposes /metrics and waits, holds no credential, and needs no reconfiguring when the collector changes. Because the collector goes looking, a target that stops answering produces up=0 — silence becomes a signal. Under push, a dead sender and a healthy-but-idle sender are indistinguishable. The cost taken is service discovery and reachability, both of which are free inside one cluster.
The kubelet endpoint alone emits about 46,560 series, roughly 88% of everything collected, and almost none of it reaches the page. A prometheus.relabel stage keeps 47 and discards the rest, which is the only reason the stack fits in the free tier. It buys quota and bandwidth and none of the memory — the discard happens after the parse, not instead of it, so frequency is the lever that actually helps.
The WAL is a shipping queue, not a history: it holds only what the backend has not acknowledged, and each entry is discarded on ack, so its size tracks how far behind the network is rather than how long the cluster has run. A queue that empties itself has nothing to persist, so an emptyDir is the correct size of storage for it — not a corner cut to stay inside a budget.
An anonymous Grafana exposes a datasource proxy, and the proxy forwards arbitrary PromQL — a visitor never has to look at the panels chosen for them. A panel allowlist is decoration when there is a query interface behind it. What ships instead is a file: no live path from a visitor to the data, and nothing new listening.
data.json is a public URL, so anything left in it is published whether or not a panel renders it — "the layout does not show it" is not a control. The allowlists run before the ConfigMap is written: namespaces and databases by name, so anything created later is dropped; mountpoints renamed rather than published as paths. A target that is down is published as a count and never as a name.
Percentages feel safe and are not: a percentage divided into one absolute figure gives back the magnitude it came from, and two more divisions give the capacity behind it. Ratios divided by ratios stay ratios, so the public payload carries no absolute quantity anywhere — which is a rule about arithmetic, not about what looks sensitive.
Metric volume follows the shape of the fleet — container and namespace counts, which change slowly and under my control. Trace volume follows traffic, so it costs most exactly when the system is busiest, which is exactly when a silent quota failure would land. Metrics also need no application change, while traces need an instrumentation SDK in every app. The ordering is deliberate; the coverage is the part still owed.
A drawing of the stack, not a service running in it. The live readings are on the status page; the reasoning is in the write-up.