Platform Observability Architecture — lans-h.cc fleet

Telemetry for the whole fleet, with nothing stored in the cluster

A monitoring stack has four jobs — collect, store, query, present — and only the first has to run on my own hardware. The numbers exist in the kernel and the container runtime, so a collector has to be there; storage and query can be anywhere, and they are the expensive half. Everything below follows from that split, and from one rule: a monitoring system that competes for the resources it is monitoring will lie to you exactly when you most need it to be honest.

collector: Grafana Alloy · storage + query: Grafana Cloud free tier · public page: one static file

MY CLUSTER SOURCES — each exposes /metrics and waits. None holds a credential. node-exporter kernel and host cAdvisor per-container CPU and memory postgres-exporter connections, cache hit ratio kubelet cluster state — 88% of the volume scrape — the collector goes and asks Grafana Alloy one Deployment, one replica — a second would scrape the same targets twice prometheus.scrape five jobs, one per source prometheus.relabel drops 46,513 of kubelet’s 46,560 series WAL emptyDir — only what is not yet acknowledged prometheus.remote_write the one push in the design remote_write outbound TLS 443 GRAFANA CLOUD · FREE TIER Time-series database retention and compaction I never operate Query engine arrives with the storage Grafana — the private view authenticated; any question, absolute numbers the tool I actually use when something is wrong every 5 min · a fixed PromQL list systemd timer — on the host, not in Kubernetes reduces the answers to one small JSON document the three allowlists run here: namespaces, databases, mountpoints redaction has already happened ConfigMap web/fleet-public-metrics 96 KB against a 1 MiB ceiling nginx mounts it as a volume kubelet swaps a symlink, atomically one static file A visitor gets a file — there is no query interface behind it every published figure is a ratio, never a magnitude live page ↗ Every arrow that crosses the cluster boundary starts inside it — solid arrows are pushes the cluster makes, dashed arrows are pulls it makes. Nothing out there opens a connection inwards, so the whole stack adds no public listener and no inbound port.

Where each job runs, and why there

JobWhereWhy there
Collectin the clusterthe numbers only exist next to the kernel and the containers
StoreGrafana Clouda managed TSDB whose retention I never operate
QueryGrafana Cloudit arrives with the storage
Presentbothtwo audiences, two very different sets of rules

"No TSDB here" is a placement decision, not a reduced build — the expensive half exists, it is just somebody else's operational problem.

Three ceilings the design was built against

BudgetCeilingWhat it bought
Series10,000 activethe kubelet filter, and metrics before traces
Storageno new volumesWAL in an emptyDir
Public surfaceno new listenersa static file instead of a Grafana

Every choice on this page that looks independent turns out to be downstream of one of these three.

What an outage costs, by where it breaks

BreakCountersGauges
exporter or collector downtotal intactpermanently lost
network to the cloud downintactbackfilled from the WAL
public snapshot timer failsintactintact — public page only

Scraping and shipping are decoupled, so where a break happens decides whether a gauge dies at all. The backfill guarantee holds as long as the collector's pod is not recreated during the outage.

Read more

Why it's built this way — design trade-offs

Pull, not push

Every source is passive: it exposes /metrics and waits, holds no credential, and needs no reconfiguring when the collector changes. Because the collector goes looking, a target that stops answering produces up=0silence becomes a signal. Under push, a dead sender and a healthy-but-idle sender are indistinguishable. The cost taken is service discovery and reachability, both of which are free inside one cluster.

One filter carries the whole budget

The kubelet endpoint alone emits about 46,560 series, roughly 88% of everything collected, and almost none of it reaches the page. A prometheus.relabel stage keeps 47 and discards the rest, which is the only reason the stack fits in the free tier. It buys quota and bandwidth and none of the memory — the discard happens after the parse, not instead of it, so frequency is the lever that actually helps.

No PVC is a consequence, not an economy

The WAL is a shipping queue, not a history: it holds only what the backend has not acknowledged, and each entry is discarded on ack, so its size tracks how far behind the network is rather than how long the cluster has run. A queue that empties itself has nothing to persist, so an emptyDir is the correct size of storage for it — not a corner cut to stay inside a budget.

A static file, not an anonymous Grafana

An anonymous Grafana exposes a datasource proxy, and the proxy forwards arbitrary PromQL — a visitor never has to look at the panels chosen for them. A panel allowlist is decoration when there is a query interface behind it. What ships instead is a file: no live path from a visitor to the data, and nothing new listening.

Redaction lives in the producer

data.json is a public URL, so anything left in it is published whether or not a panel renders it — "the layout does not show it" is not a control. The allowlists run before the ConfigMap is written: namespaces and databases by name, so anything created later is dropped; mountpoints renamed rather than published as paths. A target that is down is published as a count and never as a name.

Ratios only

Percentages feel safe and are not: a percentage divided into one absolute figure gives back the magnitude it came from, and two more divisions give the capacity behind it. Ratios divided by ratios stay ratios, so the public payload carries no absolute quantity anywhere — which is a rule about arithmetic, not about what looks sensitive.

Metrics before traces

Metric volume follows the shape of the fleet — container and namespace counts, which change slowly and under my control. Trace volume follows traffic, so it costs most exactly when the system is busiest, which is exactly when a silent quota failure would land. Metrics also need no application change, while traces need an instrumentation SDK in every app. The ordering is deliberate; the coverage is the part still owed.

A drawing of the stack, not a service running in it. The live readings are on the status page; the reasoning is in the write-up.