Observability v1.0 launch: monitoring a cluster in public

Launching observability v1.0 — the telemetry stack for everything I run, and the public status page that ships with it, at lans-h.cc/dashboard.html.

I have four apps, a shared Postgres, and the cluster’s own ingress and TLS plumbing all competing for the same memory, and I had no numbers on any of it. I wanted a fast way to answer five questions without SSH-ing anywhere: am I running out of disk, am I running out of memory, is everything actually up, is Postgres serving from cache or from the volume, and what did all of that look like an hour ago.

The second motive was a self-imposed one, and it is the one that shaped the build: I wanted to find out whether I could get a full-function telemetry stack out of a limited budget. Free tier only, no new persistent storage, no new public listener. Partly because that is what I have, and partly because it is a more interesting problem than the version with money in it.

The architecture

The finding that shaped everything else: a monitoring stack has four jobs — collect, store, query, present — and only the first has to run on my hardware. The numbers exist in the kernel and the container runtime, so a collector has to be there. Storage and query can be anywhere, and they are the expensive half.

That mattered because of the rule I started from:

A monitoring system that competes for the resources it is monitoring will lie to you exactly when you most need it to be honest — during a resource crunch.

So the split follows the requirement rather than the convention:

Job	Where it runs	Why there
Collect	in the cluster	the numbers only exist next to the kernel and the containers
Store	Grafana Cloud	a managed TSDB whose retention I never operate
Query	Grafana Cloud	it arrives with the storage
Present	Grafana Cloud for me, a static file for everyone else	two audiences, two sets of rules — below

TSDB runs on the free tier, with an internal full query dashboard

What lands on the free tier is precisely the half I would otherwise be running: a time-series database, its retention, and a query engine on top. Getting that for nothing is what made a limited budget workable, and it leaves the cluster doing one thing.

For me it also means an authenticated Grafana with the full query engine behind it, where I can ask anything, including absolute numbers. That is the tool I actually use when something is wrong.

The public half is the part I wanted to get right, though. I decided early that it should live on a URL a stranger can open rather than stay a dashboard only I can see — which sounds like a small decision and is not, because it means every question about what I am willing to say out loud has to be answered before launch instead of after. It also means the public half cannot work the way the internal one does, and the obvious design for it fails for a reason worth spelling out.

The Grafana I did not install

The obvious answer is a self-hosted Grafana with anonymous access on and a dashboard of panels I chose deliberately. I got some distance into that before it fell over: an anonymous Grafana exposes a datasource proxy, and the proxy forwards arbitrary PromQL. A visitor never has to look at my panels — they can query anything the collector has ever sent and get an answer. A panel allowlist is decoration when there is a query interface behind it.

The other two columns were not close either:

	Anonymous Grafana	Static snapshot (shipped)
Redaction	a proxy answers queries no panel makes	the visitor gets a file; there is no query interface
Cluster cost	+150–250 MB resident, on the namespace that is already the heaviest thing running	one curl and one jq for about two seconds every five minutes
New surface	another deployment, another ingress, another public listener to patch	nothing new listens; the existing nginx serves one more static path

What ships instead is a systemd timer on the host that runs a fixed list of PromQL against Grafana Cloud, reduces the answers to one small JSON document, and publishes it as a ConfigMap the website’s nginx pod mounts. The page fetches that file and draws it: capacity for CPU, memory and the system volume, a per-workload breakdown, a healthy-target count, the Postgres cache hit ratio, and a rolling history behind each one.

Redaction lives in the producer, never in the page

The allowlist lives in the producer, before the ConfigMap is written, rather than in the layout: data.json is a public URL, and anything left in it is published whether or not a panel renders it. “The layout does not show it” is not a control. Namespaces and databases are allowlisted by name, so anything created later is dropped; mountpoints are renamed rather than published as paths. Health follows the same rule — a target that is down is published as a count, never as a name, because naming it tells a stranger where to aim.

How the data actually moves

One agent does the collecting: Grafana Alloy, configured as a short chain of components — prometheus.scrape for each source, prometheus.relabel in front of the noisy one, and a single prometheus.remote_write that ships everything outward over TLS.

Four sources feed it, and none of them knows any of the above exists:

Source	What it reports	What it ends up as
node-exporter	kernel and host — CPU, memory, filesystem, load	the capacity tiles; it also inlines two metrics that have no exporter at all, which a timer writes into files it reads
cAdvisor	per-container CPU and memory	the per-workload table
postgres-exporter	connections, cache hit ratio, database sizes	the cache hit ratio tile
kubelet	cluster-level state	~88% of the series volume, and almost none of the page — see the filter below

Pull, and what it buys

Every source exposes a /metrics endpoint and waits. No source knows Grafana Cloud exists, none holds a credential, and none needs reconfiguring when the collector changes. Adding a target is a change to the collector, not to the target.

Pull is what makes up possible, and up is the whole liveness story. Because the collector goes looking, a target that stops answering produces up=0. Silence becomes a signal. Under a push model, a dead sender and a healthy-but-idle sender look identical — you cannot tell “broken” from “nothing to say”, because both are an absence of packets. The 6/6 counter on the public page is entirely a product of that one choice.

Pull is not free. It costs service discovery — the collector must know where every target is, whereas a pushing agent only needs to know one address. It also costs reachability: the collector has to be able to open a connection to each target, which is fine inside a cluster and would not be across a NAT.

The only push in the design is the last hop, collector to storage, and it is outbound HTTPS. That direction matters as much as the protocol: nothing in this stack requires an inbound port on any host, so the entire observability system adds zero public attack surface.

The Filter

A series is a finer unit than the word suggests: not one metric, but one metric plus one exact combination of labels. A single container-memory metric name is thousands of series, one for every container in every pod in every namespace.

The first full scrape went straight past 10,000, and almost all of it came from kubelet — about 46,560 series, roughly 88% of everything collected, and almost none of it headed for the page. So that job got a prometheus.relabel stage in front of it, which despite the name is there to drop: 47 series survive and the other ~46,513 do not. That one rule is why the stack fits in the free tier.

Going over the cap would have been the worst failure available here, and it took me a while to see why. It is silent and partial: everything keeps running, some series simply stop arriving, and there is no up=0, no error, no gap that looks like a gap. Every other failure mode degrades visibly; this one degrades by lying.

What the filter does not do is make those series free. The full response still crosses the wire and is parsed into label sets before any rule looks at it — the discard happens after the allocation, not instead of it. It buys quota and bandwidth, and none of the memory. Frequency is what helps there: that job scrapes every five minutes rather than every minute, which also cuts work on the target, since kubelet has to serialize all 46,560 series before the collector throws them away.

Where the data is, at any given moment

A question worth being able to answer about any pipeline: if I unplug it right now, what is sitting where? Here the answer is almost always “nothing, at this stage”.

The kernel is the only thing keeping running totals — CPU seconds since boot, bytes written, and so on. node-exporter does not copy them anywhere. It reads them when asked and builds its reply on the spot, so /metrics is a rendering of the current state rather than a store of past states. Ask twice and you get two renderings; nothing was kept in between.

The collector is the one component that holds anything, and only briefly. Its write-ahead log keeps the samples Grafana Cloud has not acknowledged yet and drops each one the moment it has. That makes it a shipping queue, not a history — its size tracks how far behind the network is, not how long the cluster has been running.

Which is why the whole stack owns no persistent volume, and the direction of that sentence matters. I did not go looking for a way to avoid one. A queue that empties itself has nothing to persist, so an emptyDir is the correct size of storage for it rather than a corner cut to fit a budget.

So there is exactly one durable copy of anything, and it is not on my hardware:

Hop	What moves	What is held here
kernel → /metrics	nothing until asked	the running totals, in the kernel
/metrics → collector	one HTTP GET per interval	nothing — the reply is a rendering, not a store
collector	parse, relabel, append to the WAL	the WAL
collector → cloud	batched samples over TLS	the WAL, until the write is acknowledged
cloud	accepted samples	the only durable copy
cloud → page	a fixed set of queries, every 5m	the last good snapshot

Constraints and budgets

Three ceilings governed the whole build, and every design decision above is one of them cashed out:

Budget	Ceiling	What it bought
Series	10,000 active	the kubelet filter, and metrics before traces or profiles
Storage	zero new persistent volumes	WAL in an emptyDir; retention is the cloud’s problem
Public surface	zero new listeners	a static file instead of an anonymous Grafana

What an outage costs

Owning no storage has one consequence worth knowing before an outage rather than during one. A counter is redundant with every later sample — miss ten scrapes and the eleventh still contains what happened in between. A gauge is the only record of its instant: if memory peaked at 90% during the outage, that fact exists nowhere afterwards. Because scraping and shipping are decoupled, where the break happens decides whether a gauge dies at all:

Break	Still scraping?	Counters	Gauges
node-exporter down	no	total intact	permanently lost
collector down	no	total intact	permanently lost
cluster ↔ cloud network down	yes	intact	backfilled from the WAL
over the series quota	yes	partial	partial
public snapshot timer fails	yes	intact	intact — public page only

A network outage does not stop collection: the collector keeps writing into its WAL and ships the backlog on reconnect, with the original timestamps. Gauges survive that. They do not survive anything that stops the scrape itself.

What v1.1 is for

Four things, roughly in the order I expect to close them: the collector does not scrape itself, so its own failure arrives as silence; up counts detect “present but not answering” and never “gone”; the two file-backed metrics have no freshness check, so a dead timer would draw a flat line rather than a gap; and the load-per-core tile has read 0 since the first snapshot, which I left on the page rather than dropping, because a visibly wrong number gets fixed and a quietly missing one does not.

Traces and profiles come after those, and the order is deliberate: metric volume follows the shape of the fleet, which I control, while trace volume follows traffic — costing most exactly when the system is busiest.

Summary

I set out to build the most complete observability I could on the smallest resource budget I could, and got a good way there — but the budget is what wrote the architecture, not the ambition. Every choice that looks independent is downstream of a ceiling. Pull instead of push, because it makes silence legible and hands you up for free. A static file instead of a Grafana, because an allowlist a visitor can walk around is not an allowlist. Ratios only, because ratios are the only numbers that survive being divided into each other. And 46,513 series on the floor, because the free tier holds 10,000 and I would rather answer five questions reliably than fifty questions until the end of the month.