Prometheus and Grafana: The Metrics-First Stack
The first time I opened an exporter page I thought something was broken. It was plain text, four hundred lines of names and numbers, served by the web server I was meant to be monitoring. That page is the whole idea. The first three families in this tour begin with a central server that decides what to ask; the fourth begins with the thing being monitored. Every server, container, and application, and through a translator every switch, publishes a page like that one. The monitoring system visits those pages on a schedule, remembers what it saw, and answers questions about it later.
That inversion sounds small and changes everything. There is no list of checks and no list of items. There are only time series — numbers with names and labels, recorded over time — and queries against them. A graph is a query. An alert is a query that has been true for a while. A dashboard is a wall of queries. The stack's two best-known open-source projects divide the work: Prometheus collects, stores, and evaluates alert rules; Grafana draws.
This tour explains the pull model and exporters, and the label-and-query mental model that makes the stack click. It explains how alerting and the alert router work, and what Grafana adds. It also explains — honestly — how well all of this fits network monitoring in particular. It closes with strengths, learning curve, a table of what carries over when you migrate between families, and what to check in the projects' current documentation.
In this guide
- The design: pull numbers from exporters, store them with labels
- Labels and queries: the mental model
- Alerting: rules that are queries, and a router that delivers
- Grafana: the visualization layer
- Network-monitoring fit: SNMP through an exporter
- Strengths, learning curve, and where it fits
- Migration paths between the families
- What to check in the current docs before committing
- Where to go next
The design: pull numbers from exporters, store them with labels
Start with the page of numbers. An exporter is a small program that sits next to something — an operating system, a database, a web server, a switch. It serves that system's current measurements over HTTP in a simple text format. Open the page in a browser and you see lines like these:
# HELP node_filesystem_avail_bytes Filesystem space available to non-root users.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",mountpoint="/"} 41200000000
node_filesystem_avail_bytes{device="/dev/sdb1",mountpoint="/data"} 903000000000
# HELP node_network_receive_bytes_total Total bytes received.
# TYPE node_network_receive_bytes_total counter
node_network_receive_bytes_total{device="eth0"} 918273645123
node_network_receive_bytes_total{device="eth1"} 44120933
Each line is a metric name, a set of labels in curly braces, and a value. The labels are what make one metric name describe many things. node_filesystem_avail_bytes is one name. With the mountpoint label it becomes a separate series for every filesystem. The TYPE line says whether the number is a gauge (a right-now value that goes up and down) or a counter. A counter is an ever-climbing total that only makes sense as a rate. Those two words will be familiar if you have read any SNMP walk output; the ideas are identical. Treat a counter as a gauge and the graph climbs forever, which looks like success.
Now the collector. Prometheus is configured with a list of targets grouped into jobs — "the servers job has these twelve addresses on this port." It also has a scrape interval. On every interval it fetches each target's page. It attaches two labels of its own (job and instance, the latter being the target's address). It writes every line it read into its own purpose-built time-series database, stamped with the time. That fetch is called a scrape, and the fact that the server fetches rather than the target sending is the pull model. A minimal configuration is short:
global:
scrape_interval: 30s
scrape_configs:
- job_name: servers
static_configs:
- targets: ["web-01:9100", "web-02:9100", "db-01:9100"]
- job_name: snmp-switches
static_configs:
- targets: ["sw-core-01", "sw-edge-02"]
# ... plus relabeling that routes these through an SNMP exporter (see below)
The target list does not have to be typed by hand. Service discovery lets the server ask a cloud provider, a container orchestrator, a DNS zone, or a file on disk for the current list of things to scrape. That way, servers which appear and disappear are monitored while they exist. This is the stack's answer to auto-discovery, and it is excellent for infrastructure that is born from an API. It knows nothing about walking a subnet for switches; more on that later.
Storage is local and compact: the time-series database is built for exactly this shape of data, indexed by labels. It keeps a configurable window of history. Long-term retention and querying across many servers are handled by separate projects layered on top; the current documentation lists the options.
Labels and queries: the mental model
Here is the moment the stack either clicks or does not. Every question you will ever ask is a query in the server's query language. And every query works the same way: select some series by name and labels. Optionally transform them over a time window. Optionally aggregate them by a label. Three examples say more than a page of explanation:
# 1. Free space on every root filesystem, right now
node_filesystem_avail_bytes{mountpoint="/"}
# 2. Incoming bandwidth on eth0 of every server, in bits per second,
# averaged over the last five minutes (counter -> rate)
rate(node_network_receive_bytes_total{device="eth0"}[5m]) * 8
# 3. Total incoming bandwidth per job, summed across all servers
sum by (job) (rate(node_network_receive_bytes_total[5m]) * 8)
Read the second one slowly, because it is the subtract-and-divide arithmetic every bandwidth graph uses, written as one expression. Take the counter, compute its per-second rate over five minutes, and multiply by eight for bits. In the other families that arithmetic is hidden inside the tool. Here it is in your hands, which is exactly why the third query is possible at all. Once rates are just values, adding them up by any label is one more function call. Nobody had to anticipate "bandwidth per job." You asked, and the labels answered.
This is what "query-centric" means. Metrics are one of the three kinds of telemetry. The metrics, logs, and traces guide explains how they differ. The metrics-first stack takes the numeric kind as far as it will go. The mental model to read its documentation with: exporters expose, the server scrapes, series get labels, queries answer questions, and everything you see is a query.
Tip: label discipline is the stack's equivalent of configuration discipline. A label with many possible values (a user ID, a request path with IDs in it) multiplies series and swamps storage. Keep labels to things with a bounded set of values — host, job, interface, environment — and the stack scales gracefully.
Alerting: rules that are queries, and a router that delivers
An alerting rule is a query plus a duration. The server evaluates the query on a schedule. If it returns any series, those series are "pending." If they keep returning for the rule's for duration, they become firing alerts. The duration is the stack's answer to soft-versus-hard states: a blip that clears within it never fires.
groups:
- name: capacity
rules:
- alert: RootFilesystemLow
expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "Root filesystem below ten percent free"
Notice what the rule does not contain: who to tell or how. The server only decides that an alert is firing. It hands the alert — with all of its labels — to a separate component, the alert router (the project calls it Alertmanager). The router's job is everything between "this is firing" and "a human knows." It groups related alerts into one notification. Twenty servers in one rack losing a filesystem at once is one message, not twenty. It routes by label (severity critical goes to the pager, warning goes to chat). It silences alerts during maintenance. And it inhibits lower-level alerts when a higher-level one is already firing (do not page about the web tier if the whole site is down). Those are the same dependency and escalation ideas the other families have, arranged as a label-matching pipeline.
Two consequences follow. First, alerting is entirely yours to write: the stack does not ship with an opinion about what "disk full" means. So the alert design guides are required reading rather than optional. Second, because the router is a separate service, it must itself be monitored and, in serious installations, run as more than one copy.
At Northgate Retail the alert router ran on the one host that had never been added to the servers job. It was "just the alerting box". When its disk filled, the server kept evaluating rules and kept trying to hand firing alerts to a process that was no longer there. For eleven days nothing was reported, which everyone took for good news. The next report came from the till on aisle four.
Grafana: the visualization layer
Prometheus includes a basic interface for running queries and seeing a graph, and that is all it tries to be. Dashboards belong to Grafana, a separate open-source project whose whole purpose is drawing data from other systems. You add Prometheus as a data source, then build dashboards out of panels. Each panel is a query and a way of showing its result: a time-series graph, a gauge, a single big number, a table, a status grid.
Three properties matter here. Grafana is multi-source. The same dashboard can show a panel from Prometheus beside panels from a log system, a relational database, or another monitoring tool. That is how the stack becomes a front end for things it did not collect. Dashboards are files: a dashboard is a structured text document you can keep in source control, template, and share. Communities publish ready-made dashboards for common exporters so you rarely start from a blank page. And Grafana has grown its own alerting and other features over time. You must decide whether to use the server's rules, Grafana's, or both. The current documentation of each project will inform that decision better than this guide can.
The build-versus-buy question for dashboards is when assembling your own views is worth it. It is also when a tool's built-in ones are the better use of an afternoon. That is the subject of the dashboards guides. In this stack the honest answer is that you will be building, and the tooling makes building pleasant. I meant to build six panels; I have forty-one, and I can defend nine.
Network-monitoring fit: SNMP through an exporter
Switches do not serve pages of metrics. To bring them into the stack you use the SNMP exporter concept: a translator that sits between the server and the device. The server scrapes the exporter, telling it which device to ask. The exporter walks the device over SNMP. It converts each OID it was configured to fetch into a metric line with labels, and returns the page. Interface counters come back as series labeled by interface name and index. From there the queries above work unchanged: rate() on a port's byte counter is a bandwidth graph.
The exporter needs to be told which OIDs to fetch and how to label them. This is normally done by generating its configuration from MIB files for your device types. That is a one-time setup with a real learning curve, but once it is done, adding a switch is adding a target. A companion idea, the blackbox exporter, does the same translation for ping, TCP, HTTP, and certificate checks, which covers the reachability side of network monitoring.
So the honest fit assessment. The stack is strong for servers, containers, and cloud services. Exporters are plentiful, service discovery keeps the target list current, and labels map naturally onto how those systems are organized. It is capable for network devices — the numbers are all there, the queries are powerful, and large network operators run it at scale. But it is weaker out of the box for the device-centric view a network administrator expects. There is no device page and no automatic walk of a subnet to find switches. There is no graph-for-every-port until you build the dashboard. There is no inventory of ports, VLANs, and neighbors unless you collect and model it yourself. The network-first tools give you all of that on day one, which is why the two are so often run together.
Strengths, learning curve, and where it fits
Strengths
Flexibility: any question expressible over labeled numbers can be asked, including questions nobody anticipated when the metrics were collected. Scale: the pull model, compact storage, and label indexing handle very large numbers of series, and the components can be run in multiples. Ecosystem: exporters exist for an enormous range of software, and many applications expose metrics natively in the format. Shared dashboards and rule collections shorten the path from install to useful.
Learning curve
The effort goes into the query language and into building from scratch. The language is compact but unfamiliar: rates, ranges, aggregations, and label matching take a few weeks to become fluent in. Until then, every dashboard and alert is slow to write. Then, because the stack is parts rather than a product, you assemble the parts. You decide which exporters, which dashboards, which rules, what "healthy" means for each thing, and how the router routes. Shared dashboards help, but they are starting points, not finished monitoring. Expect the first month to feel like building a monitoring system, because it is.
Where it fits
Teams running containers, cloud services, and modern applications, where the things being monitored appear and disappear and already speak the metrics format. Teams that read dashboards all day and want to shape them exactly. Organizations large enough that flexibility and scale outweigh the effort of assembly. For a small network with a few switches and servers it works. But you will spend your first weekend building what the other families hand you on install. That is a fair trade only if you value the flexibility afterwards.
Migration paths between the families
Sooner or later most teams move between families, and the question is always the same: what carries over? The answer depends on the atom. A check is a program, a threshold is a number attached to a concept, and history is data in a particular shape. Each survives a move differently. The migrating monitoring platforms guides cover the process; this table covers what to expect for the cargo.
| What you are carrying | From check-based | From all-in-one | From network-first | To or from metrics-first |
|---|---|---|---|---|
| Checks (the questions) | Plugins are portable programs; other families can run them via a plugin-style item or exporter, so the questions survive even when the scheduler changes | Agent items and SNMP items are configuration, not programs; rewrite as checks, exporters, or the new tool's items | Discovered, not written; nothing to carry, the new tool rediscovers | Exporters and scrape targets stay if the new tool can scrape; otherwise the question must be re-asked in the new tool's way |
| Thresholds (the numbers) | Live in check arguments and templates; export the list, re-enter as triggers, rules, or query expressions | Live in triggers; the expressions translate to alerting rules reasonably directly since both look at recent history | Live in alert rules over inventory fields; re-express as queries over labeled series | Live in rule files; the numbers move anywhere, the query logic must be rewritten unless the target also uses queries |
| History (the data) | Mostly state logs plus performance data in a graphing store; rarely worth importing beyond a few key graphs | Rows in a relational database; exportable, but the receiving tool needs a matching series for each item | Round-robin files at diminishing resolution; usually left behind and kept read-only for a while | Time series with labels; importable by tools that understand the format, otherwise archived alongside the old system |
| Dependencies and topology | Hand-written parents; re-enter as inhibition rules or discovered topology | Host relationships and maps; partially re-enterable | Discovered; the new tool must discover or be told | Inhibition rules by label; re-express in the target's dependency model |
The pattern across every row: the questions survive, the numbers survive, the data mostly does not. Plan migrations around a period of overlap in which both systems run, and treat old history as a read-only archive rather than something to convert.
What to check in the current docs before committing
- Long-term storage and high availability: what the server does on its own, what the layered projects add, and which the documentation currently recommends for your scale.
- Exporters for your estate: which exist for your operating systems, databases, and applications, and how the SNMP exporter is configured for your device types.
- Service discovery mechanisms available for wherever your targets live, and how to manage the static list for everything else.
- The alert router's current features — grouping, routing, silencing, inhibition — and its own high-availability setup.
- Where alerting should live, in the server's rules, in Grafana, or both, and how the two projects currently recommend dividing that.
- Security of the whole chain: exporters are web pages of internal facts, so read how to restrict who can scrape them and how to authenticate between components.
- Grafana's data sources and dashboard sharing as they stand today, since both projects evolve quickly and a summary like this one cannot keep up.
Where to go next
The metrics-first stack is built on one inversion — the monitored thing publishes, the collector pulls. It uses one abstraction: numbers with labels, stored as time series, answered by queries. Everything else follows. Alerts are queries with a duration, handed to a router that groups and delivers. Dashboards are queries with a shape, drawn by a separate project that can also draw everything else. Network devices join through a translator and are fully welcome, though the device-centric view must be built rather than found.
If that model excites you, you have found your family, and the dashboards guides will help you decide how much to build. If it sounds like more assembly than your week can absorb, that is a legitimate finding, not a failure. The Guided Tour of the Open-Source Monitoring Stacks has three other families, each of which hands you more on day one in exchange for less freedom later. Fit decides. Open one exporter page before you do; if four hundred lines of numbers look like an invitation rather than a fault, you already know.
Put this into practice with SysPeep Monitor
The metrics-first stack rewards teams who want to build their monitoring exactly as they imagine it. If you would rather compare that against something assembled for you, SysPeep Monitor is one packaged option. It is a Windows service that discovers devices and collects over SNMP, WMI, and SSH. It provides dashboards, maps, and alert rules from a single console. Neither approach is superior in general. The free trial lets you judge the fit on your own network before you commit a weekend to either.
Download Free TrialLearn About SysPeep