Home › Topics › Monitoring Architecture › Database Backends

The Monitoring Database: Backend Choices and Growing Pains


Under every monitoring dashboard is a database taking a quiet beating. Each ping result, each interface counter, each CPU reading — every answer your monitoring system ever collects becomes a row, written now and consulted for months. The dashboards and maps are the showroom. The database is the warehouse floor where the actual work happens, around the clock, whether or not anyone is watching.


Most monitoring tools let you choose what runs that warehouse. One option is an embedded database that ships inside the tool and needs zero care. The other is a full database server you operate alongside it. New admins tend to either fear this choice ("do I need to become a database administrator?") or ignore it until the day the dashboard takes thirty seconds to load. Neither is necessary. The choice follows a well-worn ladder, and the symptoms of outgrowing a rung are recognizable in advance. Moving up is a scheduled afternoon rather than a crisis — if you know what to look for.


This guide explains why monitoring data stresses storage in its own peculiar way. It walks the backend ladder honestly — what each rung gives, what it costs, where it fits. It catalogs the growing-pain symptoms with first responses and covers the care every monitoring database needs regardless of rung.


In this guide

  • Why monitoring data beats up databases
  • The backend ladder
  • The growing-pains symptom catalog
  • When and how to move up a rung
  • Care and feeding, on every rung
  • Where the database itself should live
  • The backend decision table

Why monitoring data beats up databases


Office applications use databases politely: a record created here, one updated there, long quiet stretches. Monitoring does not. Its workload is a steady drizzle that never, ever stops. The arithmetic makes the point better than adjectives — take an unremarkable small setup and multiply:


50 devices x 8 metrics each          =    400 readings per cycle
400 readings, every 60 seconds       =    576,000 rows per day
576,000 rows x 365 days              =  ~210 million rows per year (if kept raw)

Four hundred checks is a modest estate — and it is already writing half a million rows a day. Scale to a few hundred devices or quicken the polling and you are into millions of rows daily without doing anything exotic. (This is exactly why retention policies exist: nobody actually keeps a billion raw rows, as we will get to.) Beyond the volume, the shape of the workload is distinctive in three ways:


  • Write-heavy, in tiny sips. Constant small inserts, arriving in time order, appended forever and almost never updated. The database spends its life absorbing a drizzle, not serving bursts.
  • Delete-heavy, in gulps. Old data ages out, so retention purges periodically delete rows by the million. Big deletes are among the most expensive things you can ask a database to do — and monitoring asks on a schedule.
  • Reads that scan ranges. The classic read is the graph query: "every value for this interface over the last 30 days". It sweeps across a time range, sometimes across months for a report, colliding with the drizzle of writes that never pauses.

Write-heavy, delete-heavy, range-scanning: that combination stresses storage differently than any office app. It is why a database that handles your accounting without breaking a sweat can wheeze under monitoring duty. It also explains a sizing rule from elsewhere in this pillar: monitoring servers want fast disk more than fast anything else.


It helps to know that "the monitoring database" is really three kinds of data sharing a roof, with very different values per byte. Configuration — your devices, checks, thresholds, notification rules — is tiny and precious: months of tuning that cannot be regenerated. Metric history is the giant: those millions of readings, individually worthless, collectively the baselines and graphs you reason with. Events and alerts — what fired, when, who was told — sit in the middle: modest in size, gold during post-incident forensics. Keep the proportions in mind when you think about backups: the smallest slice matters most, and the biggest slice is the one making every operation slow.


The backend ladder


Monitoring tools commonly offer their storage in two rungs, and the industry's shorthand names are worth learning even though the exact products vary.


Rung one: the embedded database

An embedded database (SQLite is the famous one) is a library inside the monitoring application writing to a single file on disk. There is no database service to install, no port, no separate credentials, no administration whatsoever. It simply works the moment the tool starts, which is why tools ship it as the default. For small estates it is genuinely excellent. The single-file design is fast for exactly the append-and-scan pattern monitoring produces. And "zero care" is not marketing — there is truly nothing to tune.


Its ceilings are just as honest. Concurrent writes serialize — one writer at a time gets the file. So when heavy polling, a big report, and a retention purge all want the database at once, they queue. You feel it as stalls. The file grows relentlessly unless retention is enforced, and space from purged rows is not returned to the operating system until a compaction pass runs. And backing up a single file while it is being written to takes care. A naive file copy of a live database can capture a torn, inconsistent snapshot. None of these are defects; they are the shape of the rung.


Rung two: the server database

A server database is a separate service the monitoring tool connects to over a credentialed connection. The MySQL/MariaDB family and Microsoft SQL Server are the common pairings. You gain real concurrent access, so writers, readers, and purges stop queueing behind one another. You gain mature grown-up backup tooling that snapshots a live database consistently. You gain the option to host the database on separate hardware. And you gain room at the top — this rung carries estates the embedded file never could. High-availability designs also tend to live here, since a database both cluster members can reach is a natural foundation.


What it costs: you now run a database. It must be installed, patched, granted memory, backed up, and occasionally tuned; it has credentials to manage and a port to protect. It is one more thing on the dependency list of the monitor itself. If your organization already operates database servers, the marginal cost is small — one more schema on managed infrastructure. If it does not, be honest that the monitoring database will be the first database anyone here has administered, and budget the learning accordingly.


You may also hear about a third species: dedicated time-series databases, purpose-built engines for exactly this append-and-scan workload. They exist, they are excellent, and at very large scale they are the norm. But integrated monitoring tools typically offer the two rungs above and handle the time-series optimizations internally. And that is genuinely fine at the sizes this guide serves. Know the term so vendor pages do not intimidate you; do not go shopping for one because a blog said your two-hundred-device network needs it.


Here is where each rung fits, in capability terms. Take an estate of dozens of devices at ordinary polling intervals, with a couple of dashboard users and sane retention. It can live on the embedded rung for years and never notice a limit. Hundreds of devices, brisk polling, long retention, many simultaneous report readers, or high-availability ambitions all pull toward the server rung. But the truest boundary is not a device count — it is the symptom list below showing up and refusing to leave.


The growing-pains symptom catalog


Databases outgrow their rung gradually, then suddenly. These are the classic early signs, what each usually means, and the first response that is not "panic":


SymptomWhat it usually meansFirst response
Dashboards and graphs load noticeably slower each monthRange scans crawling over ever-larger tablesEnforce retention; shorten default graph ranges; run the tool's maintenance job
Gaps in graphs at busy times; polling cycle overrunsWrites queueing behind reads or purges — the concurrency ceilingMove reports and purges to quiet hours; if it recurs, that is a ladder signal
Database file (or volume) steadily eating the diskRetention absent, or purged space never compactedSet retention today; schedule a compaction; alert on the volume's free space
Purge or cleanup jobs running for hoursDeleting millions of rows in one giant bitePurge in smaller batches, more often, at quiet hours
Backup window stretching, or backups quietly failingDatabase outgrew the naive backup methodSwitch to a hot-backup method the backend supports; prune before backing up
Sporadic "database is locked" or "busy" errors in the tool's logMultiple writers colliding at the embedded rung's limitReduce simultaneous heavy operations; treat repeats as the clearest climb signal there is

One symptom deserves its own sentence: the monitoring tool's own log is where these signs appear first, often months before users complain. A slow-query warning or a lock timeout in that log is the database clearing its throat. Glance at the log monthly; it is the cheapest database diagnostics you will ever run.


When and how to move up a rung


Climb when the symptoms recur after you have done the honest tuning — retention enforced, purges scheduled sensibly, graph ranges sane. Climbing to escape self-inflicted bloat just ships the bloat somewhere with more disk. And rule out the hardware first. If the database sits on a slow or shared spinning disk, moving the same file to fast local solid-state storage is often the cheapest "migration" available. That move fixes more symptom-catalog entries than any backend change. Beyond recurring symptoms, three signals justify the move on their own. One is that the estate is about to grow substantially (a merger, a big rollout). Another is that you want high availability the embedded rung cannot provide. The third is that your organization already runs managed database infrastructure and the monitoring data belongs with it.


The migration itself is calmer than its reputation. Exact steps are tool-specific — read your tool's migration documentation before touching anything — but the shape is nearly universal:


  • 1. Stand up the target database: service installed and patched, a dedicated schema, a dedicated credential with rights to only that schema, disk sized with generous headroom.
  • 2. Rehearse on a copy. Export or migrate a snapshot of the current database into the target and let the tool (a test instance, if you can) read it. The rehearsal finds the surprises while they are still free.
  • 3. Schedule the cutover like the maintenance it is: announced window, quiet hours, a written rollback line ("if X by time Y, we revert to the old backend").
  • 4. In the window, stop collection, take a final backup, and run the migration or export/import. Point the tool at the new backend, start, and verify. Check for fresh data arriving, old graphs intact, and alerts firing on a test check.
  • 5. Keep the old database file archived read-only for a comfortable while. It is your parallel-run comfort: history you can still consult if anything about the import is later questioned.
  • 6. Watch the first week: write rates, query times, the tool's log. Then add the new database service itself to your monitoring, because it just became infrastructure.

Total downtime is typically the length of the copy. The work is almost entirely preparation and verification — which is to say, the work is optional. Skipping it is how migration horror stories get written.


Care and feeding, on every rung


Whichever rung you occupy, four habits keep the warehouse floor safe. None takes more than minutes to set up, and each one exists because somebody's monitoring died without it:


  • Retention enforcement is the primary size control. Decide how long raw readings live and when they collapse into coarser summaries, and make the tool enforce it. This one policy determines whether your database weighs gigabytes or hundreds of them — the retention policy guide walks the actual numbers.
  • Alert on the database volume's free space. The monitor must watch its own foundation, because nothing else will. A full database volume does not degrade politely; it stops the monitor cold — and with it, every alert about everything else.
  • Back up the database like it matters, because it does. It holds every baseline, every incident's history, every threshold you ever tuned. Use a method that produces a consistent copy of a live database. That means the embedded rung's snapshot mechanism or the server rung's backup tooling, never a naive copy of files mid-write. Restore-test it occasionally. Monitoring the Monitor covers protecting the whole system.
  • Run the maintenance jobs. Server databases want their index and statistics upkeep on a schedule; embedded databases want their periodic compaction. Ten quiet minutes a week, or a mysteriously slowing system six months from now — the choice ages well in only one direction.

Tip: whatever backup schedule you pick, do one restore drill before you trust it. Restore last night's backup to a scratch location and open it. A monitoring database backup that has never been restored is in the same category as a fire extinguisher that has never been inspected: probably fine. And you are betting an outage on "probably."


Where the database itself should live


On the embedded rung there is no choice: the database is a file on the monitoring server, full stop. On the server rung you choose — database on the same machine as the monitor, or on a separate one. The default answer at small scale is same machine, or at least same room. Two reasons. First, latency: the monitor writes constantly, in tiny operations, and every one crosses the gap to the database. A database across a slow or busy link drags the entire collection cycle. Second, failure coupling runs in only one direction here. If the database is down, the monitor is down regardless of where either lives. So separating them adds a second machine that must stay alive without removing any dependency.


External placement earns its keep in two situations. One is that your organization already runs a managed, backed-up, cared-for database service. In that case, the monitoring data inherits all that maturity for free. The other is that you are building high availability. In that case, a database reachable by more than one monitoring node becomes part of the design. Both are fine reasons. "The database team said the VM cluster had space" is not. That is especially true if that cluster is on the monitor's own patient list, a trap the server placement guide covers in detail.


One placement detail pays off on every rung: give the database its own volume, separate from the operating system's. A database that fills a shared system drive takes the whole server down with it. A database that fills its own volume stops the monitor but leaves the OS healthy enough to log in and fix things. A dedicated volume also makes the free-space alert from the previous section clean and unambiguous. You get one number, one meaning, and no wondering whether it was temp files or history that ate the disk.


The backend decision table


Estate, roughlySensible rungWatch for
Dozens of devices, default polling, one or two dashboard usersEmbedded — the default is correctSet retention on day one; alert on the database volume
Low hundreds of devices, several users, standard retentionEmbedded still fine; reassess on symptomsBusy-hour stalls, lock errors, backup method aging out
Hundreds and climbing, long retention, many report usersServer rungSomeone must own the database; maintenance jobs scheduled
Any size, with high-availability requirementsServer rungDatabase placement becomes part of the HA design itself

Hold the table loosely — polling frequency, retention, and report habits move the boundaries more than raw device counts do. The symptoms catalog outranks any row here.


Where to go next


Here is the reassurance to carry out of this guide: start simple; the ladder exists for when you need it. The embedded default is the right choice for most small estates. Outgrowing it announces itself in recognizable symptoms long before anything breaks. And the climb is a rehearsed afternoon, not a leap. Set retention, watch the disk, and back up like you mean it. Then the database under your dashboard will stay the most boring part of your monitoring — which is the highest compliment infrastructure can earn. The wider design questions around it — where the server sits, how it reaches devices, when the system grows distributed — are the rest of the Monitoring Architecture pillar. When growth itself is the story, Scaling Monitoring picks up the thread.


Put this into practice with SysPeep Monitor

This is exactly the ladder SysPeep Monitor ships. It has an embedded SQLite backend by default — nothing extra to install for a small estate. MySQL/MariaDB and SQL Server are supported when your symptoms say it is time to climb, with high-availability clustering at the top. Start on the free trial with the default backend and move up when your data says so.

Download Free TrialLearn About SysPeep