Archiving Old Monitoring Data (and Proving You Can Get It Back)
Retention tiers end in "gone", and they should. A database that keeps everything forever is a disk waiting to fill. But some of what the purge deletes is worth more than the disk it sits on. That includes the raw samples from the night the finance system fell over. It includes the monthly summary that proves a link was watched. And it includes the availability figure a contract depends on. Those need to leave the database before the purge reaches them, and land somewhere they will still open in ten years.
That somewhere is the archive, and the archive is easy to get half right. Plenty of shops export something, put it on a share, and never look at it again. That works perfectly until someone needs "May, the finance outage" and finds a folder of files named export (3).zip. The files use a format only a decommissioned server could read. An archive you cannot open is a disk you paid for twice.
By the end of this guide you will be able to say what belongs in the archive and what does not. You will be able to choose formats that outlive your monitoring product. You will know how to name and index the files so a stranger can find them. You will be able to take an incident snapshot on the day with a short procedure. And you will be able to run the annual restore test that turns "we have an archive" into "we have checked." There is an archive index template and a small export script on the way.
In this guide
- Archive versus retention versus backup
- What is worth archiving, and what is not
- Formats that will still open in ten years
- Where to keep it
- Naming and indexing so it can be found
- The archive index template
- Exporting a date range to CSV
- The incident snapshot procedure
- The annual restore test
- War story, checklist, where to go next
Archive versus retention versus backup
Three words get mixed up here, and the mix-up is how archives go missing. Retention is what stays in the live database, at what resolution, for how long. It is the subject of designing retention tiers. An archive is data that has left the live database on purpose but is not gone. It is selected, exported, stored elsewhere, and findable. A backup is a copy of the whole live database, taken so you can rebuild it after a disaster. The monitoring backups guide covers that.
A backup is not an archive, and the difference matters on the day you need one. A backup contains whatever the database held at the moment it was taken. That means it contains exactly what the purge had not yet deleted. Last year's backup holds last year's raw month only if you still have last year's backup, and the product to restore it into. An archive holds what you chose to keep, in a form that does not need the product. It is the difference between keeping every filing cabinet you ever owned and keeping the folder you will be asked for.
What is worth archiving, and what is not
Archive the things that answer a question someone will ask after the database has forgotten. Four kinds of data qualify, and the three stakeholders guide explains who asks for each.
- Monthly summaries. One file per month: availability per service, peak and average utilisation per link, alert counts, device inventory as of month end. Small, cheap, and what the auditor and the capacity planner both want. Produce it while the hourly data still exists.
- Incident snapshots. The raw samples, status changes and alert history for the window around an incident, exported on the day at full resolution. This is the only place last year's raw data will ever exist.
- Availability and SLA reports. The figures a contract, a customer, or a provider dispute may hinge on, in the form they were reported. The uptime evidence file guide covers what those need to contain.
- Capacity register snapshots. A yearly copy of the daily maximums for the counters you plan against, so that trend analysis can reach back further than the database does.
Do not archive raw samples of a quiet Tuesday. Do not archive hourly rows of anything; the monthly summary already carries what they can tell you. Do not archive "the whole database, just in case," because that is a backup with a different label. It inherits every problem a backup has. The test for each candidate is a sentence: "In three years, someone will need this to answer the question X." If you cannot finish the sentence, the purge can have it.
Formats that will still open in ten years
The archive will outlive the product that wrote it. Products get replaced, old versions stop being installable, and the licence key is in an inbox that was deleted with the leaver's account. So the archive must not depend on the product. The format that survives is the dullest one available: CSV. It is plain text with one row per line and commas between the columns. A header row names each column. It can be opened by every spreadsheet, every scripting language, and every text editor ever written.
A CSV needs three things to be useful in a decade. A header row with the unit in the column name: in_bits_per_second, not value. Timestamps in one unambiguous form: year, month, day, hours, minutes, seconds, in UTC, and say so in the header. And a README next to it. This plain text file says what the data is, which system produced it, who exported it, and when. It explains what each column means and what happened that made the data worth keeping. The README is the part most people skip, and the part the person in ten years needs most.
Avoid proprietary binary formats and native database dumps as the only copy. A dump can be restored only by the same engine, sometimes only by the same version. And a product's own export format is readable only by the product. Keep them if the product offers them, but never as the sole copy. A PDF of the report as it was presented is fine alongside the CSV for the auditor. The auditor wants to see what people saw, not to run queries on it. Compress if you must, with a common format, and record the checksum of the uncompressed file too. That way, a bad decompression is detectable.
Where to keep it
Keep the archive off the monitoring server, in two places, with the same backup as everything else you care about. A file share that the normal backup already covers is the right first home. It costs nothing extra. It is backed up without anyone remembering, and it is findable. Cold storage (slow, cheap, offline or nearly so) is the right second home for the copy you hope never to touch. Nothing about monitoring data needs a special repository; it needs the ordinary one, used properly.
Make the archive write-once in practice, if not in technology. Files go in and do not get edited; a corrected export is a new file with a note in the README, not an overwrite. If the share supports versioning or immutable retention, turn it on. Part of the archive's value to the auditor is that it has not changed since it was written. A checksum recorded in the index is how you show that.
Control access, but not so tightly that nobody can find it. An archive only the departed administrator could reach is an archive that does not exist. Two people should know where it is and be able to open it. The location should be written in the retention policy. That is the one document a successor will be handed.
Naming and indexing so it can be found
The question the archive must answer is never "show me everything." It is "May, the finance system outage," asked by someone who was not there. Two things make that answerable. One is a naming convention that sorts by date and says what a file is. The other is an index that maps the human description to the file. Use a folder per year. Use a file name that starts with the period and continues with the type and scope, and one README per folder.
archive/
YYYY/
README.txt
YYYY-MM_monthly-summary.csv
YYYY-MM_monthly-summary.pdf
YYYY-MM-DD_incident_finance-erp-outage_raw.csv
YYYY-MM-DD_incident_finance-erp-outage_status.csv
YYYY-MM-DD_incident_finance-erp-outage_README.txt
YYYY_capacity-register_daily-max.csv
archive-index.csv
The index is a single small table. Keep it as its own CSV at the top of the archive. Ideally, keep it in the ticket system or wiki as well. The index must be findable even when the archive share is not. One row per archived item. It carries the human description, so that "finance system outage" is searchable text rather than a memory.
The archive index template
| Period | Contents | Format | Location | Checksum | Restore-tested |
|---|---|---|---|---|---|
| Month and year | Monthly summary: availability per service, link peaks, alert counts | CSV plus PDF | Share path and cold-storage reference | Hash of each file | Date last opened and graphed, or blank |
| Date and window | Incident snapshot: finance ERP outage, raw samples for 14 series, 15:00 to 21:00 UTC | CSV plus README | Share path and cold-storage reference | Hash of each file | Date last opened and graphed, or blank |
| Year | Capacity register: daily maximums for planned counters | CSV | Share path and cold-storage reference | Hash of the file | Date last opened and graphed, or blank |
| Quarter | Availability report as delivered to the provider dispute | PDF plus CSV | Share path and cold-storage reference | Hash of each file | Date last opened, or blank |
The checksum column is what makes the archive evidence rather than files. A checksum is a short fingerprint computed from a file's contents; change one byte and the fingerprint changes. Every operating system can compute one from the command line. Record it in the index when the file is written, and recompute it during the restore test. If the two match, the file is what it was.
Exporting a date range to CSV
Most products can export a graph's data or a report to CSV from the console, and for a monthly summary that is usually enough. For an incident snapshot, where you want every raw sample for a set of series across a window, use a query against the database. It is more reliable and more repeatable for this job. The SQL below is illustrative: the table and column names are whatever your product uses, and its documentation will tell you. Run it through the engine's command-line client with output sent to a file, or wrap it in a few lines of script.
-- Illustrative: table and column names differ by product. SELECT sample_time, series_id, value FROM metric_samples WHERE sample_time >= 'YYYY-MM-DD 15:00:00' AND sample_time < 'YYYY-MM-DD 21:00:00' AND series_id IN (1042, 1043, 1188, 2210) -- the affected devices and their neighbours ORDER BY series_id, sample_time;
If the database is SQLite, which is common for small installs, a short script can do the export without any client tools. In that case, the language's standard library can read the file directly. For MySQL/MariaDB or SQL Server, swap the connection line for that engine's connector or use its command-line client's export option. The shape is the same.
# export_window.py : illustrative export of a raw window to CSV (SQLite shown).
# Table and column names differ by product; check the documentation.
import csv, sqlite3, sys
db, start, end, out = sys.argv[1:5] # history.db "YYYY-MM-DD 15:00" "YYYY-MM-DD 21:00" out.csv
con = sqlite3.connect(db)
rows = con.execute(
"SELECT sample_time, series_id, value FROM metric_samples "
"WHERE sample_time >= ? AND sample_time < ? ORDER BY series_id, sample_time",
(start, end))
with open(out, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["sample_time_utc", "series_id", "value"])
w.writerows(rows)
print("wrote", out)
Whichever route you take, open the resulting file once before you file it. A CSV with a header and no rows is the most common export failure, and it looks exactly like success from the outside.
The incident snapshot procedure
The incident snapshot is the one archive item with a deadline. The raw tier is measured in weeks, and the purge does not know which week mattered. Take the snapshot on the day of the incident, or the day after at the latest, as part of the same routine that opens the post-incident review. The incident timeline reconstruction guide is what the snapshot is for; this is how to make sure it exists.
- Set the window. Start two hours before the first symptom and end one hour after full recovery. Wider is cheap; too narrow is permanent.
- List the series. The affected devices, and their neighbours: the uplink they sit behind, the firewall in the path, the servers the service depends on. When in doubt, include it.
- Export the raw samples for those series across the window to CSV, using the console's export or the query above.
- Export the status changes and alert history for the same window, for the whole estate. They are small and they are the skeleton of the timeline.
- Write the README. Say what happened in two sentences. Include the window, the series and why they were chosen. Record who exported the data, when, and from which system. Include the columns, the units, and the time zone.
- Compute the checksums and write them into the README and the index.
- Copy the files to the archive, in both locations, under the naming convention.
- Add a row to the archive index with the human description someone will search for.
- Open one file from the archive location and confirm it has rows. Then record the snapshot in the incident ticket.
Nine steps, twenty minutes, once per incident. The alternative is a discovery during a review, or a dispute. The only surviving record of the outage is a screenshot someone pasted into a chat message, cropped to hide the time axis.
The annual restore test
An archive that has never been opened is a hope. And the way to turn it into a fact is to open it, on a schedule, and write down that you did. Once a year is enough. The test is short and the point of it is not the data. The point is to discover problems in a calm week. Perhaps the share moved, the checksum tool changed, or the README is missing. Or the file was compressed with something nobody has any more.
- Pick one archived item at random from the index. Not the newest, and not the one you remember; the point is the one you do not.
- Retrieve it from the cold-storage copy, not the share, so that the slow path gets exercised too.
- Recompute the checksum and compare it with the index.
- Open the CSV in a spreadsheet or a script and make one graph from it: one series, across the window. Look at it. Does it show what the README says happened?
- Write the date, the item, the result and the time it took in the "restore-tested" column of the index.
- Fix whatever went wrong, then test a second item to prove the fix.
Tip: do the restore test with the newest member of the team, using only the retention policy and the index for directions. If they can find and open the file, the archive works. If they need you, the archive works only while you do.
A war story from Acme Foods
Acme Foods had, to its credit, archived three years of monthly monitoring exports, on a share, backed up, with a spreadsheet listing every file. The exports were in the monitoring product's own backup format, because that was the button the console offered. The certification auditor asked for evidence that the warehouse links had been monitored two years earlier. The team retrieved the correct file in under five minutes. Then they discovered that the product had been replaced. The old version's installer was no longer available. And the one server that had been able to open the file had been recycled the previous spring. The archive was complete, intact and unreadable, which the auditor described as "a nuance." Every export since then has been a CSV with a README.
Checklist
- You can say, for each archived item, which question it will answer in three years.
- Monthly summaries are exported before the hourly tier ages out.
- Incident snapshots are taken on the day, using the nine-step procedure, and recorded in the ticket.
- Every archived item is CSV with a header row, units in the column names, UTC timestamps, and a README.
- No item exists only in a proprietary or database-native format.
- The archive lives in two places, off the monitoring server, under the normal backup.
- Files are never edited in place, and checksums are recorded in the index.
- The naming convention sorts by date and names the type and scope.
- The index exists, carries human descriptions, and is findable without the archive share.
- The annual restore test was done, from cold storage, by someone who did not build the archive, and its date is in the index.
Where to go next
The archive is the last rung of the ladder, the one where "gone from the database" is allowed to mean "kept somewhere better." With it in place, the tiers can be as short as the disk wants, because nothing that matters depends on them alone. The whole set covers housekeeping that keeps the live database honest, plus the arithmetic and stakeholders behind the tiers. It is at Retention and Granularity: How Long to Keep What. The evidence packs guide shows how archived summaries are bundled when the auditor arrives. Export it, index it, open it once a year, and the finance outage will still be there when someone asks.
Put this into practice with SysPeep Monitor
Exporting a window of history is a short job when the data sits in an ordinary database. SysPeep Monitor keeps its history in SQLite for small installs, or MySQL/MariaDB or SQL Server for larger ones. It shows history, graphs and reports from one web console. For its export options and retention settings, check the product documentation. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep