Home › Topics › Migrating Between Monitoring Platforms Without Losing Your History › Data Export Realities

Data Export Realities: What History You Can Keep


Somewhere near the start of every migration, someone asks: "We'll keep all the history, right?" It is a reasonable question. The old system holds years of graphs — interface traffic, disk growth, CPU trends, availability — and those graphs are not decoration. They are how you know the internet link is getting busier. They are how you predicted the file server would fill in the spring and proved to the provider that the outage was theirs. Losing them feels like losing evidence, because it is. I have been asked the question in the first meeting of every migration I have run. Usually, nobody has yet asked what the new tool is called.


The honest answer is: probably not all of it, and almost certainly not as live graphs. Raw monitoring history rarely migrates cleanly between platforms, in any direction. Teams that promise otherwise tend to find out at the point where the promise is hardest to take back. That is the week after the old server has been wiped. But "the raw history will not move" is not the same as "the history is lost." A great deal can be kept, in forms that answer the questions people actually ask. That requires planning the export before the old system is switched off rather than mourning it afterwards.


This guide explains why raw history resists migration and what can be moved and how. It explains what to do instead of migrating: keep the old system readable, export reports and baselines, and back it up restorably. It also explains how to set expectations with the people who will ask for a graph six months from now.


In this guide

  • The honest truth about raw history
  • Why it does not move: schemas, rollups, identifiers
  • What usually can move
  • What moves, how, and what does not
  • Instead of migrating history: the four-part plan
  • An example: exporting a capacity baseline
  • Setting expectations with management
  • The export checklist
  • Where to go next

The honest truth about raw history


What a monitoring system stores is, at its simplest, a very long list of timestamped numbers. Each entry says: at this moment, this check on this device had this value. Multiply that by every check, every interval, every day, and it is a great many numbers. So every platform stores them in its own carefully optimized way. That optimization is the problem. There is no common format for monitoring history that two platforms both read and write — no equivalent of the comma-separated file that every spreadsheet understands. Some tools can export raw data. Fewer can import it. Almost none can import another tool's export in a way that produces the graph the other tool showed.


This is not a shortcoming of either tool. It is the consequence of two tools each making sensible storage decisions that were never intended to match. A migration from a commercial platform to an open-source one meets exactly the same wall as the reverse. So does a migration between two products of the same kind.


Northgate Retail exported "everything" the evening before the old monitoring server was rebuilt. The export was a single file, in the old tool's own format, and the handover notes described it as five years of history. It was five years of hourly averages, because the rollup had done its work long before. The peaks — the only column the case for a bigger internet link had ever rested on — were not in it. The file could be opened by exactly one program, which had been on the server that was rebuilt the following morning. The link upgrade was argued from memory and approved on the strength of a screenshot somebody had emailed to a colleague the previous winter. The screenshot is now in the project folder, labelled "history".


Why it does not move: schemas, rollups, identifiers


Different schemas

A schema is the structure a database uses: what the tables are and what each column means. One tool stores a check as a row keyed by a numeric ID with the device's ID alongside. Another stores it under a path-like name; another as a set of labels. Converting between them is a translation project, and one that has to be done per check type. That is because a counter-based interface metric and a gauge-based disk metric are stored differently even inside the same tool. The background on how monitoring databases are organized is in the database backend choices guide.


Rollups

Almost no system keeps every raw reading forever. After a while, readings are consolidated: a week of one-minute samples becomes five-minute averages, then hourly, then daily. This is a rollup, and the details differ in every platform. Those details include when it happens, which functions are kept (average only, or also the peak?), and how long each level is retained. The rollups and retention guide explains the mechanism. The practical consequence for migration is that even if you could move the data, anything older than a few weeks would already be a summary. Summaries made by one tool's rules do not fit another tool's storage levels.


Identifiers

Even where the numbers could be moved, they need to be attached to the right thing. The old system knew the core switch as device 47 and its uplink as interface index 25. The new system discovered the same switch as a fresh device with its own ID, and may index the interface differently. Consider renamed devices, re-addressed devices, and interfaces whose index changed after a firmware update. Each is a place where imported history would be attached to the wrong object, or to nothing. Getting the mapping right for thousands of checks is possible, but it is a project on the scale of the migration itself.


Put the three together and the realistic verdict is this. For a handful of high-value metrics, with a skilled person and both tools' cooperation, some raw history can sometimes be brought across. For the general case, plan as if it will not, and be pleasantly surprised.


What usually can move


Plenty, as it turns out. The trick is that the things that move are not "history" in the raw sense; they are the things the history was for.


  • Device inventories. Names, addresses, groups, locations, custom fields. Every tool exports these, and the inventory phase already extracted them.
  • Configuration, as a description. Thresholds, intervals, alert rules, dependencies, contacts. Rarely importable as a file, but reproducible from the migration spreadsheet.
  • Reports as documents. Every scheduled report the old system produced can be produced one more time for each period you care about and saved as a document. A year of monthly availability reports is a year of availability history, in a form any manager can open forever.
  • Exported summaries. Daily average and daily peak per interface, per disk, per CPU, as comma-separated text. This is the capacity story in a form a spreadsheet can graph. And it is small: a year of daily values for a thousand checks is a modest file.
  • Graphs as images. For the dozen metrics that matter most, save the one-year graph as an image with the axis labels visible. Those metrics cover the internet link, the core uplinks, and the main storage. The saved graph is the exact picture people remember, and it will be shown in the budget meeting.
  • Alert history as a list. The last year's alerts, exported, are both a parity-testing input and a record of what went wrong and when.

What moves, how, and what does not


ItemMoves?HowWhat does not come with it
Device listYesExport, then re-import or rebuildThe old internal IDs
Check definitions and thresholdsMostlyMigration spreadsheet; rebuilt by hand or scriptedIdentical behavior; confirmation logic differs between tools
Alert rules and contactsMostlySpreadsheet; rebuiltDelivery quirks of the old integration
DependenciesPartlyRebuilt from the inventory and the network mapRelationships the old tool generated automatically
DashboardsNot as filesRebuilt to match what people actually openLayout and widgets
Scheduled reportsAs documentsRun once per past period and saveInteractive drill-down
Raw time seriesRarelyOnly for a few high-value metrics, with effortEverything else
Rolled-up historyNoExport daily averages and peaks as textSub-daily detail older than the raw window
GraphsAs imagesSave one per key metricZoom and hover values
Alert historyYesExport as textLinks back into the old console
Availability figuresAs numbersExport per device per monthRecalculation inside the new tool
The old database itselfAs a backupFull database and configuration backup, test-restoredUsability without the old software installed

The "rarely" in the raw time-series row deserves a word, because sometimes it is worth attempting for a very short list. If the new tool offers any way to import historical values, pick the five or ten metrics people argue about most. Some tools do offer this, usually through a scripting interface rather than a button. For that import, export the metrics from the old system at the finest granularity it still holds. That means the shortest sample interval that has not yet been rolled up. Map each one to its new identifier by device name and interface description rather than by index, and import. Then put the new tool's graph beside the saved image of the old one and check that the shape matches. If it does, you have kept a few live graphs. If it does not, or if the effort is running past a day or two, stop. The exports below already answer the questions those graphs were for. And a wrong graph is worse than a saved picture of a right one.


Instead of migrating history: the four-part plan


1. Keep the old system read-only for the retention period

The simplest way to keep history is to not destroy it. After cutover (the day the new system becomes the one people act on), the old system is silenced. Its polling is optionally reduced or stopped, and it is left running as a read-only reference. Anyone with a question about last year opens the old console and asks. How long depends on your retention needs — how far back questions typically reach, often about a year. It also depends on the old system's terms of use. Those range from "keep running indefinitely" to "must be decommissioned at renewal." Read the agreement, or the license text if it is open source, and plan the read-only period around it. Moving the old system onto a smaller virtual machine with a snapshot is often enough to keep it alive cheaply.


Keep it patched, or at least off the internet, while it lives on. A monitoring server that nobody maintains but everyone can reach is a security finding waiting to happen.


2. Export monthly reports for the last year before shutdown

Before the old system loses any ability to generate reports, generate all of them. Generate each scheduled report for each month of the last year. Save each as a document in a folder that is backed up and that people can find. Name them consistently — report type, then year and month — so a manager can find "the March availability report" without help. Include the one-off reports too, for the metrics people ask about most.


3. Export capacity baselines into a spreadsheet

For every interface, disk, and CPU that anyone has ever asked a capacity question about, export daily average and daily peak for the last year. Put those into a workbook with one tab per category. This is the material the capacity conversation will use until the new system has a year of its own. It is small, it is durable, and a spreadsheet graph of it looks close enough to the old console's graph that nobody minds.


4. Back up the database and configuration

Take a full backup of the old system's database and configuration files. Verify it by restoring it somewhere (a throwaway virtual machine is fine), and store it with the exports. The monitoring backups guide covers what a complete backup of a monitoring system includes and how to test it. The point of this backup is not to migrate from it; it is insurance. If a question comes up two years from now that only the raw history can answer, and the old software can still be installed, the backup makes an answer possible. A backup of a database that only the old software can open is, without the old software, a very safe paperweight.


An example: exporting a capacity baseline


Here is the shape of the baseline file for interfaces. Most tools can produce it as a report with daily aggregation. Many also expose a query interface that a short script can loop over, one interface at a time, asking for daily averages and peaks. The month-day column stands in for a full date.


# interfaces-daily-baseline.csv (excerpt)
device,interface,day,in_avg_mbps,in_peak_mbps,out_avg_mbps,out_peak_mbps
fw-edge-01,wan1,03-01,142.3,388.9,61.7,210.4
fw-edge-01,wan1,03-02,151.0,402.6,64.2,231.8
fw-edge-01,wan1,03-03,98.4,301.2,40.9,166.0
sw-core-01,Te1/0/1,03-01,2210.5,6980.2,1845.1,6402.7
sw-core-01,Te1/0/1,03-02,2305.8,7120.9,2140.6,6598.4
sw-core-01,Te1/0/1,03-03,1490.2,5211.0,1302.7,4877.5

Two columns per direction, on purpose. The average tells you the trend; the peak tells you when the link was actually full, which is what the upgrade case rests on. A year of this for one interface is a few hundred rows. A spreadsheet chart of the peak column reproduces the argument the old graph used to make. Do the same for disks (used and free, daily) and CPUs (average and peak). Add a tab of monthly availability per device from the old system's availability report. That workbook, plus the saved images of the dozen most-discussed graphs, is the capacity story in portable form.


Tip: Ask each stakeholder which single graph they would be upset to lose. Export that one as an image and as text, and send it to them before shutdown. It takes an afternoon and removes most of the anxiety in the room.


Setting expectations with management


Say it early and plainly, in a sentence they can repeat: after cutover, the capacity conversation will use exported summaries and saved images rather than live graphs until the new system has enough history of its own; for anything further back, the old system stays readable until a named date; after that, the exports and the backup are the record. Explain that this is the normal outcome of any platform change, in any direction, and not a defect of the tool that was chosen. Then show them the exported workbook, so it is a thing they have seen rather than a promise they have heard.


Two questions to answer before they are asked. "How long until the new system's graphs are as useful as the old ones?" The answer is as long as the questions reach back — a year, for annual trends. That is another reason to start the parallel run early. In that run, both systems watch the same devices while only the old one is trusted. Starting early matters because the new system begins accumulating its own history the day it is pointed at the network. And "What if we need something we did not export?" The read-only old system until its date; the backup after that. Managers cope well with honest limits stated in advance and badly with surprises in the budget meeting.


The export checklist


Run this during the sunset phase — the read-only period between cutover and the old system's shutdown. Do it before the shutdown date, and have a second person tick each item independently.


  • 1. Confirm what the old system's terms allow for a read-only period, and set the sunset date accordingly.
  • 2. List every scheduled report; run and save each one for each month of the last year, consistently named.
  • 3. List the key metrics by asking stakeholders; save the one-year graph of each as an image with axes visible.
  • 4. Export daily average and peak for every interface, disk, and CPU of interest into the baseline workbook.
  • 5. Export monthly availability per device.
  • 6. Export the last year's alert history.
  • 7. Refresh the device inventory export from the inventory phase, so it reflects the final state.
  • 8. Take a full database and configuration backup; restore it somewhere to prove it works; store it with the exports.
  • 9. Have a second person open each export and find one specific value in it, then sign the list.
  • 10. Put everything in a backed-up, findable location and write a one-page index of what is there and how to read it.
  • 11. Tell management and stakeholders what exists and where, in writing.
  • 12. Reduce the old system to read-only, put the shutdown on the calendar, and on the date, shut it down and keep the backup.

Item nine is the one people skip and the one that matters. An export nobody has opened is an assumption, and the whole point of the sunset phase is to leave behind facts.


Where to go next


Raw monitoring history rarely survives a change of platform, because of schemas, rollups, and identifiers, and that is true whichever direction you are moving. What survives is everything the history was for. That includes inventories, configuration, reports as documents, daily baselines as text, graphs as images, and alert history as a list. It also includes the old system itself, kept read-only for as long as its terms allow and backed up restorably after that. Plan the exports before shutdown and have a second person check them. Tell management in advance that the capacity conversation will run on exported summaries for a while. Then, when someone asks "we'll keep all the history, right?", the honest answer is "we'll keep everything it was for." That is a better answer than yes, and unlike yes, it survives a rebuild.


This is the last phase of the plan described in Why Monitoring Migrations Fail. And the reports and dashboards you are exporting here are the ones listed during the inventory. If the migration is one step in a larger tidy-up of overlapping tools, the Monitoring Tool Sprawl pillar covers the rest of that journey. The Uptime and Availability Monitoring pillar explains what the availability figures you are exporting actually measure. The whole topic is collected at Migrating Between Monitoring Platforms Without Losing Your History.


Put this into practice with SysPeep Monitor

History does not move, but a new system starts building its own the day it is installed. That is one argument for starting the parallel run early. If SysPeep Monitor is a candidate on your list, the free trial stores its history in SQLite, MySQL/MariaDB, or SQL Server. Its reports and dashboards come from a single web console. So the same export checklist you are applying to the old system can be applied to it in turn.

Download Free TrialLearn About SysPeep