Home › Topics › Trend Analysis and Degradation Creep › Alerting on Trends

Alerting on Trends, Not Just Thresholds


A threshold alert tells you a number is bad. A trend alert tells you a number is going to be bad, and roughly when. One email says "disk at 92 percent" at two in the morning. The other says "disk will be full in eighteen days" at nine on a Tuesday. Both are true. Only one of them arrives at a time when you can do something about it without waking anyone.


Trend alerts have a reputation for being complicated, and a smaller reputation for being noisy. The first is undeserved: there are only three kinds. Each fits in one sentence. And each can be built with a spreadsheet's worth of arithmetic. The second is earned. This guide spends as much time on the false alarms as on the rules. A trend alert that fires every month end is a trend alert that gets switched off.


By the end you will know the three rules. You will know how to implement each either in a tool that supports it or with a short scheduled script. You will also know what makes each one cry wolf, and how to quiet it. You will also have a table for writing down each rule so that the person who receives the alert knows what it means and what to do.


In this guide

  • Why a threshold alone is not enough
  • The three trend alerts, in plain words
  • Implementing them simply
  • A time-to-full script
  • The false alarms, and how to damp them
  • Which metrics deserve a trend alert
  • The trend alert spec template
  • The rule that was switched off
  • A trend alert checklist
  • Where to go next

Why a threshold alone is not enough


A threshold is a fixed line, and an alert fires when a metric crosses it. That is the right tool for anything with a hard limit and a sudden failure. Examples include a device that stops answering, a temperature above the vendor's maximum, or a scope with no addresses left. The threshold tuning pillar covers setting those lines well. And its guides on dynamic thresholds and percentile thresholds make the line move with the data. None of that changes what a threshold sees, which is a level.


The guide on what degradation creep looks like made the case in full: slow decline crosses no line until the day it crosses the last one. A threshold at 90 percent on a disk that grows one percent a week gives you a warning with a few weeks in hand, which is fine. The same threshold on a disk that has started growing five percent a week gives you a warning on a Friday with two days in hand, which is not. The threshold did not change. The slope did, and the threshold could not see it.


A trend alert is a rule that looks at the slope. It needs history, which a threshold does not. It produces a different kind of message: not "this is bad" but "this is heading somewhere bad by roughly this date". That message is worth building the rule for, because it is the only alert in the system that arrives with time attached.


The three trend alerts, in plain words


Rate of change: "rose more than X in Y days"

A rate-of-change alert compares the metric now with the metric Y days ago and fires if the difference is more than X. "CRC errors per day rose by more than 50 in 7 days." "Free memory fell by more than 10 percent in 14 days." It is the simplest of the three and the best for metrics that should be flat, where any sustained movement is news. It needs one number from the past and one from the present. The look-back, Y, is the whole design: too short and it fires on noise, too long and it fires late.


Time to full: "at this rate, full in less than N days"

A time-to-full alert draws a straight line through the last few weeks of the metric, extends it to the limit, and fires if the limit arrives within N days. "At the current growth rate, this volume will be full in under 30 days." It is the best rule for anything with a known ceiling: disk space, a DHCP scope, a PoE budget, free memory on a device (where the "full" is zero). It turns a slope into a date, and a date is something a manager can put in a calendar. It is also the only alert most people will ever thank you for.


Deviation from a moving average: "more than X above the 30-day average"

A moving average is the average of the last 30 days, recomputed every day so the window slides forward. A deviation alert fires when today's value is more than X percent above it. "Today's 95th-percentile response time is 40 percent above its 30-day average." It is a baseline alert with a window that moves by itself, and the guide on baselines in alerting covers the family in detail. It suits metrics that are noisy but stable, like response times and traffic, where the question is not "is it rising" but "is today unusual against recent normal". It is also the noisiest of the three, for reasons that get their own section.


Implementing them simply


Some monitoring tools support one or more of these rules natively, usually under a name like "trend", "forecast", "predictive", or "baseline deviation". If yours does, use it: the tool already has the history, the scheduler, and the notification path. Not every tool supports every rule, and the ones that do implement them differently. Check the product documentation for what yours calls the feature and which metrics it applies to. If the rule exists, configuring it is a matter of picking the metric, the look-back, and the limit. The rest of this guide is about picking them well.


If the rule does not exist in your tool, you do not need a new tool. You need three things you already have: an export of the metric's history, a scheduled task, and a way to send an email. The pattern is the same for all three rules. Once a day, a script reads the exported history, does the arithmetic, and if the rule is met, writes one line and exits with a non-zero code. The scheduler runs it, and a second step mails the output when the exit code is non-zero. Any command-line mail sender will do; the script does not care which.


The export is the only part that varies by tool. Most monitoring systems can write a metric's daily values to a CSV file on a schedule, or expose them through a report you can save as CSV. Failing that, a query against the monitoring database works, whether that is SQLite for a small install or MySQL, MariaDB, or SQL Server for a larger one. What you want is two columns, date and value, one row per day. Everything below assumes that shape.


A time-to-full script


This is the whole time-to-full rule in under thirty lines. It fits a straight line through the last thirty days by least squares. That is the standard way to draw the best straight line through a scatter of points. The script then extends the line to the limit and reports the days remaining.


# time_to_full.py <history.csv> <limit>  - days until a metric reaches its limit
# history.csv has one row per day: YYYY-MM-DD,value
import csv, sys
from datetime import date

LIMIT = float(sys.argv[2])            # 100 for percent used, or the size in GB
LOOKBACK = 30                         # days of history to fit the line through
WARN_DAYS = 30                        # alert when full is closer than this

rows = list(csv.reader(open(sys.argv[1], newline="")))[-LOOKBACK:]
xs = [date.fromisoformat(d).toordinal() for d, _ in rows]
ys = [float(v) for _, v in rows]
n = len(rows)
mx, my = sum(xs) / n, sum(ys) / n
slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)

if slope <= 0:
    print(f"not growing over the last {n} days")
    sys.exit(0)

days_left = (LIMIT - ys[-1]) / slope
print(f"growing {slope:.2f} per day; at {ys[-1]:.1f} of {LIMIT:.0f}; full in about {days_left:.0f} days")
sys.exit(1 if days_left < WARN_DAYS else 0)

Run it as python time_to_full.py fileserver_d.csv 100 for a percentage or with the volume size for gigabytes. The same skeleton gives you the other two rules by changing four lines. Rate of change is ys[-1] - ys[-8] against a limit. Deviation from a moving average is ys[-1] against sum(ys) / n times one plus your percentage. In a spreadsheet the fit is =SLOPE() over the last thirty rows and the days left is the gap to the limit divided by it. I built my first version of this in a spreadsheet with a macro that emailed me, and it was ugly and it worked for four years.


The false alarms, and how to damp them


Every trend rule has a characteristic false alarm, and each false alarm has a characteristic fix. The fixes all amount to the same thing: make the rule wait for more evidence before it speaks. A trend is a pattern over time, so a trend alert that fires on one day's data is not alerting on a trend. It is a threshold with extra steps.


Step changes

A step, a single jump to a new level, is the natural enemy of the rate-of-change rule. The metric rose by more than X in Y days, so the rule fires, correctly. Then it keeps firing every day until the step leaves the look-back window, because for Y days the value is still X above where it was. The fix is to fire once and then require the rise to continue. Alert when the change over the window is above X and the change over the last three days is also positive. A step fails the second test after day one. A real slope passes it every day.


Backups and batch jobs

Suppose a staging folder fills to 95 percent during the nightly backup and empties by six in the morning. That will convince a time-to-full rule with a short look-back that the disk has three hours left. The fix is a minimum look-back: never fit the line through fewer than thirty days. Fit it through the daily average or the daily minimum rather than the maximum. That way, a nightly spike is one point among thirty. The second fix is confirmation over several days: require the rule to be met on three consecutive daily runs before it sends anything. A real trend is still there on the third day. A backup is not.


Month end, term time, and the calendar

The deviation-from-average rule fires at month end, on the first day of term, and on the Monday after a long weekend. On those days the metric genuinely is far above its 30-day average, and the rule genuinely cannot tell why. One fix is to compare with the same day last week or the same period last month rather than the plain average. The guide on seasonal patterns explains this. Another fix is to run the rule during business hours only. That way, a quiet weekend does not drag the average down and make Monday look like an emergency. A rule that only evaluates at ten in the morning on weekdays misses nothing that matters and ignores most of what does not.


Tip: the first two weeks of any trend rule are a trial. Send its output to yourself only, read every message, and write down why each one fired. Most of the tuning you will ever do happens in those two weeks. And it is far easier to do it before anyone else has learned to ignore the rule.


Which metrics deserve a trend alert


Not every creeping metric deserves a rule. A trend alert costs attention every time it fires. And the guide on early-warning metrics makes the case that most slow-moving numbers are better served by a weekly look with a person attached. The test for promoting a metric from the weekly page to a rule is whether a week is too long to wait. If the metric can go from "fine" to "outage" inside seven days once it starts moving, it needs a rule. If it takes a month, the weekly look catches it with three weeks to spare.


By that test, the metrics that deserve a trend alert are the ones with a hard ceiling and a habit of accelerating. These include disk space, free memory on a device with a leak, DHCP scope usage, and error rates on uplinks. Those error rates can go from a dozen a day to a dead link in a fortnight. Give the metrics in that first list a time-to-full or rate-of-change rule. The metrics that deserve only the weekly look are the slow ones: temperature, fan speed, certificate counts, monitoring database size, alert volume. Nothing in that second list changes in a week by enough to matter, and a rule on any of them would fire on noise more often than on trouble.


Response times sit in the middle. A deviation-from-average rule on the 95th percentile of your most important service is worth having, with a confirmation window. A service that is 40 percent slower than normal for three days running is a problem someone should be told about. A deviation rule on every service is a rule nobody reads. Pick the two that matter.


The trend alert spec template


Every trend rule should be written down before it is switched on, in a table that anyone receiving the alert can read. The columns force the questions that make a rule useful. They cover what it watches, what it says, how far back it looks, and how many days it waits. They also cover who gets it and what that person is expected to do. The rows below are examples; replace them with your own.


MetricRuleLook-backConfirmationWho gets itWhat they do
Disk used, file server D:Time to full under 30 days30 days, daily average3 consecutive daily runsServer adminIdentify the growth, plan the extension, record the date in the capacity register
Free memory, main firewallTime to zero under 21 days30 days, daily minimum3 consecutive daily runsFirewall ownerSchedule a reboot in a maintenance window, open a vendor case for the leak
DHCP leases, office scopeTime to full under 45 days60 days, daily maximum2 consecutive daily runsNetwork adminExtend the scope or shorten lease time, check for a device flood
CRC errors per day, core uplinksRose more than 50 per day in 7 days, still rising over 3 days7 days, daily totalRise continues 3 daysNetwork adminInspect cable and transceiver, swap and re-check within a week
95th-percentile response, main applicationMore than 40 percent above the 30-day weekday average30 weekdays, business hours3 consecutive weekdaysApplication ownerCompare with change log, check load on the server and the path to it

The last column is the one people leave blank, and it is the one that matters. An alert that arrives without a next action is a fact, and facts do not fix disks. (They do, however, make excellent evidence at the review afterwards.) Write the action in plain words, and make sure the person in the "who" column has agreed to do it.


The rule that was switched off


Northgate Retail set up a time-to-full rule on its main file server with a seven-day look-back, because seven sounded responsive. Every night the backup software staged a copy on the same volume and deleted it before dawn. So every morning the rule saw a disk that had gained thirty percent overnight. It announced that the disk would be full by Thursday. After the fourth Thursday that did not happen, the rule was disabled, with a note to "revisit look-back", and the note went where notes go. Five months later a departmental share began growing at a steady two percent a week. The disabled rule said nothing, and the volume filled on a Saturday during the stocktake. The post-incident review re-enabled the rule with a thirty-day look-back on the daily minimum and three days of confirmation. That is what it should have had on the first day. It has not fired falsely since, which nobody at Northgate finds as reassuring as they expected.


The rule was not wrong. It was tuned wrong, and the difference between those two things is why the false-alarm section of this guide is longer than the rules section. A trend alert with the wrong look-back is worse than no alert, because it takes the good idea down with it when it is switched off.


A trend alert checklist


Run this for each rule before it goes live, and again the first time it fires for real.


  • Confirm the metric passes the test: could it go from fine to outage in under a week once it starts moving?
  • Pick the rule: time to full for anything with a ceiling, rate of change for anything that should be flat, deviation from average for anything noisy but stable.
  • Set the look-back to at least 30 days, on the daily average or minimum, never on the raw samples.
  • Require confirmation on at least three consecutive daily runs before anything is sent.
  • Evaluate during business hours on weekdays only, unless the metric is genuinely a round-the-clock risk.
  • Write the rule into the spec table with a named recipient and a plain-words action.
  • Run it to yourself alone for two weeks and record why each message fired.
  • Check that the export or query it depends on still runs, and put that check on the weekly-look page.
  • Review every rule quarterly: has it fired, was it right, and does anyone still act on it?

Where to go next


Three rules, one script skeleton, and a table that makes each rule answerable. Trend alerts cover the handful of metrics where a week is too long, and the weekly look covers everything else. The guide on the weekly trend review shows how to run that look in thirty minutes. It also explains where the trend alerts that fired last week fit into it. The guide on reading growth curves explains why a step and a slope need different responses even when the same rule fires on both. The rest of the Trend Analysis and Degradation Creep pillar is here.


Put this into practice with SysPeep Monitor

A trend rule needs the metric's history kept for months and an alert path that already reaches the right person. A central Windows service gives you the history and the delivery. It collects over SNMP, WMI and SSH and keeps the long-term graphs. It sends email and other notifications from thresholds and alert rules configured in one web console. Check the product documentation for which trend rules it supports natively. The free trial installs in minutes.

Download Free TrialLearn About SysPeep