Home › Topics › Syslog & Log Collection › Log Alerting

Alerting on Logs: The Patterns That Deserve a Page


Once you have a central syslog server collecting messages from your network, an uncomfortable truth arrives with the flood. Almost none of it matters right now. A modest network produces tens of thousands of log lines a day. And the vast majority answer questions later — what happened before the crash, who changed what. Logs are an archive first. Treating every line as urgent is how admins learn to ignore all of them.


But buried in that stream is a small set of patterns that mean something is happening right now. The skill of log alerting is drawing that line deliberately. Name the handful that earn an interruption. Wire up the machinery that watches for them. Tune until every alert that fires is one you are glad fired.


This guide is the starter ruleset: eight alert-worthy patterns. Each comes with the reason it matters, an example log line, and tuning advice. The guide also covers the mechanics of log alerting and the threshold-and-window logic that keeps it from burying you. It covers the discipline that keeps the ruleset honest.


In this guide

  • Archive vs. alarm: the bar a log line must clear
  • The alert-worthy catalog: eight patterns
  • How log alerting actually works
  • Threshold, window, quiet period: the universal grammar
  • Fire drills: test your alerts on purpose
  • Noise discipline and response tiers
  • The starter ruleset table
  • Where to go next

Archive vs. alarm: the bar a log line must clear


First, a definition: a page is a notification designed to interrupt a person — a phone buzz, a loud email, whatever breaks into your attention. The name survives from the pager era; the idea survives because interruption is the most expensive thing monitoring can spend, and false pages cost trust. An alert rule is a standing claim that a pattern is worth that price. The bar is simple: if this pattern fired at 3 AM, would a competent admin be glad they were woken? Everything below that bar is still valuable — it becomes a ticket, a report, or archive.


The tempting shortcut is to alert on severity. Syslog messages carry a severity level from 0 (emergency) to 7 (debug), so why not page on everything level 3 and worse? Because vendors assign severities inconsistently and often theatrically. One device calls a routine interface transition a level-3 error while another logs a genuine hardware fault at level 5. The syslog fundamentals guide tours this zoo. Severity is a hint for filtering and triage, not a decision. Real log alerting is pattern-based: you decide which events matter and match them by content. The catalog that follows clears the 3 AM bar for most shops — or comes close enough to deserve a deliberate decision.


The alert-worthy catalog: eight patterns


For each pattern: why it matters, an example log line (your vendors' wording will differ — match on the stable parts), and tuning advice.


1. Authentication failure bursts

One failed login is Tuesday — someone fat-fingered a password. Ten failures in a minute from one source is an event. It means a person or script guessing passwords (a security problem). Or an automated process is hammering away with a broken credential (an outage in the making).


Oct  4 03:11:52 vpn-gw sshd[19218]: Failed password for invalid user admin from 203.0.113.77 port 51694 ssh2

Tuning: the rule you want is a threshold within a window. For example, "10 or more failures from the same source address within 60 seconds." It is a grammar we will return to. Count per source, or one noisy scanner will mask everything else. And add a second, looser rule for distributed attempts: say 50 failures in 10 minutes from any mix of sources against one account. Expect your first real catches to be boring — a service account with an expired password, a monitoring tool with stale credentials. Still wins: broken automation left alone becomes a mystery outage later.


2. Any config change outside a change window

Here is the honest-team rule: legitimate changes announce themselves. A configuration change that nobody announced is either an intruder, a colleague who forgot to mention it, or you-from-last-week — and all three deserve a ping. The first is an incident; the other two are how undocumented networks are born.


Oct  4 14:02:31 core-sw1 812: %SYS-5-CONFIG_I: Configured from console by jsmith on vty0 (10.0.8.14)

Tuning: define your change windows — even informally ("weekday evenings after 6 PM") — and alert only on changes outside them. Inside a window, log and move on. Note the severity in the example: level 5, "notice." A severity-based rule would never have paged on one of the most alert-worthy events a switch can emit. That is the whole argument for pattern-based alerting in one log line.


3. Infrastructure link flaps

A flap is a link going down and coming back up, over and over. On access ports this is life — laptops sleep, phones undock — which is why the what-to-collect guide keeps access-port link messages out of your stream. But consider infrastructure links — switch uplinks, the WAN handoff, anything between two pieces of network gear. There, a flap means a dying cable, a failing transceiver, a duplex mismatch, or a shaky provider circuit.


Oct  4 09:40:07 core-sw1 845: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to down

Tuning: the key word is dampening. Do not page on each transition, because a flapping link can transition hundreds of times an hour and your phone becomes the flap. Alert on the third down event on the same interface within an hour. A link that stays down is your reachability monitoring's job. The flap rule exists for the marginal link that is technically "mostly up."


4. HA failovers and redundancy events

HA (high availability) means two devices — firewalls, most commonly — sharing one job so that either can carry it alone. When the secondary takes over, users notice nothing, which is the point. It is also the trap: the system saved you, and if nothing tells you, you will never go find out from what. A failover means the primary had a problem serious enough to fail out — and that you are now running without a net.


Oct  4 02:17:44 fw-b HA: (Secondary) Switching to ACTIVE - no response from mate

Tuning: threshold of one — these are rare and always meaningful, so every event fires. Do add a quiet period (below): a sick HA pair can flap ownership back and forth. You want one page saying "the pair is unstable," not forty. The same logic covers other redundancy events: a RAID array running on its spare, a power supply dropping to one, a routing protocol failing over to a backup path.


5. Device reboots you didn't schedule

An unscheduled reboot means a crash, a power blip, a watchdog reset, or a person rebooting things without telling anyone. Any of those is worth a conversation — and a reboot storm, many devices restarting within a minute or two, is your power infrastructure talking.


Oct  4 05:58:03 branch-rtr1 %SYS-5-RESTART: System restarted

Tuning: threshold of one, suppressed during announced maintenance windows so patch nights do not page you eight times. Match your vendors' restart and cold-start wording. And remember a rebooting device wasn't sending logs during the outage, so pair this rule with reachability monitoring.


6. Site-to-site VPN tunnel down

A site-to-site VPN tunnel is the encrypted connection joining two locations into one network. When it drops, the branch office is an island: no file server, no internal apps, no phones if they ride the tunnel. The branch experiences "everything is down" while every device is powered and healthy — a failure that hides from device-health monitoring and announces itself plainly in the firewall's logs.


Oct  4 11:23:55 fw-a IPsec: Tunnel to peer 198.51.100.12 (branch-office) DOWN, reason: dead peer detected

Tuning: alert on down events that are not followed by an up event within a couple of minutes. Tunnels renegotiate routinely, and a two-second blip during rekey is not a page. If one branch rides a shaky circuit, give its tunnel the flap treatment from pattern 3 — and open a provider ticket with the log evidence.


7. Storage and hardware warnings from hosts and NAS

Servers and NAS units (network-attached storage — the shared disk box) are unusually polite about dying. Drives report SMART warnings (the drive's built-in self-monitoring) before failing. RAID arrays announce degradation when a member drops. Fans and power supplies log their own decline. The gap between "disk 3 has failed, array degraded" and "second disk failed, data gone" is the most valuable head start in IT. And it arrives as a log line.


Oct  4 22:41:18 nas1 RAID: Disk 3 removed from volume1, volume is now DEGRADED

Tuning: split by seriousness. Array degraded, power supply failed, temperature critical: threshold of one, page. Early wear indicators like a few pending sectors: raise a ticket, not a page. Deduplicate to one alert per device per day, because SMART daemons love to repeat themselves.


8. Admin login from an unfamiliar place

A successful administrative login to network gear from an address you do not recognize may be a colleague working from somewhere new. Or it is the day you are very glad you had the rule. Honesty first: this pattern is false-positive-prone — new laptops, DHCP reshuffles, a VPN pool you forgot about. Most hits will be innocent, which is why for most shops it belongs in the review tier, not the 3 AM tier.


Oct  4 08:02:19 core-sw1 %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: jsmith] [Source: 198.51.100.201] [localport: 22]

Tuning: make it tractable by defining a known-management list — the subnets and jump hosts admin logins normally come from. Alert only on successes from outside it. Better still, restrict management access to those sources at the device and firewall level — then this rule becomes a tripwire that should almost never fire.


How log alerting actually works


A pattern list is only half the machinery: something has to watch the stream, match lines, count them, and notify. Four shapes cover almost every real-world setup, from duct tape to platform.


A script tailing a file

The simplest thing that works: a script watching the collector's log file. The naive version fits on two lines:


# Naive tail-and-match: pages on EVERY match (no counting yet)
tail -F /var/log/remote/network.log \
  | grep --line-buffered "Failed password" \
  | while read line; do send-alert "$line"; done

The charm: twenty minutes to build, no new software. The flaw is in the comment: it pages on every single match — point it at an auth-failure burst and it sends six hundred emails. To be usable it needs counting, windows, and suppression bolted on, and by then you have reinvented a small alerting engine. A fine way to understand log alerting; a bad way to run it long-term.


The syslog daemon itself

Syslog daemons like rsyslog and syslog-ng are not just receivers. Their filter engines can match messages by content or source and route them to a file, a database — or a program. Conceptually:


# rsyslog: route config-change messages to their own file
:msg, contains, "CONFIG_I" /var/log/network/config-changes.log
# rsyslog can also hand matches to a program that counts and notifies

This is the classic budget setup: the daemon does the matching, and a small helper script does the counting and notifying. It scales further than pure tail-and-grep and adds no new software beyond what your central syslog server already runs.


Saved-search alerts in a log platform

If your logs land in a search platform, alerting usually comes built in as saved-search alerts. With these alerts, you write the query ("failed password, grouped by source address"). The platform runs it on a schedule, and it fires when the count crosses your threshold. This is threshold-and-window logic wearing a friendly interface. It gets better if you have parsed your logs into fields. Then "count by source address" becomes a real query instead of a regex stunt.


Your monitoring platform

Many network monitoring platforms can ingest log events (and SNMP traps) alongside the metrics they already collect. The appeal is unification: log alerts flow through the same notification rules, escalation chains, and quiet hours as your ping and disk-space alerts. That gives you one place to answer "what pages me, and when?" If terms like threshold, escalation, and alert fatigue are still settling in, the monitoring glossary pins them down. Whichever shape you choose, insist on three capabilities: match by content, count within a window, suppress repeats.


Threshold, window, quiet period: the universal grammar


Strip away the tools and every good log alert rule says the same sentence: "When N matching events occur within M minutes, notify — then stay quiet about further matches for Q minutes." Three numbers. Think in this grammar and you can express your ruleset in any tool you will ever meet.


The threshold (N) encodes how meaningful a single event is. Rare-and-serious patterns — failover, reboot, RAID degraded — get N=1. Patterns meaningful only in bulk — auth failures, flaps — get higher N: the event becomes signal when it clusters. The window (M) gives the threshold its meaning: ten failures in one minute is an attack; ten in a month is a user. The quiet period (Q) is the anti-storm device: after a rule fires, further matches are still counted and logged but do not re-page until Q passes. Without it, one broken device can emit the same message ten thousand times overnight, each one dutifully becoming a notification — a car alarm that never stops.


Fire drills: test your alerts on purpose


An untested log alert is decorative. Between you and the page stands a chain of assumptions. The device sends the message. The collector receives it. Your pattern matches the vendor's exact wording. The threshold logic works. The notification reaches a human. Any link can break silently — a firmware update rewords a message, an email rule files alerts under "later." The only way to know the chain holds is to pull it.


So trigger a harmless instance of each and watch it page:


  • Auth burst: fail SSH login against a test target eleven times in a minute — wrong password, real source.
  • Config change: enter and exit configuration mode on a switch outside your change window (a no-op change counts).
  • Link flap: bounce a designated test port three times within the hour — a lab port, not the uplink.
  • Failover: fail your HA pair over during a maintenance window — doubly worthwhile, because a failover you have never rehearsed is a hope, not a design.
  • Reboot: restart a lab device, unannounced as far as the ruleset knows.

For each drill, verify the whole path: the line arrived at the collector, the rule matched, the count crossed the threshold, and the notification landed on a phone. Fix what broke; re-drill. Patterns you cannot safely trigger — nobody pulls a disk to test the RAID rule — get a paper drill. Capture a real example of the message from vendor docs or your archive. Replay it at the collector or temporarily lower the threshold, and confirm the plumbing.


Tip: put the fire drill on the calendar — quarterly is plenty. Add one test of the pipeline itself: stop the collector service for a minute. Did anything tell you? If not, add a heartbeat — a scheduled log message that alerts when it fails to arrive. Silence should never be ambiguous.


Noise discipline and response tiers


Every log alert must earn its keep forever, not just the day you wrote it. The mechanism is a short weekly review of what fired: for each alert, was it acted on or ignored? That ratio is the health metric of the ruleset. An alert that is consistently ignored is worse than useless — it trains you to ignore the next one. It gets tuned, narrowed, or demoted the same week. No guilt; a demotion is the system working.


Not every worthwhile pattern deserves an interruption; mature rulesets use graduated response tiers:


  • Page: interrupt a human now. Reserved for patterns where acting within minutes changes the outcome — failover, unscheduled reboot, tunnel down, array degraded.
  • Ticket: create a work item for the next business day. Early hardware warnings, off-hours config changes where evening tinkering is normal, first-offense auth bursts from internal sources.
  • Review: no notification at all; the pattern is flagged into a daily or weekly digest. Login-from-new-place hits, anything on probation in either direction.

Patterns move between tiers as evidence accumulates — that movement is the tuning. The table below marks a sensible starting tier for each pattern; treat every one as provisional until your own network has voted.


The starter ruleset table


PatternExample matchThreshold logicStarting tier
Auth failure burstFailed password10 in 60 s per source; quiet 15 minPage
Distributed auth failuresFailed password50 in 10 min, any sourceTicket
Config change, outside windowCONFIG_I1 match outside change windowPage
Infrastructure link flapLINK-3-UPDOWN on uplink/WAN list3rd down on same interface in 60 minPage
HA failover / redundancy eventSwitching to ACTIVE1 match; quiet 30 minPage
Unscheduled rebootSYS-5-RESTART / cold start1 match outside maintenance windowPage
Site-to-site VPN downtunnel down messageDown without up within 2 minPage
Array degraded / PSU / temp criticalDEGRADED, PSU fail1 match; dedup per device per dayPage
Drive early warnings (SMART)pending sectors1 match; dedup per device per dayTicket
Admin login, unfamiliar sourcelogin success outside known list1 matchReview

Ten rules — a deliberate number: small enough to test, tune, and trust, large enough to cover the events that change your week. Resist adding an eleventh until the first ten have each fired usefully or survived a fire drill.


Where to go next


You now have the whole shape of log alerting. Logs are archive by default. A short catalog earns interruption. Every rule speaks the threshold-window-quiet grammar, and fire drills plus weekly review keep it honest. Build the ten rules, drill them once, and you have something most networks never get: a log stream that taps you on the shoulder only when it should.


Alerting sits at the top of a stack: collecting the right events, parsing them into searchable fields, keeping clocks sane so patterns line up across devices. The rest of the Syslog & Log Collection guides cover each layer.


Put this into practice with SysPeep Monitor

Log alerts cover the story your devices tell; you still need eyes on the vitals they don't narrate. SysPeep Monitor watches device health, port status, disk space, and reachability over SNMP and WMI. Its thresholds and dependency-aware alert rules are built to keep the noise down — the same discipline this guide teaches. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep