Home › Topics › Network Monitoring as a Security Layer › Security Signals in Ops Data

Security Signals Hiding in Your Ops Monitoring


Most of the alerts a small network produces are boring, and that is exactly the problem. When packets per second spike on a switch port, you assume a loop. When a burst of failed logons shows up, you assume someone changed a password. When the backup job fails, you assume the usual thing that makes backup jobs fail. You are right most of the time.


Being right most of the time is how the one alert that was not boring slides past. I know the mechanism from the inside. For years I cleared the packets-per-second alert on port 14 with a note that said "loop again". The note was correct for a long time. That is the most dangerous thing a note can be. The habit is not carelessness. It is experience, pointed in only one direction.


This guide is a catalog. It walks through the ordinary signals your monitoring already produces. These include interface counters, DHCP and DNS logs, flows, authentication events, inventory changes, and the various kinds of silence. For each one, it gives you the boring explanation, the worrying explanation, and the single first check that tells them apart. It finishes with a reference table and a practical section on wiring these signals into thresholds you already have, without buying a new tool. By the end you will look at your existing dashboard and see a second set of things it has been trying to tell you.


In this guide

  • How to read this catalog
  • Interface, switch, and DHCP signals
  • DNS, flow, and traffic signals
  • Authentication, inventory, and change signals
  • Silence as a signal
  • The signal table
  • Wiring signals into thresholds you already have
  • Where to go next

How to read this catalog


Every entry follows the same shape, because the shape is the skill. The boring explanation is the operational cause you would normally assume, and it is usually right. The worrying explanation is what the same signal looks like when something hostile is behind it. The first check is the one quick look that separates the two. The rule is "boring until proven otherwise, but actually do the check." Assuming boring without looking is how you get surprised; assuming worrying without looking is how you burn out.


Almost none of these signals is conclusive alone. Hostile activity tends to trip two or three of them in the same hour on the same host. That coincidence is far stronger than any single entry. When a first check does not produce a boring answer, the next move is to look at what else happened to that host in the same window.


Interface, switch, and DHCP signals


Broadcast or packets-per-second spike on a port

Boring: a loop, a failing NIC (network interface card) chattering, or a device with a broken network stack. Broadcast packets climb along with total packets, and usually several ports on the segment show it at once. Worrying: a host scanning the network. Packets per second jump while bytes per second barely move, the broadcast counter stays flat, and it is one port, not the segment. First check: compare packets to bytes, and check whether broadcast rose with them.


A port that should be empty comes up

Boring: facilities plugged in a printer, or someone moved desks. Worrying: a device was plugged into a port in an unlocked area by someone who should not have been there. The area might be a lobby, a meeting room, or a hallway jack. First check: which port, what MAC address, what time of day. A device that appeared at 6:40 PM on the lobby jack is a different question from one that appeared at 9:15 AM in accounts.


A MAC address moves between ports

Boring: a laptop docked in a new spot, or a spanning-tree change (the switches re-deciding which links carry traffic) shifting the path. Worrying: a device impersonating another device's hardware address, which is one way to slip past port-based access controls. First check: is the original device still online and answering at its old location? If both appear to exist at once, that is not a laptop that moved.


Lease churn: a burst of new leases

Boring: a visitor event, a batch of new phones, or a device rebooting in a loop and asking for an address each time. Worrying: a device requesting many addresses under many hardware addresses to exhaust the pool, so that clients accept an offer from somewhere else. First check: how many distinct MAC addresses requested leases, and do their vendor prefixes look like real hardware or random noise.


A second DHCP server appears

Boring: a home router someone plugged in to get more Wi-Fi, with its DHCP still switched on. Worrying: a device deliberately handing out its own gateway and DNS so it can sit between clients and the world. The client-side symptom is identical either way: machines end up with the wrong gateway. First check: on an affected client, which server issued the lease; on the switch, which port that server's MAC lives on. Either way the device comes off the network, and the difference between the two explanations is whether anyone claims it.


DNS, flow, and traffic signals


Query volume spike from one host

Boring: a misconfigured application retrying a lookup in a tight loop, or a monitoring probe with an aggressive interval. Worrying: data being carried out inside DNS queries. This shows up as a steady, high volume of queries for long, random-looking subdomains of one unfamiliar domain. These queries often use record types ordinary software rarely asks for. First check: list the domains that host queried most. A retry loop hits one legitimate name over and over; tunneling hits thousands of unique names under one odd parent.


Failure storm (NXDOMAIN burst)

Boring: a decommissioned server whose name is still baked into a hundred config files, or a search-suffix problem that makes every lookup fail once before succeeding. The server was retired long ago; nobody told the config files. Worrying: one host burning through gibberish names looking for a controller that has not registered yet. First check: whether the failing names are readable (old hostnames, typos) or noise, and whether the failures come from many clients or one.


Clients using an outside resolver

Boring: a laptop with a hard-coded public DNS server from home, or a device with its own built-in resolver settings. Worrying: software deliberately bypassing your DNS so its lookups never appear in your logs. This one is technically a flow signal, since it shows up as DNS-port traffic leaving the network from a client rather than from your resolver. First check: which hosts are sending DNS traffic directly outward, and whether the destination is a well-known public resolver or something nobody recognizes.


An unexpected top talker

Boring: a large download, a cloud sync client catching up, a backup running late. Worrying: a host sending far more than it receives, to somewhere it has never talked to before. Ordinary workstations download; they rarely upload in bulk. First check: direction and destination. Inbound to a known service is boring. Outbound to a new address is the one to look at.


New external destinations

Boring: a software update moved to a new content network, or someone started using a new web service. Worrying: a host checking in with a controller on a regular schedule, around the clock. First check: timing. Human-driven traffic is bursty and stops at night. Beaconing is regular and does not sleep.


Workstation-to-workstation traffic

Boring: a screen-sharing session, a local file share between colleagues, a peer-to-peer update feature. Worrying: one workstation reaching out to many others on file-sharing or remote-management ports. This is what moving sideways through a network looks like from the outside. First check: how many distinct internal destinations, and on which ports. One peer is a colleague; forty peers is not.


Authentication, inventory, and change signals


Failure burst

Boring: a stale password on a phone or a scheduled task after a password change: one account, one source, then it stops. Worrying: guessing at scale. Many accounts from one source, a few attempts each; or one account from many sources; or attempts against names that do not exist. First check: group the failures by source and by account, and look at the hour.


A success right after a run of failures

Boring: the user finally remembered the password. Worrying: the guessing worked. First check: was the success from the same source and device the user normally uses, and was the user actually at work at that moment. This is the one signal in the catalog that should never wait for the weekly review.


Odd-hours or odd-place logons

Boring: a night owl, a traveler, or a maintenance window. Worrying: a service account logging on interactively, a dormant account waking up, or an administrator account used from a workstation it has never touched. First check: whether the account's owner can confirm the logon, and whether the account type matches the logon type. I have been the night owl. Nobody asked, which was the actual problem.


New device alert

Boring: new hire, new printer, someone's smart speaker. Worrying: an unclaimed device, especially one that appeared in a server or management VLAN. A virtual LAN is a logical segment the switches keep separate from the rest. Nothing new should ever show up in a server or management VLAN without a change ticket. First check: vendor prefix, port, VLAN, first-seen time, and whether anyone claims it within the day.


Configuration-change events

Boring: you, or your colleague, doing planned work. Worrying: a firewall rule added, a switch port reassigned, a new administrator account created, or a logging setting turned off. These changes have no change ticket and no one owning up. Attackers tidy up after themselves, and "audit logging disabled" is a favorite. First check: match the event's timestamp and source against your change records and your own memory. Unowned changes are suspicious by definition.


Silence as a signal


The most underrated entries in the catalog are the ones where something stops. Monitoring is naturally tuned to notice things that happen; it has to be deliberately configured to notice things that stop happening. A quiet collector does not complain; it waits politely for data that is not coming.


Monitoring agents go quiet on many hosts at once

Boring: a network problem between the hosts and the collector, or the collector itself has fallen over. In that case the hosts also stop answering pings, or the collector shows other signs of distress. Worrying: something on those hosts stopped the agent, and the hosts are otherwise fine. Disabling monitoring and protection services is a common early step before anything destructive. First check: are the silent hosts still pingable, and is the collector healthy? Silent but reachable, on several hosts within minutes, is one of the strongest signals on this page.


A log source stops sending

Boring: a rotated log file, a full disk, a service restart. Worrying: logging was switched off on purpose. First check: the host's disk and service state, and whether a configuration-change event landed in the same window.


Backup jobs fail

Boring: the usual reasons — a full target, an expired credential, a locked file. Worrying: backup services stopped or backup files deleted, on several systems at once, often shortly before mass encryption begins. The Ransomware Early Warning pillar is built around this and a handful of related signals. First check: whether it is one job with a familiar error or several jobs with unfamiliar ones at the same time. Also check whether backup retention shrank unexpectedly.


At Harbourview Clinic the monitoring agents on four servers went quiet at ten past six on a Friday evening. The on-call administrator closed the ticket with the words "collector again". That was because the collector had in fact fallen over twice that month. The hosts were still answering pings. Nobody checked, because nobody had written down that this was the check. On Monday the server team mentioned, in passing, that they had pushed an update on Friday evening. That update restarted the agent service and never brought it back. The two teams had been looking at the same four hosts all weekend, one of them wondering why the graphs were flat. The ticket template now has a line that reads "hosts pingable? Y/N", and it is not optional.


The signal table


SignalBoring explanationWorrying explanationFirst check
Packets-per-second spike on a portLoop or broken NICHost scanning the LANBroadcast counter rose? pps vs. bps?
Empty port comes upPrinter, desk moveDevice on an unsupervised jackPort location, MAC, time of day
MAC moves between portsDocked laptopAddress impersonationIs the original still online?
DHCP lease churnVisitors, new phonesPool exhaustion attemptDistinct MACs and their vendors
Second DHCP serverHome router plugged inTraffic being redirectedWhich server issued the lease; which port
DNS volume spike, one hostRetry loopTunnelingUnique names under one odd domain?
NXDOMAIN stormStale hostnamesController huntingReadable names or noise? One client?
Client sends DNS outwardHard-coded public resolverBypassing your logsDestination known or unknown?
Unexpected top talkerDownload, sync, backupBulk upload to a new placeDirection and destination
New external destinationNew service or CDNBeaconingRegular timing, active at night?
Workstation-to-workstationScreen shareSideways movementHow many peers, which ports
Auth failure burstStale passwordSpraying or brute forceGroup by source and account
Success after failuresFinally rememberedGuessing workedUsual device? User present?
Odd-hours logonNight owl, travelerService or dormant account misuseOwner confirms? Logon type fits?
New deviceNew hire, new printerRogue, especially in server VLANVendor, port, VLAN, claimed?
Config change eventPlanned workUnowned rule, account, or logging changeMatches a ticket or a memory?
Agents quiet on many hostsCollector or network faultAgents deliberately stoppedHosts still pingable? Collector healthy?
Log source stopsRotation, full diskLogging disabledDisk, service, config change in window
Backup jobs failFull target, expired credentialBackups sabotaged before encryptionOne familiar error or several unfamiliar?

Tip: print this table, or pin it next to your dashboard. The value is not in memorizing it. The value is that when a boring-looking alert arrives, the worrying column is one glance away. And the first check takes less time than deciding whether to bother.


Wiring signals into thresholds you already have


None of this requires a new tool. Every signal in the catalog lives in a metric or event your monitoring platform already handles. The work is turning a graph you glance at into a rule that taps you on the shoulder. Here is the sequence, followed by starting points for each source.


The sequence

  • Find where the signal already lives. Packets per second and broadcast counters are SNMP interface metrics. Failed logons are events in a Windows security log or a syslog stream. New devices are discovery events. Backup results are job reports or emails. If you are not collecting one of them yet, that is your to-do list. The SIEM-Lite pillar covers getting logs into one place cheaply.
  • Set the threshold from the baseline, not from a guess. Look at a few weeks of history and put the line well above the normal peak. Where a source has no baseline yet, start loose and tighten monthly.
  • Label the rule suspicious. Put the word in the alert name. And add the first check to the alert text so the reader does not have to look it up.
  • Route it to a distinct channel from your sick alerts, and require acknowledgment rather than letting it auto-clear.
  • Whitelist the known noise: your own scanner, backup targets, update servers, the monitoring system itself.
  • Review monthly. Every rule that fired and turned out boring is a tuning opportunity; every rule that never fires may be set too high.

Starting points by source

These are shapes, not magic numbers; your baseline supplies the numbers. Written as generic rules, the starter set looks like this:


interface   pps on any access port  > 5x its 30-day peak for 2 min,
            AND broadcast pps NOT rising                 -> [SUSPICIOUS] scan?
interface   any port in group "should-be-empty" goes UP  -> [SUSPICIOUS] new jack
dhcp        new leases per hour > 3x baseline            -> [SUSPICIOUS] churn
dns         NXDOMAIN per client per 10 min > 200         -> [SUSPICIOUS] storm
dns         queries per client per hour > 5x baseline    -> [SUSPICIOUS] volume
flows       outbound bytes from a workstation > 3x its
            daily peak, to a destination first seen today -> [SUSPICIOUS] upload
auth        failed logons from one source > 50 / 10 min  -> [SUSPICIOUS] burst
inventory   new device in VLAN "servers" or "management" -> [SUSPICIOUS] rogue?
heartbeat   agents missing on >= 3 hosts within 5 min,
            hosts still answer ping                       -> [SUSPICIOUS] quiet
backup      >= 2 jobs failed in one night, OR no success
            report by 07:00                               -> [SUSPICIOUS] backups

Most platforms can express every line above with a threshold, an event-count rule, or a "missing heartbeat" check, sometimes with a small script feeding a custom metric. Two of them deserve a note. The "first seen today" destination test needs your flow collector to keep a list of destinations per host. Many can do that as a report even if not as a live rule. A nightly report that lists new destinations per workstation, reviewed the next morning, is an acceptable substitute. The "agents missing on several hosts" rule is easiest when your platform can count down hosts in a group rather than alerting per host. If it cannot, three individual "agent missing" alerts arriving within five minutes is your rule, applied by eye.


Expect the first month to be noisy, in a different way from ops noise. You will discover that the backup server uploads a great deal at night. You will discover that the developer's build machine talks to hundreds of addresses. And you will discover that the authorized vulnerability scan looks exactly like the thing you are trying to catch. Each of those is understood once, whitelisted once, and never bothers you again. What is left after the tuning is a small, trusted set of rules that fire rarely and mean something when they do.


Where to go next


You now have a catalog of the signals your ops monitoring already produces. You have the boring and worrying readings of each, and the first check that separates them. You also have a way to turn each one into a rule without a new tool. The boring explanation will still be right most of the time, and you will still write "loop again" on the ticket. You will just do the check first. One natural companion is Where Monitoring Ends and Detection Begins, which explains why these rules should raise a suspicious label rather than trigger a response. Another is Building a Security View in Your Monitoring System, which arranges the rules into a dashboard and a weekly ritual. Both live in Network Monitoring as a Security Layer.


Put this into practice with SysPeep Monitor

Most of the signals in this catalog come from interface counters, discovery events, and host metrics. SysPeep Monitor already collects these over SNMP, WMI, and SSH. Its thresholds, dependencies, and alert rules let you put a suspicious label on the ones that matter. Those settings let you route those signals to their own notification channel, all from one web console. Install the free trial on a Windows machine and start with the rules above.

Download Free TrialLearn About SysPeep