Building a Security View in Your Monitoring System
By now the idea is familiar: the data your monitoring system already collects can answer "is it hostile?" as well as "is it healthy?" The catch is that the answer is scattered. The new-device event is on one page, and the failed logons are in a log somewhere. The flow report is a PDF you generate when you remember to. The interface graphs are on a dashboard built for a different question.
A signal you have to assemble from six places is a signal you will stop assembling. I know because I stopped. For about a year my security review consisted of opening four tabs and looking at each for a minute. I closed them with a feeling of having been thorough. The feeling was the only output.
This guide is about building one place. You will design a security view — a single dashboard plus a small set of alerts. It surfaces the security-relevant signals from your existing data: new devices, new external destinations, authentication failures, and DNS failure rate. It also surfaces outbound volume by host, storms and scans, and the health of the sensors themselves. Along the way you will set severity labels that separate "suspicious" from "sick" and derive thresholds from baselines instead of guesses. You will route security alerts to their own channel and adopt a weekly half-hour review with a checklist. And you will keep evidence long enough to be useful. The guide ends with a starter layout you can copy.
In this guide
- What the view is for
- What goes on it
- Severity labels: suspicious vs. sick
- Thresholds from baselines
- A distinct channel for security alerts
- The weekly 30-minute review
- Retaining evidence long enough to matter
- The starter dashboard layout
- Where to go next
What the view is for
A security view in a small shop is not a wall of screens. Nobody is going to sit in front of it. It is a page you read, on a schedule, in about half an hour. It also includes a handful of alerts that reach you between readings when something cannot wait. Three design rules follow from that.
First, every panel answers one question and implies one action. "Which devices appeared this week that nobody has claimed?" implies "claim or remove them." A panel that is merely interesting does not belong. Second, the view is time-boxed: if it takes longer than thirty minutes to read, it has too much on it. Third, it is built entirely from data you already collect. If a panel needs a new sensor, park it on a wish list and build the view with what you have. A modest view you actually read beats an ambitious one you abandon.
At Ashgrove Housing the first security view had twenty-two panels, took most of a morning to read, and was read twice. The second one was built in an afternoon from three dashboards that already existed. Those were the agent-heartbeat page, the failed-logon report, and the new-device list. Each was copied onto one page and given a question and an action. The view was read every Tuesday, because it fitted between two meetings. The insurer's renewal questionnaire asked how quickly an unknown device on the server network would be noticed. The administrator opened the review log, found the answer, and wrote it down in about a minute. The previous renewal's answer had taken a week of drafting and said "promptly".
What goes on it
Seven panels cover the ground. Each is described as what it shows, the question it answers, and the action a reader takes.
1. New devices
Shows: every device first seen in the last seven days. Each has its switch port, VLAN, MAC vendor prefix, first-seen time, and a "claimed by" field you fill in. Question: is there anything here nobody owns? Action: claim it or remove it. Anything first seen in a server or management VLAN should also raise an alert rather than waiting for the weekly look.
2. New external destinations
Shows: from flow data or firewall logs, each internal host paired with the external addresses it contacted for the first time in the last day. The panel also shows how many connections and at what hours. Question: which workstations started talking to somewhere new, and does the timing look human? Action: explain each new destination for a workstation; regular, round-the-clock contact with an unfamiliar address goes to triage.
3. Authentication failures
Shows: failed logons over the last day, grouped two ways — by source address and by account — with any success that followed a burst highlighted. Question: is anyone guessing, and did it work? Action: one account from one device is a help-desk ticket. Many accounts from one source is triage. A success after a burst is a same-day phone call to the user.
4. DNS failure rate and top clients
Shows: the resolver's failure rate over time, plus the top ten clients by query count and by failed-query count. Question: is the failure rate climbing because everyone is failing (a resolver problem) or because one host is? Action: a single host burning through gibberish names, or towering over the rest on volume, goes to triage.
5. Egress volume by host
Shows: the top ten internal hosts by outbound bytes in the last day, each compared with its own usual daily peak. Question: is anything sending far more than it normally does? Action: servers and backup targets with known jobs are expected. (The backup server will top this list every day, and it is allowed to.) A workstation at several times its normal outbound volume goes to triage. The Data Exfiltration Patterns pillar covers what the worrying shapes look like.
6. Storms and scans
Shows: access ports whose packets per second exceeded their baseline in the last day, with bytes per second and broadcast rate alongside. Question: for each spike, did broadcast rise with it (a storm) or not (a possible scan)? Action: storms go to the ops queue; a high-packet, low-byte, low-broadcast spike from one port goes to triage with its flows.
7. Sensor health
Shows: hosts whose monitoring agent has stopped reporting, log sources that have gone quiet, and last night's backup results, all on one panel. Question: did anything stop talking, and did several things stop at once? Action: one silent host is ops; several silent hosts that still answer pings is the most urgent signal on the whole view.
Severity labels: suspicious vs. sick
A security view needs two labels working together. The first is a category. Every alert is either sick (something is broken; the fix is operational) or suspicious (something is behaving without an operational explanation). For a suspicious alert, a person must judge whether it is hostile. The second is a severity: how fast it needs a look. Keep them separate. A suspicious alert can be low severity. And a sick one can be critical. Confusing the two axes is how a broken transceiver ends up in the security channel and a scan ends up in the ops queue.
| Label | Meaning | Channel | First look within | Example |
|---|---|---|---|---|
| [SICK] warning | Degraded, not urgent | Ops queue | Next business day | Rising CRC errors on a port |
| [SICK] critical | Broken, users affected | Ops pager | Minutes | Core switch unreachable |
| [SUSPICIOUS] review | Needs a human judgment, can wait for the weekly look | Security channel, digest | Weekly review | Unclaimed new device on a user VLAN |
| [SUSPICIOUS] triage | Needs a first check within the working day | Security channel, immediate | Four business hours | Failed logons across many accounts from one source |
| [SUSPICIOUS] act | Contain first, then investigate | Security channel and phone | Minutes, any hour | Agents quiet on several reachable hosts; success after a failure burst |
Put the label in the alert name itself, so it survives every email client and phone notification. Then put the first check in the alert body, so whoever reads it knows what to do without opening this guide. Two rules make the labels stick. A suspicious alert never clears itself because the metric returned to normal (a scan that stopped is still a scan). And only the "act" level is allowed to interrupt an evening. If your act-level alerts fire more than rarely, they are mis-tuned, and the tuning is the review's job.
Thresholds from baselines
Guessed thresholds are where security views go to die: too tight and they shout all day, too loose and they never speak. Set them from baselines instead. A baseline is a documented measurement of what normal looks like for one metric in one place. The guide What Is a Baseline covers the concept properly. For the security view, the procedure is short.
- Pick the metric per object, not per network: outbound bytes per host, packets per second per port, failed logons per source, queries per client. Security signals live in individuals, not aggregates.
- Look at a few weeks of history and note the normal peak for each object, by time of day where it matters.
- Set the line well above the peak — three to five times is a common starting multiplier. Require it to hold for a couple of minutes, so a single burst does not fire it.
- Add a "first seen" rule where the interesting thing is novelty rather than volume. Examples are a destination this host never contacted, a device never inventoried, or a source address never seen in auth logs. Novelty has no baseline; the baseline is the list of things already seen.
- Whitelist by name the objects whose normal is legitimately extreme: the backup server, the update server, the authorized scanner, the monitoring system.
- Rebaseline monthly, or whenever the network changes shape.
Worked example, with made-up numbers. You want an egress threshold for workstations. History shows most send well under 200 MB outbound on a normal day. A handful of heavier users peak near 600 MB when they upload video for marketing. You set the per-workstation rule at four times each host's own thirty-day peak. That lands most machines near 800 MB and the heavy users near 2.4 GB. You require the excess to be sustained across an hour rather than a single minute. Then you add a "first seen" condition. Fire at a lower multiple, say twice the peak, when the destination is one this host has never contacted before. The result rarely fires. When it does, it is a workstation sending several times its usual volume to somewhere new. That is exactly the sentence you want to read.
A distinct channel for security alerts
Security alerts mixed into the ops stream get treated like ops alerts: skimmed, snoozed, and cleared when the graph goes green. Give them their own home. A separate mailbox, a separate chat channel, a separate notification group — the mechanism matters less than the separation. What you are buying is a place where everything is suspicious by definition, so nothing gets lost among disk-space warnings.
Within that channel, apply the severity rules. Review-level alerts arrive as a daily or weekly digest, not one message each, so the channel is not itself a source of fatigue. Triage-level alerts arrive individually, during working hours, and stay open until a person marks them boring or escalates. Act-level alerts arrive individually at any hour, by whatever means actually wakes you, and are the only ones that do. If your platform supports acknowledgment, require it for everything suspicious. If it does not, a simple rule does the same job: nobody may delete a suspicious alert without writing a one-line verdict. The Who Owns What guide covers who reads the channel and who they call.
The weekly 30-minute review
The dashboard is worth exactly as much as the habit of reading it. Put thirty minutes on the calendar, the same slot every week. And follow the same sequence so the review becomes mechanical enough to survive a busy week. Five minutes per panel is the budget. If one panel needs more, that is a triage item, not a longer review. A review that runs long is the one that gets skipped next week.
- Minutes 0–5, new devices: for every unclaimed device this week, find an owner or schedule its removal. Write the owner into the panel so next week's list is shorter.
- Minutes 5–10, new destinations: read the workstation rows only. For each new destination, either recognize it, ask the user, or open a triage item. Servers get a glance for anything surprising.
- Minutes 10–15, authentication: confirm each burst had a boring verdict recorded. Chase any success-after-burst that was not already called the same day.
- Minutes 15–20, DNS and egress: check the failure-rate trend and the two top-ten lists for a host that towers over the rest. Check the egress list for any workstation above its own peak.
- Minutes 20–25, storms, scans, and sensor health: confirm each packet spike was classified. Confirm that every quiet agent, quiet log source, and failed backup has an operational explanation on record.
- Minutes 25–30, tuning notes: list every alert that fired and turned out boring, and every rule that never fired. Adjust one or two thresholds or whitelist entries. Write one line in a review log: date, anything open, what you changed.
Tip: the review log matters more than it looks. A page of one-line entries saying "reviewed, nothing open" is proof that the view was actually read. It is proof to management, to an insurer, and to yourself. It is also the fastest way to spot when a signal that was quiet for months starts appearing every week.
Retaining evidence long enough to matter
Operations data is usually kept for the present tense. High-resolution counters stay for a few days, then are rolled up into averages that show trends but lose detail. Security questions live in the past tense. When did this host first contact that address? Which account failed before the one that succeeded? And intruders are often present for weeks before anything visible happens. If your evidence ages out faster than you notice, the investigation starts from nothing.
Here is the rule of thumb. Keep each raw source at least as long as it plausibly takes you to notice a problem in it. Keep it longer where storage is cheap. (Storage is nearly always cheaper than the conversation about why it was deleted.) Rollups are fine for graphs. They are useless as evidence, because a five-minute average of outbound bytes cannot tell a responder which connection carried the data. Starting points, to be adjusted to your storage and any legal or insurance requirements you are under:
| Source | Keep raw for at least | Why that long |
|---|---|---|
| Flow records | Several weeks to a few months | Beaconing and slow uploads are found by looking back, not forward |
| DNS query logs | Several weeks to a few months | "When did this host first ask for that name?" is the key question |
| Authentication logs | Months | Password guessing is slow by design; the successful attempt may be far from the first failure |
| Inventory history (first seen, port, VLAN) | A year or more | Small, and "when did this appear?" never stops being useful |
| Interface counters | Days at full resolution, then rollups | Scans and storms are investigated the same week; long history only needs trends |
| Config-change and admin events | A year or more | Tiny, and the first thing a responder asks for |
| Review log and alert verdicts | Indefinitely | Proof the layer was operated, and a record of what was decided |
Two practical notes. First, retention is only useful if the data is where a responder can read it. A log that exists on a server that has just been encrypted is not evidence. Ship logs off the hosts that generate them, to a collector that is itself monitored. Second, alert on the collector going quiet, because a log source that stops sending is both an ops problem and a security signal. The mechanics of collecting logs centrally and raising alerts from them are covered in Alerting on Logs.
The starter dashboard layout
Here is the whole view as a single table you can build from. Top row is what you look at first, because it is where the "act" signals live. The bottom row is the weekly-review material. Each panel names its data source, its time window, its question, and its action.
| Row | Panel | Data source | Window | Question | Action if not boring |
|---|---|---|---|---|---|
| Top | Sensor health | Agent heartbeats, log collector, backup reports | 24 hours | Did several things go quiet together? | Act: check reachability, then contain and escalate |
| Top | Authentication failures | Domain, VPN, Wi-Fi auth logs | 24 hours | Guessing? Did it work? | Triage; act on success-after-burst |
| Top | Storms and scans | SNMP interface counters | 24 hours | Broadcast rose with the spike, or not? | Storm to ops; scan to triage with flows |
| Middle | New external destinations | Flows or firewall logs | 24 hours, per host | Who started talking somewhere new, and when? | Explain or triage; regular night-time contact is triage |
| Middle | Egress volume by host | Flows, per host vs. own peak | 24 hours | Anything sending far more than usual? | Triage workstations above their peak |
| Bottom | DNS failure rate and top clients | Resolver logs | 7 days | Everyone failing, or one host? | Resolver problems to ops; one-host storms to triage |
| Bottom | New devices | Discovery and inventory | 7 days | Anything unclaimed? | Claim or remove; server-VLAN arrivals alert immediately |
Build it in that order, too. Sensor health and authentication can usually be assembled in an afternoon from data you already have, and they carry the most urgent signals. New destinations and egress need flow data, which you may already collect or may need to enable on the firewall. DNS and inventory panels depend on how your resolver and discovery expose their data. A half-built view with three panels you read is already a security layer; finish the rest as the data allows.
Where to go next
A security view is seven panels, two labels, and thresholds borrowed from baselines. It has a channel of its own, a half-hour every week, and evidence kept long enough to be worth having. None of it needs a new sensor. It does need to be one page; four tabs and a feeling of thoroughness do not count. If you have read the rest of this pillar, you have the reasoning behind each piece. There is One Dataset, Two Lenses for why the data works. There is Security Signals Hiding in Your Ops Monitoring for the full catalog behind the panels. And there is Where Monitoring Ends and Detection Begins for what happens after triage. All of them live in Network Monitoring as a Security Layer.
Put this into practice with SysPeep Monitor
The view in this guide is a dashboard, a set of thresholds, and a notification channel. That is what a monitoring platform is for. SysPeep Monitor gives you live dashboards and reports from one web console. It provides thresholds and alert rules with dependencies designed to keep noise down, plus email and other notifications you can route by label. It also offers a choice of database backends for keeping the history. Start the free trial on a Windows machine and build the top row this afternoon.
Download Free TrialLearn About SysPeep