What to Keep, What to Review, What to Ignore
Once your firewall is logging properly, a new problem replaces the old one. Yesterday the trouble was "we have no logs." Today it is "we have far too many, the disk is filling, and nobody is reading any of it." Both states feel bad, but the second one is fixable with three decisions that most small teams never make on purpose. Decide what to keep and for how long. Decide what to actually read and how often, and what to deliberately ignore.
This guide walks through those three decisions in plain language. You will leave with a retention plan you can defend. You will have a method for estimating how much disk it needs (using your own numbers, not someone else's). You will have a review rhythm that fits a team of one or two, and permission to ignore the internet's background noise without feeling guilty about it. If you have not yet read what a firewall log contains, start there. This guide assumes you know what a deny line and a session-end record are.
In this guide
- Three verbs, three separate decisions
- The events you keep longest
- Bulk allow logs: sample or summarize
- Retention tiers as concepts, not numbers
- Estimating storage: a method, not a guess
- A review cadence a small team can sustain
- What to ignore on purpose
- The keep, review, ignore table
- Where to go next
Three verbs, three separate decisions
The single most useful idea in this guide is that keeping, reviewing, and ignoring are independent choices. New admins tend to fuse them: "if we keep it, someone should read it," or "nobody reads it, so why keep it?" Neither follows. A log you keep but never read is a witness you can call later. A log you review weekly but keep for only a month is a sensor you check but do not archive. And a log you ignore can still be counted. That way, a change in its volume gets your attention even though its individual lines never will.
Separating the verbs also separates the costs. Keeping costs disk and a little collector bandwidth, both cheap. Reviewing costs human attention, the most expensive thing you have. Ignoring costs nothing until the day it turns out you should not have. So the shape of a good plan is: keep generously, review narrowly, and ignore explicitly, writing down what you ignore and why. Note that none of these is alerting. Deciding which few events should interrupt someone in real time is a different discipline, covered in firewall logs and alerts. Everything here is about the retrospective side.
The events you keep longest
Some records are small, rare, and disproportionately valuable. They earn the longest retention, and the reasoning for each is worth understanding so you can extend the list for your own environment.
Denies
A deny is one line per refused attempt and there is no session behind it, so denies are cheap to store. They are also the raw material for nearly every finding in this pillar. Those findings show which internal hosts are reaching for things they should not. They show which services the internet is probing, and which rules are doing real work. Investigations routinely ask "when did this host first start trying that?" and the answer lives in the oldest deny you kept. Keep every deny on every internet-facing and inter-zone rule, and keep them long.
Administrative logins and sessions
Every login to the firewall's management interface, successful or failed, from which address, by which account. These are rare (a few a day at most in a small shop) and tiny. They are the first thing anyone asks for after an unexplained change: "who was logged in at the time?" A failed-login record from an unexpected address is one of the few firewall log lines that means something entirely on its own.
Rule and configuration changes
Most firewalls write an audit record when the policy or configuration changes: which object, which account, sometimes a before-and-after. This is the firewall's own change history. It is the one record you will want when a rule you do not recognize turns up during a cleanup. Keep it for as long as you keep the firewall. If your product's audit records are terse, complement them with the change log described in closing the loop from logs to rules.
VPN and authentication events
Remote-access VPN connections and disconnections, the account and source address, and every authentication failure. These are the records that answer "was this user actually connected when that happened?" and "has anyone been guessing passwords against the VPN?" They are the border between your network and everyone's home Wi-Fi, and a failed-auth pattern here is worth more than a thousand internet-noise denies.
The pattern across all four: low volume, high value, and frequently the subject of a question asked long after the fact. Anything else in your environment with those three properties belongs on the list too. That might be a site-to-site tunnel going up and down, a high-availability failover, or an inspection engine firing on a threat signature.
Bulk allow logs: sample or summarize
Allow logs are the opposite: enormous volume, mostly routine, and occasionally the only place an answer exists. You cannot keep every allow line forever, but "just turn them off" throws away the investigation record. Two techniques let you keep the value while shedding the bulk.
Sampling means keeping a fraction of the raw lines and discarding the rest. You keep one session in every so-many, chosen at random or by a simple rule. Sampling preserves the shape of traffic (who talks to what, roughly how much) at a fraction of the size. It is fine for capacity questions and rough patterns. It is poor for investigations, because the one session you need is probably in the discarded fraction. Some firewalls and collectors can sample at the source; otherwise a log platform can do it on arrival.
Summarizing means rolling raw lines up into aggregates before the raw lines age out. The aggregates give hourly counts of sessions and bytes for each combination of rule, source zone, destination zone, and destination port. They may include the top destination addresses. The summary is tiny compared to the raw data and answers most historical questions ("was this server always talking to that address?"). What it loses is the individual session, and the individual session is what an investigator wants.
The sensible compromise, and the one this guide recommends, is layered. Keep the raw allow lines for a short window that covers your typical investigation. Keep summaries for a long window. And keep raw allows on rules touching sensitive zones as long as you keep denies. That way the recent past is fully reconstructible, the distant past is answerable in outline, and the parts that matter most are always complete.
Retention tiers as concepts, not numbers
You will find retention advice online expressed as fixed numbers of days. Resist copying them. Retention should be derived from what you need to answer, and three questions produce the tiers.
- The investigation window. If you discovered a compromised workstation today, how far back would you need to look to find when it began? Compromises are often discovered long after they start, so this window is longer than instinct suggests. It sets the minimum for raw denies, admin events, auth events, and raw allows on sensitive rules.
- The troubleshooting window. When someone reports "this stopped working last week," how far back do you need raw allow and deny lines to reconstruct what changed? This is usually much shorter than the investigation window and sets the raw-allow tier for bulk rules.
- The pattern window. To notice that something happens "every month-end" or "every quarter," you need at least one full cycle of it. Ideally, you need two, so you can compare. This sets the tier for summaries, which are cheap enough to keep across several cycles.
From those answers you get a short tier (raw everything, sized for troubleshooting). You get a long tier (denies, admin, auth, sensitive allows, sized for investigation). And you get an archive tier (summaries, sized for patterns). If your organization is subject to regulatory or contractual retention requirements, those override your own numbers upward, never downward. The general considerations are in the log retention guide. Write your tiers down with the reasoning beside each. A retention policy that says "ninety days because a blog said so" will not survive its first audit or its first argument about disk.
Tip: Retention is also a deletion promise. Logs you no longer need are a liability during a legal discovery request and a privacy exposure the rest of the time. "Keep everything forever" is not the cautious choice; it is the choice that has not been thought through.
Estimating storage: a method, not a guess
Nobody can tell you how much disk your firewall logs need, because it depends on your traffic, your logging choices, and your collector's storage format. What anyone can give you is a method. It takes one day of measurement and ten minutes of arithmetic.
- Log the way you intend to log for one full day. Denies on, allows per your plan, session-end records enabled where you chose them. If your logging plan is not final, measure the most generous version so the estimate is conservative.
- Measure that day's raw size on the collector. On a file-based collector this is the size of the day's file or directory. On a log platform it is the index or storage size reported for that day. Measure it as stored, because compression and indexing change the number substantially in both directions.
- Measure it per tier, not in total. Denies, admin and auth events, and allows have wildly different sizes and different retention. Filter the day by category and record each category's size separately.
- Correct for the day you picked. A weekend day understates a weekday; a quiet week understates month-end. Either measure a full week and average, or measure a busy weekday and accept a high estimate.
- Multiply each category by its retention in days. Sum the categories. That is your steady-state footprint.
- Add margin. Traffic grows, you will enable more logging once you see its value, and a scanning burst can double a day's denies. Add a generous margin, and if your budget cannot cover it, shorten the bulk-allow tier first and the deny tier last.
- Re-measure quarterly. The estimate ages. A recurring calendar entry to repeat step two is cheaper than a full disk at 2 AM.
On a plain file-based collector, a day's measurement is a one-liner. Suppose logs land in a per-day directory and denies and allows are in separate files:
# Size of one day, total and per category, as stored on disk du -sh /var/log/firewall/day-14/ du -sh /var/log/firewall/day-14/deny.log /var/log/firewall/day-14/allow.log /var/log/firewall/day-14/admin.log # How many lines is that? (useful for spotting a change later) wc -l /var/log/firewall/day-14/*.log
Write the results in the same document as your tiers. The arithmetic is then trivial: each category's daily size times its retention, summed, times your margin. When someone asks "why does the log server need that much disk?", you will have an answer built from your own measurements. That is the only kind that survives the question.
A review cadence a small team can sustain
Review is where good intentions die. A plan that requires an hour a day will be abandoned in a fortnight; a plan that requires nothing will find nothing. The cadence below is built for a team of one or two people who also have a helpdesk queue. It assumes a log platform or a few scripts that pre-aggregate the data. That way, reviewing means reading summaries, never scrolling lines. The tooling guide covers how to build those summaries.
Daily: the five-minute glance
One dashboard, looked at with coffee. It should show yesterday's deny count against the previous few days (is the shape normal?). It should show the count of internal hosts that hit denies (should be a short, familiar list). It should show admin logins and any configuration change (should match what you know happened). It should show VPN authentication failures by account and source, and whether the collector is actually receiving logs from every firewall. The purpose is not to find anything specific; it is to notice when today does not look like yesterday. Take five minutes, with no clicking through. If anything looks odd you write it on the list for the weekly review rather than chasing it now.
Weekly: the thirty-minute review
This is the real work, and it is the subject of the deny-log mining guide. The review covers the top denied external sources and whether any are persistent rather than passing. It covers the top denied destination ports and whether the mix changed. It covers every internal host that hit a deny, with the destination and port, each one explained or ticketed. It covers deny bursts in time and any hit on a rule that should never fire. It includes a read-through of the week's admin logins and configuration changes against the change log. New patterns get a note. Findings get one of three verdicts — tune, investigate, or ignore-and-note — and the notes accumulate into the environment's institutional memory.
Monthly: the deep dive
An hour or two, ideally with a second person. Compare this month's summaries with last month's: total denies, top ports, top internal offenders. Then turn to the rulebase. Look at rules that carried no traffic, broad rules carrying only a few real flows, and temporary rules past their date. That is the substance of closing the loop. Finally, revisit the ignore list (below) and confirm each entry still deserves to be there, and check disk usage against the estimate.
The cadence works because each level feeds the next. The daily glance produces a short list for the weekly review. The weekly review produces notes and tuning candidates for the monthly dive. And the monthly dive produces rule changes that make next month's daily glance quieter. Put all three on the calendar. A review that lives only in someone's good intentions is not a review.
What to ignore on purpose
A public address on the internet is probed constantly by strangers who are probing every address. And every probe is a dutiful deny in your log. This background noise is broad (it hits everyone), random (addresses and ports vary), and low-rate per source (each scanner touches you briefly and moves on). Reading those lines is a waste of your only expensive resource. Ignore them. Specifically:
- Individual denies on the outside interface from sources that appear briefly and never return.
- Denies for services you do not run, from addresses that probed one port and left.
- Broadcast and multicast chatter denied at internal zone boundaries when it is the ordinary kind (discovery protocols, printers announcing themselves to the void).
- Denies you already understand and have documented, such as a known application that retries a legacy port before falling back to the modern one.
The caveat is the whole point: ignore the lines, not the count. Background noise has a normal level and a normal shape, and a change in either is information. If the daily deny count on the outside interface doubles, the noise has stopped being noise. The same is true if one source starts appearing every day, or if the port mix suddenly tilts toward a service you actually run. So the ignore list is a list of patterns you have decided not to read. Each is still counted on the daily dashboard, and each has a written note of what it is and why it is fine. Review the list monthly. An ignore entry that nobody can explain anymore is no longer an ignore entry; it is a blind spot.
The keep, review, ignore table
Everything above, as a single reference. "Long," "short," and "archive" refer to the tiers you derived from your own investigation, troubleshooting, and pattern windows.
| Log category | Keep | Review | Ignore |
|---|---|---|---|
| Denies, internet-facing and inter-zone rules | Long tier, raw | Weekly aggregations; daily count on the dashboard | Individual background-noise lines; never the count |
| Denies from internal hosts | Long tier, raw | Every one, weekly, each explained or ticketed | Nothing until documented |
| Administrative logins, success and failure | Long tier, raw | Daily glance; weekly read-through against the change log | Nothing |
| Rule and configuration changes | Life of the device | Daily glance; monthly during rule cleanup | Nothing |
| VPN connect, disconnect, and auth failures | Long tier, raw | Daily failures by account and source; weekly patterns | Nothing |
| Allows on rules to sensitive zones | Long tier, raw | Monthly: does the traffic still match the rule's purpose? | Individual lines outside an investigation |
| Bulk allows, internal to internet | Short tier raw; archive tier as summaries | Monthly summaries; raw only during investigations | Individual lines |
| Session-start records on bulk rules | Short tier, if logged at all | Only during investigations | Otherwise entirely |
| Broadcast and multicast denies inside | Short tier | Only when the count changes | Individual lines, once understood |
| Collector health (logs received per firewall) | Archive tier as counts | Daily glance | Nothing; a silent firewall is a finding |
One row deserves emphasis: the last one. A firewall that stops sending logs looks, from the collector's side, exactly like a firewall with nothing to report. The count of lines received per device per hour is therefore a log category in its own right. And a zero in it is the most important thing your daily glance can show you.
Where to go next
You now have the three decisions separated and made. Keep denies, admin events, configuration changes, and auth events longest. Keep raw allows briefly and their summaries long. Derive the tiers from your own windows. Estimate disk from a measured day. Review on a daily, weekly, monthly rhythm. And ignore the internet's noise line by line while watching its count. Written down, that is a retention and review policy, and most small teams have never had one.
The natural next step is to make the weekly review pay for itself, which is what deny-log mining is about. When the findings start pointing at the rulebase, closing the loop shows how to turn them into safer rules. The tooling guide covers the practical side of collection, aggregation, and dashboards. All of it lives under Firewall Log Analysis Fundamentals.
Put this into practice with SysPeep Monitor
Retention policies fail quietly: the collector's disk fills, or a firewall stops sending. Nobody notices until the day the logs are needed. SysPeep Monitor can watch the log server's disk and the firewall's health over SNMP and WMI. It can alert you when either drifts toward trouble, so the plan you just wrote keeps working. The free trial runs on any Windows machine.
Download Free TrialLearn About SysPeep