Log Retention: Policy, Compliance, and Storage Math
Ask how long your network keeps its logs and you will usually get one of two answers: a shrug, or a number that nobody chose. On most small networks, retention is decided by accident. Logs pile up until a disk complains, then somebody deletes the oldest files, and that becomes the policy. That sets the most important property of your log archive — how far back it can answer questions. The size of a disk someone bought for unrelated reasons is making the decision.
Retention is simply the answer to "how long do we keep which logs, and where?" — written down and enforced by machinery instead of by disk pressure. Get it wrong in one direction and the evidence you desperately need next quarter has already been deleted. Get it wrong in the other and you pay forever to store noise you will never read. In that case, you drown every search in years of sludge, and hold data that can be demanded from you later.
This guide gives you the whole decision. It covers the four forces that drive retention lengths and a hot-warm-cold tiering strategy that satisfies all of them cheaply. It works the storage arithmetic end to end with a fill-in worksheet and covers the rotation machinery that enforces the policy. It provides a sample policy table you can adapt and get signed off this week.
In this guide
- The two ways retention goes wrong
- The four retention drivers
- Hot, warm, cold: the tiering strategy
- The compression dividend
- The storage math, worked end to end
- Rotation: the machinery that enforces the policy
- Deletion as policy, integrity as posture
- A sample retention policy to adapt
- Where to go next
The two ways retention goes wrong
Too short fails like this. A user reports their account doing things they did not do. You dig, and the digging points backward: the odd logins started well before anyone noticed. The trail you need — the first bad login, the address it came from, what was touched in week one — lives about forty days in the past. Your collector keeps thirty. The most important pages of the story have already been shredded, on schedule, by your own cleanup job. This is the defining pain of under-retention: incidents are routinely discovered long after they begin. Compromises sit quiet for weeks or months; slow hardware decay and creeping misconfigurations have even longer fuses. An archive that only remembers last month can testify about almost nothing that matters.
Too long fails more politely, in three ways. First, cost. Not so much disk (we will do that math, and it is smaller than you fear). The cost is the quiet operational tax of ever-growing backups, migrations, and indexes. Second, search sludge: every hunt now wades through years of irrelevant lines, so queries slow down and answers hide better. Third — the one admins forget — liability. Logs you hold can be demanded from you — by lawyers in a dispute, by regulators, by anyone with standing to compel it. Old logs are full of things you must then produce, review, and explain. That includes personal data about users you had no reason to still be holding. Data you have deliberately and uniformly deleted, per written policy, cannot be demanded; data you kept "just in case" can.
So the goal is not "keep everything forever" or "keep it lean." It is a deliberate window per kind of log, long enough for every question you genuinely expect to ask, and not much longer.
The four retention drivers
Four different needs pull on the retention dial, each with its own natural timescale. Your policy is whatever satisfies all four for each class of log.
1. Operational hindsight — days to weeks
Most troubleshooting questions look back hours or days. What happened before the crash? When did this interface start flapping? Did the backup job log anything odd this week? The strongest tool here is comparison — "what does a normal Tuesday look like?" — which needs a few weeks of history to answer. Beyond a month or two, day-to-day troubleshooting value fades fast.
2. Security forensics — months
This is the day-forty problem from above. Security questions are the ones asked longest after the fact, because intrusions are discovered late and investigations then walk backward. When did this account first behave oddly? What did that address touch? How long has this been going on? Authentication and access logs — who logged in, from where, succeeding or failing — are the class this driver cares about most. Months of look-back is the working need.
3. Compliance obligations — often years, and not yours to guess
If your organization handles payment cards, health records, government work, or regulated money, some framework almost certainly has opinions about your logs. The honest, evergreen shape of it: many regimes set minimum retention periods, most commonly for authentication and access logs, and the minimums are often a year or more. The specifics depend entirely on your industry, your location, and your contracts. They include which framework binds you, which log classes it names, and what its floor is. And they change. Do not take retention minimums from a blog post, this one included.
Tip: this guide teaches the shape of the problem, not your legal answer. Ask whoever owns compliance where you work — a manager, an auditor, a contract — one concrete question. "What are our minimum retention obligations for auth and access logs?" Bring the sample policy table from the end of this guide; a concrete draft gets a faster, better answer than an open-ended question ever does.
4. Internal evidence — the records someone will eventually ask for
The quietest driver: logs as receipts. When the ISP disputes your outage report, the archive is your side of the argument. The same goes when a vendor insists their device never rebooted, or when management asks what actually happened last quarter. This is the same instinct behind keeping an uptime evidence file. Certain records exist not for daily use but because someday, someone with an opinion will need to be shown a fact. A year of history covers nearly every argument of this kind.
Hot, warm, cold: the tiering strategy
The four drivers pull toward different lengths, and the trick is noticing they also pull toward different speeds. Yesterday's logs get searched constantly; last spring's get searched during incidents; the old stuff exists to be produced on demand. So do what your kitchen does: daily things on the counter, monthly things in the pantry, yearly things in the deep freezer. In storage terms, three tiers:
Hot — recent, raw, instantly searchable. Days to weeks. The newest files sit uncompressed on the collector, exactly as your central syslog server wrote them, where grep and tail answer in seconds. This tier serves operational hindsight, and it is where alerting reads from. Two to four weeks is the usual sweet spot.
Warm — compressed, still online, searched at coffee-break speed. Months. Older files get compressed in place (or moved to a bigger, slower disk) and stay reachable with zgrep, the version of grep that reads compressed files directly. Searching a year of warm archive takes minutes instead of seconds — entirely acceptable for the incident-investigation questions this tier exists to answer. This is where security forensics and internal evidence live; twelve months is a sensible default.
Cold — exported, offline or off-box, kept for obligation-driven years. When a compliance floor or contract demands multi-year retention, export the relevant classes — typically auth and access logs — as compressed archives. Use storage separate from the collector: another box, removable media, wherever your backups live. Be honest about what this tier is: a compliance artifact, not an operational tool. You will almost never search it. It exists so that if an auditor, a lawyer, or a future investigation asks, you can produce it. Build cold storage only if a real obligation demands it; do not invent one out of hoarding instinct.
Tiering is also what makes generous retention affordable, because of one convenient fact about log files.
The compression dividend
Log files are among the most compressible things on your network. Every line repeats the same hostnames, the same message templates, the same field labels, thousands of times a day — and repetition is exactly what compression eats. A general-purpose compressor like gzip routinely shrinks plain-text logs to a tenth of their size, often far beyond. Ratios vary with content, so treat 10:1 as a deliberately conservative planning number. Measure your own once you have real archives: compress one old file, compare sizes, and use that ratio in the worksheet below.
Watch what this does to the cost of a year. Take the modest fleet from the collector sizing guide — about 50,000 lines a day at roughly 150 bytes per line:
50,000 lines/day × 150 bytes ≈ 7.5 MB/day raw 7.5 MB/day ÷ 10 (compression) ≈ 0.75 MB/day compressed 0.75 MB/day × 365 days ≈ 275 MB per year of warm archive
A full year of every log the fleet produces, compressed, fits in less space than a phone video. This is why the warm tier can afford to be long: past the hot window, each additional month of memory costs almost nothing. The expensive tier is hot — raw and uncompressed — which is exactly the tier you keep shortest.
The storage math, worked end to end
Now the full budget, with every assumption stated so you can swap in your own numbers. Step one: estimate daily volume per device class. Classes matter because devices differ wildly — a switch murmurs, a firewall narrates its whole day. For a composite 30-device network:
| Device class | Count | Lines/day each | Class total |
|---|---|---|---|
| Firewall (internet edge) | 1 | 20,000 | 20,000 |
| Servers | 8 | 2,500 | 20,000 |
| Switches | 8 | 500 | 4,000 |
| Everything else (APs, UPS, NAS) | 13 | ~450 | 6,000 |
| Fleet total | 30 | 50,000 |
Do not guess if you do not have to — once the collector is running, yesterday's truth is one command away, counting lines per host:
wc -l /var/log/remote/*/$(date -d yesterday +%F).log
Step two: multiply through the tiers. Policy for the example: 30 days hot, 12 months warm, and a 3-year cold obligation for the auth-heavy classes. The cold obligation is rounded here as full-fleet years to keep the arithmetic honest and pessimistic:
Raw volume: 50,000 × 150 bytes ≈ 7.5 MB/day Hot tier: 7.5 MB × 30 days ≈ 225 MB Warm tier: 0.75 MB compressed × 365 days ≈ 275 MB Cold tier: 275 MB/year × 3 years ≈ 825 MB Policy total ≈ 1.3 GB
Step three: land on a disk budget. Triple the total for headroom — volume spikes, a device left in debug mode, growth — and the whole multi-year policy needs about 4 GB. That is the punchline of the storage math: at small-network scale, disk is almost never the real constraint. The honest caveats: a busy edge firewall logging every denied connection can multiply volume tenfold or more. That is still modest disk, but worth tuning (what to log). Every line you drop is retention you can afford. And if you run an indexing log platform, indexes add real multiples on top of raw storage. The binding constraints are the ones from the top of this guide — search sludge and liability. That is why the policy limits itself even when the disk would not.
Here is the same math as a worksheet. Fill in the right column with your numbers and you have a defensible disk budget in ten minutes:
| Step | Example | Yours |
|---|---|---|
| 1. Lines/day, summed per device class | 50,000 | |
| 2. × bytes per line (150 if unmeasured) = raw per day | 7.5 MB | |
| 3. × hot days = hot tier | × 30 = 225 MB | |
| 4. ÷ compression ratio (10:1 conservative) = compressed per day | 0.75 MB | |
| 5. × warm days = warm tier | × 365 = 275 MB | |
| 6. × cold years = cold tier | × 3 = 825 MB | |
| 7. Sum of tiers × 3 headroom = disk budget | ≈ 4 GB |
Rotation: the machinery that enforces the policy
A policy on paper does nothing; rotation is what enforces it. Rotation means closing the current log file on a trigger, starting a fresh one, and handling the old files by rule. The trigger is either size ("roll when the file hits a limit") or date ("new file every day"). Date-based wins for log archives, because "one file per host per day" makes both searching and expiring old data trivial. Onto the trigger you bolt two habits. First, compress on rotate (yesterday's file gets gzipped after a short grace period, moving it hot to warm). Second, delete on schedule (files past the warm window get removed — after the cold export, for classes that have one).
The tooling is wonderfully boring. Linux systems ship a standard helper (logrotate) that does size- or date-based rotation with compression and expiry. Syslog daemons can write dated files natively. Log platforms have retention settings in a menu. If you built the collector from this library's guide, your rotation "system" is already running as two scheduled find one-liners. One compresses files older than a week. The other deletes archives past the policy age. That genuinely is a complete implementation at this scale. The only change the policy asks of it: pick the deletion age per log class deliberately, instead of one blanket number.
Now the warning that earns its space, because rotation fails in two silent and opposite ways. Rotation too aggressive: a mistyped age or size limit quietly deletes logs sooner than policy says. Nobody notices, because deletion looks exactly like working rotation until the day an investigation reaches for month eight and finds nothing. Rotation broken or absent: the compress or delete job stops running — wrong path after a reorganization, scheduler silently disabled. The disk fills until logging stops or takes the collector down with it. Both failures announce themselves in the same place: the volume of the archive itself.
Tip: monitor the log volume like any other vital sign. Alert on the collector's free disk, watch bytes-per-day per host, and check the age of the oldest archive. Each catches a real failure: a host at zero bytes has stopped sending. A host at ten times normal is having an event (or was left in debug). An oldest-file age far below the policy window means rotation is deleting too soon. Free disk shrinking fast means it is not deleting at all. Silent rotation failures are found by looking — so make the machine look.
Deletion as policy, integrity as posture
A retention policy is really a deletion policy wearing optimistic clothing, so treat deletion with the same respect as collection. Three properties make it defensible. Documented: the durations per class are written down and approved — the sample table below is the document. Automatic: machinery deletes on schedule, so the outcome never depends on someone remembering, hesitating, or being on vacation. Uniform: the policy is applied the same way every day. That lets you say, calmly and credibly, "data past our retention window is routinely deleted under written policy" — a normal, defensible answer.
What destroys that defensibility is ad-hoc keeping. Think of the personal stash of "interesting" logs on a workstation, the export someone squirreled onto a USB drive, the old collector VM nobody decommissioned. Now your true retention is unknown even to you. You carry the liability and cost of the copies while losing the predictability that justified the policy. The written document has quietly become fiction. The rule that keeps the policy real: copies follow the policy too. If an investigation needs logs preserved past their window, that is a deliberate, documented exception — a hold, noted in writing, with an end date — not a hoard.
Integrity is the other half of trusting an archive: being able to say the logs have not been quietly altered. At small-shop scale, skip the ceremony and adopt an append-only posture — a stack of ordinary-care habits. The collector is its own box, so a compromised device cannot rewrite its history there. Shell access to it is restricted to the admins who genuinely need it — the shorter that list, the more the archive is worth as evidence. Rotated archives are made read-only, owned by root. Cold exports get a checksum file (sha256sum output stored alongside) so tampering is at least detectable. The cold exports live on storage that a collector compromise cannot reach. None of this is a courtroom chain of custody — be honest about that if it ever matters. But it is the difference between "trust me" and "here is why the archive is credible," at a cost of roughly one afternoon.
A sample retention policy to adapt
Everything above, condensed into the one-page document. Durations are sensible starting points for a small network with no known compliance floor. Adjust with the drivers, replace the cold column with your confirmed obligations, and delete rows you do not collect:
| Log class | Hot (raw) | Warm (compressed) | Cold (exported) | Rationale |
|---|---|---|---|---|
| Auth & access: logins, VPN sessions, admin access | 30 days | 12 months | Per your confirmed obligations | Forensics look-back; the class compliance minimums most often name |
| Infrastructure events: link changes, config changes, HA, reboots | 30 days | 12 months | None | Troubleshooting and change history; internal evidence |
| Server system logs | 30 days | 6 months | None | Operational hindsight; comparisons |
| High-volume firewall deny / connection logs (if collected) | 7 days | 30 days | None | Bulk noise; value decays in days |
| Debug / verbose captures | Life of the investigation | None | None | Exists only while a problem is being chased |
Then the step that turns a table into a policy: get it signed off. Take it to whoever owns risk where you work — your manager, the owner, the compliance contact. Explain the columns in two minutes, incorporate their answer to the compliance question, and get an approval in writing. That signature is what makes the deletion schedule an organizational decision instead of your personal gamble, and it costs one short meeting. Revisit the table once a year and whenever your obligations change.
Where to go next
You now have what most networks never write down: a deliberate answer to "how long do we keep logs," with reasons. The drivers set the windows. Tiers make the windows cheap. Compression makes the math small. Rotation enforces it while you sleep, and a signed page makes it defensible. Retention is one layer of a larger practice — collecting the right events, keeping clocks honest, alerting on the patterns that matter. The rest of the Syslog & Log Collection guides build the other layers in the same practical style.
Put this into practice with SysPeep Monitor
A retention policy protects your view of the past; someone still has to watch the present. SysPeep Monitor keeps live eyes on device health, reachability, and disk space — including the collector disk your whole retention plan depends on. It uses auto-discovery, SNMP and WMI checks, and alert rules designed to stay quiet until something matters. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep