Log Retention for Compliance: Keeping Enough, Not Everything
The first time a compliance requirement lands on a small IT team, somebody says "fine, we'll just keep everything forever." It sounds safe. It is not. Keeping everything forever means paying for storage nobody will read and searching through debug noise to find the one login that matters. Worst of all, it means having no answer to the auditor's real question. That question is not "do you have logs?" It is "which logs do you keep, for how long, and how do you know nobody has tampered with them?"
Retention is a design problem, not a storage problem. Stop thinking of "the logs" as one pile. Start thinking of them as a few tiers, each kept for a different reason and a different length of time. Most of what your devices emit is troubleshooting chatter, worthless after a few weeks. A smaller slice is what an investigator would need months later. A smaller slice still is what your framework requires you to keep, for whatever length it specifies. That slice must be in a form you can prove has not been altered.
By the end of this guide you will have built those tiers and decided which log sources land in each. You will have understood the two words auditors care about most — immutability and integrity — in plain language. You will also have a retention policy template you can fill in this afternoon and a way to argue for the storage it needs.
In this guide
- Why "keep everything forever" is the wrong answer
- Retention as tiers driven by purpose
- What belongs in the compliance tier
- Immutability and integrity in plain language
- Proving nothing was altered
- The cost conversation
- A retention policy template
- Where to go next
Why "keep everything forever" is the wrong answer
Start with what a log actually is: a line of text a system writes when something happens. A firewall writes a line for every connection it allows or denies. A server writes a line when a service starts, a user logs in, or a disk fills. A switch writes a line when a port goes up or down. Multiply that across every device on a network and you get a steady river of text. Most of it is repetitive and most of it is useful only for a very short time.
Keeping all of it indefinitely fails in three ways. First, cost: bulk "allow" entries from a firewall can outweigh every other log source combined, and they are the entries nobody ever reads twice. Second, findability: an investigation into a suspicious login is slower, not faster, when the relevant lines are buried in millions of routine ones. Third, and this is the one that surprises people, it is worse for compliance. Suppose an auditor asks "what is your retention policy?" and hears "we keep everything." They will follow up with "where, for how long, who can delete it, and how do you know it is complete?" And "everything" has no good answer to any of those. A defined policy does.
There is also a quiet legal and privacy angle that your organization's policy people will care about. Logs can contain personal data. Holding personal data longer than you need it is itself something many frameworks frown on. "Keep enough, not everything" is not just thrift. It is the position most frameworks actually want you to take.
Retention as tiers driven by purpose
For each kind of log, ask "who will want this, and when?" There are only three answers, and they map onto three tiers.
| Tier | Purpose | Who reads it | Typical length | Where it lives |
|---|---|---|---|---|
| Troubleshooting | "Why did this break just now?" | You, this week | Short: days to a few weeks | Fast, searchable, on the collector |
| Investigation | "What happened around that incident?" | You or a responder, months later | Medium: months | Compressed, still searchable |
| Compliance | "Prove this control operated across the period" | An auditor, possibly years later | Whatever your framework requires; look it up, write it down | Protected, write-once, integrity-checked |
The troubleshooting tier is the busy one. It is where every log line lands first, where you search when a user says "the VPN was weird an hour ago". It is where volume is highest. Nobody needs a debug message from the wireless controller three months on, so this tier is short by design. (The wireless controller will not take it personally.)
The investigation tier exists because incidents are often discovered late. Somebody notices an odd account, and the useful question is what that account did over the past few months. This tier holds the security-relevant subset of the logs, compressed and still searchable, long enough to reconstruct a story.
The compliance tier is the one this guide is really about. It holds the specific log sources your framework names, for the length your framework specifies. Its storage is designed so that nobody — including you — can quietly change those logs. The retention length is not a matter of opinion. Find the requirement in your framework, your contracts, or your organization's own policy, write the number down, and treat it as an input. If several requirements apply, the longest one wins for the sources it covers.
Meridian Parts had every log source set to ninety days, because ninety was the default when the collector was installed. Nobody had asked for anything else. The year their largest customer added an audit clause to the contract, the request list asked for twelve months of authentication logs. The team had ninety days, a very tidy collector, and a retention screen that looked deliberate. The auditor recorded the shortfall in one line, and the team spent the next nine months waiting to have twelve. Nobody had ever looked up the number; they had inherited it from a dropdown.
Notice that the tiers are not three copies of everything. They narrow as they lengthen. A log line may live in the troubleshooting tier only, or in troubleshooting and investigation, or in all three. Deciding which sources flow how far is the whole design.
Tip: the mechanics of expiring old data, rotating files, and sizing disks are covered in the log retention guide in the syslog pillar. This guide assumes those mechanics exist and focuses on what to keep long and how to prove it is intact.
What belongs in the compliance tier
Your framework has the final say, but the compliance tier almost always covers the same handful of categories. They answer the auditor's recurring questions: who got in, who changed what, what was blocked, and what did you notice. The table below is a sensible starting split. Compare it against your framework's control list and adjust.
| Log category | Examples | Tier | Why |
|---|---|---|---|
| Authentication | Directory logins, VPN sessions, firewall admin logins, switch and router logins, failed attempts, lockouts | Compliance | Answers "who got in, and when" |
| Administrative actions | Account creation, group changes, privilege grants, password resets by admins | Compliance | Answers "what did privileged users do" |
| Configuration changes | Firewall rule edits, switch config saves, server policy changes, monitoring rule changes | Compliance | Backs the change-management control |
| Firewall deny and security events | Blocked connections, intrusion alerts, malware detections | Compliance | Shows the boundary control working |
| Security alerts and their handling | Monitoring alerts, acknowledgements, ticket references | Compliance | Proves detection and response |
| Backup job results | Success, failure, restore tests | Compliance | Backs continuity controls |
| Firewall allow (bulk) | Every permitted connection | Troubleshooting, sometimes investigation | Huge volume, rarely needed later |
| Debug and verbose | Application debug output, chatty service logs | Troubleshooting | Useful for days, not months |
| Performance metrics | CPU, bandwidth, latency samples | Troubleshooting (raw), investigation (summarized) | Summaries are enough for availability reports |
Two things stand out. The compliance tier is mostly events about people and changes, which are small. The excluded categories are mostly events about traffic and internals, which are enormous. This is why a well-designed compliance tier is usually a fraction of the size of the troubleshooting tier, even though it lasts many times longer. People are rarer than packets, and they do less per second.
One honest caveat: some frameworks or contracts do require bulk traffic logs to be retained for a defined period, particularly where a network carries something sensitive. If yours does, the same tiering logic applies; the compliance tier is simply bigger, and the cost conversation later in this guide matters more.
Immutability and integrity in plain language
An auditor examining your compliance tier is asking one underlying question: could someone have changed these records, and would you know? Two words carry the answer.
Immutability: it cannot be changed
Immutable means "cannot be modified or deleted once written." In practice this is done with write-once storage. This is a storage location configured so that a file, once written, can be read as many times as you like. The file cannot be edited or removed until its retention period expires. You will see this described as "write once, read many", as a retention lock, or as an object lock, depending on the product. The mechanism varies; the idea is the same. Think of a bound ledger with numbered pages: you can add a new page, but tearing one out leaves an obvious gap.
You do not need exotic hardware. Many backup targets, cloud storage buckets, and even ordinary file servers can enforce write-once behavior on a folder. One important property is that the lock is enforced by the storage itself, not by good intentions. Another is that the retention period is set to match your compliance tier.
Integrity: you can prove it has not changed
Integrity means the record you are reading is the record that was written. The tool for proving it is a hash, also called a checksum: a short fingerprint computed from a file's contents. Change a single byte of the file and the fingerprint changes completely. Compute the hash when the file is archived, store it separately, and at any later date you can recompute the hash and compare. If they match, the file is unaltered. If they do not, something happened, and you know exactly which file to worry about.
Hashes are cheap, fast, and understood by every auditor. Nobody has ever called a meeting about a checksum. A folder of archived logs plus a manifest listing each file's hash is a perfectly respectable integrity mechanism for a small team.
Separation: the admin cannot quietly edit the evidence
The third idea does not have a fancy name but matters as much as the other two. If the same account that administers the firewall can also delete the firewall's archived logs, the archive proves very little. In that case, the person most able to cause a problem is also the person most able to erase the record of it. Auditors call the fix separation of duties, and for a small team it means one practical thing. The credentials that can write to or unlock the compliance tier are different from the everyday admin credentials. They are held by a different person where possible and are used rarely and loudly.
A department of one cannot literally hand the key to someone else, but can still get most of the benefit. It can use a separate, audited account for the archive, with the write-once lock enforced by the storage rather than by the account. It can also use a hash manifest stored somewhere the everyday account cannot reach. The point is not to be suspicious of yourself. It is to make the answer to "could you have altered this?" a demonstrable "not without leaving a trace."
Proving nothing was altered
Immutability and integrity are properties. Auditors want evidence of those properties, which means records you can hand over. Two concepts from the investigative world translate well.
Chain of custody is simply the record of what happened to the evidence. The record shows when the evidence was collected, where it was stored, who had access, and any time it was copied or moved. For archived logs, chain of custody is mostly automatic if you keep three things. These include the archive job's own log (showing each file was written on a date) and the access list for the archive location. They also include a note whenever something is exported for an auditor or an investigation.
Integrity reports are the operational version of the hash idea. A scheduled task walks the archive, recomputes every file's hash, compares it to the manifest, and writes a short report. The report says how many files were checked and how many matched. It identifies any that did not. Run monthly, kept alongside the archive, that report is exactly the kind of repeated, dated, machine-generated evidence auditors trust. It answers "how do you know nothing was altered?" with "here is the monthly check that would have told us."
Here is what a minimal manifest looks like in practice. On a Windows archive server, a scheduled script can hash a month's worth of archived files. It can write the results to a manifest stored on a separate volume:
Get-ChildItem D:\archive\firewall\ -Filter *.gz |
Get-FileHash -Algorithm SHA256 |
Select-Object Path, Hash |
Export-Csv E:\manifests\firewall-manifest.csv -NoTypeInformation
# Later, to verify: recompute and compare against the stored manifest
$stored = Import-Csv E:\manifests\firewall-manifest.csv
foreach ($entry in $stored) {
$now = (Get-FileHash $entry.Path -Algorithm SHA256).Hash
if ($now -ne $entry.Hash) { "MISMATCH: $($entry.Path)" }
}
That is the whole mechanism. Notice the manifest lives on a different volume (E:) from the archive (D:), ideally with different permissions. A tamperer who can reach the archive but not the manifest cannot cover their tracks. A tamperer who can reach both would need to alter two locations in a consistent way, which is much harder to do quietly. Writing the verification output to a dated report file each month completes the picture.
The cost conversation
Retention costs storage, and somebody will ask how much. The good news is that a tiered design makes the number small and easy to justify. The bad news is that you have to work it out rather than guess. A simple worksheet does it:
For each source in the compliance tier: daily volume (MB/day) x retention (days) = raw size (MB) raw size x compression factor = stored size (MB) Example, one firewall, deny and admin events only: 40 MB/day x [your required days] = raw raw x 0.15 (text compresses well) = stored Sum the stored sizes, add a safety margin, and that is the archive.
Measure daily volume rather than estimating it. Yes, even for the firewall you are sure about. Most collectors show bytes received per source, and a week of readings gives a reliable average. Text logs compress dramatically, so the stored figure is usually a small fraction of the raw one. And because the compliance tier excludes bulk allow logs and debug output, the total is often far smaller than the person asking expects.
When you make the case for the storage, frame it in the language of the people approving it. A few arguments that land well:
- It is a requirement, not a preference. Point to the specific clause in your framework or contract. Retention shortfalls are among the easiest audit findings for an auditor to write and among the most awkward to explain.
- The tiering is the saving. Show the "keep everything" number next to the tiered number. The difference is money you are not spending.
- It is also an incident tool. The same archive that satisfies the auditor is what lets you answer "what did that account do last quarter?" during a real investigation.
- Write-once storage is cheap. It is a configuration setting on storage you may already own, not a new product line.
Where cost genuinely bites is bulk traffic logs, if your requirements include them. There, one lever is filtering at the source (log denies and admin events, not every allow). Another is summarizing rather than storing raw where the requirement permits. Another is choosing the cheapest storage class that still supports a write-once lock. Be honest in the conversation about which of those your framework allows. It is better to ask the question than to discover the answer during the audit.
A retention policy template
The auditor's first retention question is "show me the policy." A policy does not need to be a long document. One table, one page of explanation, a named owner, and no binder. Binders are where policies go to be unread. Fill in the durations from your own requirements; the placeholders are deliberate.
| Source | Tier | Duration | Storage | Owner |
|---|---|---|---|---|
| Directory authentication and admin events | Compliance | [per framework] | Write-once archive, hashed manifest | IT lead |
| VPN and remote access sessions | Compliance | [per framework] | Write-once archive, hashed manifest | IT lead |
| Firewall admin logins and rule changes | Compliance | [per framework] | Write-once archive, hashed manifest | Network admin |
| Firewall deny and security events | Compliance | [per framework] | Write-once archive, hashed manifest | Network admin |
| Switch and router config changes and logins | Compliance | [per framework] | Write-once archive, hashed manifest | Network admin |
| Monitoring alerts and acknowledgements | Compliance | [per framework] | Monitoring database plus monthly export to archive | Monitoring owner |
| Backup job results and restore tests | Compliance | [per framework] | Backup console plus monthly export to archive | Backup owner |
| Firewall allow (bulk) | Investigation | [months, per policy] | Compressed on collector | Network admin |
| Server and application logs | Investigation | [months, per policy] | Compressed on collector | Server admin |
| Debug, verbose, wireless chatter | Troubleshooting | [weeks, per policy] | Collector, fast tier | IT team |
| Performance metrics (raw) | Troubleshooting | [weeks, per policy] | Monitoring database | Monitoring owner |
The page of explanation that goes with the table should answer these questions in order. What do the tiers mean? Where is each tier stored and who can write to it? How is integrity checked and how often? What happens when a duration expires (deletion is automatic and logged)? How are exceptions handled, such as a legal hold that pauses deletion for a specific investigation? Sign it, date it, and review it once a year or whenever the framework or the network changes. The review itself is evidence.
A last word on honesty. If you are starting today, your compliance tier begins today. Do not pretend otherwise. Write the start date into the policy, keep an unbroken record from that day, and tell the auditor plainly. A clearly dated beginning is a functioning control. A vague claim of long retention that cannot be demonstrated is a finding.
Where to go next
Retention done well is small, deliberate, and provable. It has three tiers narrowing as they lengthen and a compliance tier limited to the sources your framework names. It has write-once storage with a hashed manifest and a one-page policy with an owner. It is less storage than "keep everything forever" and considerably more evidence. Once the archive exists, the next question is how to turn its contents — and the rest of your monitoring output — into an evidence pack the auditor can read. The evidence packs guide picks up there. The full set of guides is in Monitoring for Compliance & Audit Evidence.
Put this into practice with SysPeep Monitor
Retention itself lives in your log collector and archive. But the compliance tier also needs the monitoring side of the story: alert history, acknowledgements, and availability reports. SysPeep Monitor keeps that history in a database backend you choose (SQLite, MySQL/MariaDB, or SQL Server). It generates reports from a single web console. So the monitoring half of the compliance tier is there when the archive needs it. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep