Home › Topics › Firewall Log Analysis Fundamentals › Firewall Log Tooling

Firewall Log Tooling for Small Teams


The other guides in this pillar keep saying "aggregate the denies by source" and "build a dashboard for internal denies" as if that were a small thing. For a large security team with a dedicated platform, it is. For a two-person IT department, the honest question is: with what? The firewall's own log viewer scrolls, and the budget for a big log platform does not exist. And the weekly review will not happen if it starts with an hour of manual filtering.


The good news is that useful firewall log analysis has a very low floor. This guide lays out a ladder of tooling, from plain text files and shell one-liners up to the point where a SIEM-style approach begins. It covers the effort and payoff of each rung. You will see how to collect logs into one place, and how to produce the standard aggregations at every rung. You will see why extracting structured fields early saves you from a lifetime of regular expressions. You will see which dashboards are worth building, and how to make the weekly review arrive in your inbox before you have had coffee.


In this guide

  • Collection first: everything into one place
  • The tooling ladder
  • Rung one: plain text and shell aggregation
  • Rung two: a spreadsheet for the one-off
  • Rung three: a lightweight log platform
  • Parsing and field extraction
  • Dashboards worth building
  • Automating the weekly report
  • The ladder at a glance
  • Where to go next

Collection first: everything into one place


No tooling helps while the logs live only on the firewall. The device's own storage is small, and it rotates quickly. It is the first thing lost when the firewall is replaced or compromised. It can only be searched one firewall at a time. So the first rung of every ladder is the same. Send every firewall's logs over syslog (the standard protocol for shipping log lines across a network) to a central collector. And keep them there according to the retention plan in what to keep, review, and ignore. Building that collector is its own subject, covered step by step in building a central syslog server. Here, three details matter for firewall logs specifically.


  • One file (or index) per firewall per day. Everything downstream is easier when a day's worth of one device is a single unit you can measure, search, archive, and delete.
  • Synchronized clocks. The firewall, the collector, and everything you will compare them against must agree on the time, or your cross-device timelines are fiction.
  • Count what arrives. Lines received per firewall per hour is the collector's own health check. A firewall that goes silent looks identical to a firewall with nothing to say. And the count is the only way to tell them apart.

The tooling ladder


Think of tooling as a ladder with four rungs. You climb when the rung below stops fitting, not because a higher rung sounds more professional.


  • Rung one: plain text and the shell. Logs as files; searching and aggregating with standard command-line tools. Zero cost, surprisingly capable, and enough for a small network's weekly review.
  • Rung two: a spreadsheet. A filtered export, pivoted and charted. Ideal for one investigation or one presentation; wrong for anything repeated.
  • Rung three: a lightweight log platform. A tool that ingests logs, parses them into fields, indexes them, and offers search, saved queries, dashboards, and scheduled reports. This is where most small teams should aim to live.
  • Rung four: SIEM-style correlation. Suppose you need firewall logs joined with authentication, endpoint, and server logs, with rules that fire on patterns across all of them. Then you have left log analysis and entered security monitoring. The SIEM-lite guides cover what that looks like at small-team scale.

The rest of this guide climbs rungs one through three, showing the same aggregations at each so you can see what climbing actually buys you.


Rung one: plain text and shell aggregation


If your collector writes each firewall's lines to files, you already have a working analysis system. That system consists of the standard text tools of any Linux or Unix-like host (or WSL on Windows). The pattern is always the same — filter the lines you want, extract one field, count the distinct values, sort by count. The sketches below assume the generic key=value format used throughout this pillar. Your firewall's field names will differ, so adjust the patterns once and save them.


# Top denied external sources, past week (files day-08 .. day-14)
cat /var/log/firewall/day-{08..14}/fw-edge-01.log \
  | grep 'action=deny' | grep 'in=wan' \
  | sed -n 's/.*src=\([0-9.]*\).*/\1/p' \
  | sort | uniq -c | sort -rn | head -20

# Top denied destination ports on the outside interface
cat /var/log/firewall/day-{08..14}/fw-edge-01.log \
  | grep 'action=deny' | grep 'in=wan' \
  | sed -n 's/.*dpt=\([0-9]*\).*/\1/p' \
  | sort | uniq -c | sort -rn | head -20

Each line of output is a count followed by a value: the ranked list the deny-log mining guide works from. The internal-hosts aggregation needs two fields at once, which is where awk earns its keep. This sketch splits each key=value pair, then prints source, destination, and port for denies whose source is a private address:


# Internal hosts hitting denies, by source / destination / port
cat /var/log/firewall/day-{08..14}/fw-edge-01.log \
  | grep 'action=deny' \
  | grep -E 'src=(10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.)' \
  | awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); f[kv[1]] = kv[2] }
           print f["src"], f["dst"], f["dpt"] }' \
  | sort | uniq -c | sort -rn | head -40

# Denies per hour (timestamp is field 3, HH:MM:SS)
cat /var/log/firewall/day-14/fw-edge-01.log \
  | grep 'action=deny' | awk '{ print substr($3, 1, 2) }' \
  | sort | uniq -c

# Hits on rules that should never fire
grep -hE 'rule=(dmz-to-lan-deny|guest-to-servers-deny)' /var/log/firewall/day-{08..14}/*.log

That is the whole weekly review in five commands. Their limits are real but arrive later than you would expect. Searching a month of a busy firewall takes minutes rather than seconds. Multi-field questions get awkward. Every new question means a new pattern, and the patterns break the day the log format changes. Nobody but you can run them. But a small network with one edge firewall can run a genuine, useful review at this rung for a long time. And the money you did not spend is real.


Tip: Put the working commands in a script the day they work, with a comment above each saying which question it answers. The script is your first "saved search." It is the seed of the automated report at the end of this guide. And it is the thing you hand to a colleague when you go on holiday.


Rung two: a spreadsheet for the one-off


Some questions are asked once. For example: "show me everything this workstation talked to last Tuesday," or "give the manager a chart of blocked attempts by port for the quarter." For those, export a filtered slice as comma-separated values and open it in a spreadsheet. The shell can produce the export, and most log platforms have an export button. A pivot table is the spreadsheet feature that groups rows by one column and counts or sums another. It gives you the same aggregations as the shell with clicking instead of typing, plus instant charts. It also lets you hand the file to someone who will never open a terminal.


# Export one host's denies as CSV: time, destination, port, rule
grep 'action=deny' /var/log/firewall/day-14/fw-edge-01.log \
  | grep 'src=10.30.1.22 ' \
  | awk 'BEGIN { print "time,dst,dpt,rule" }
         { for (i = 1; i <= NF; i++) { split($i, kv, "="); f[kv[1]] = kv[2] }
           print $3 "," f["dst"] "," f["dpt"] "," f["rule"] }' > host-10.30.1.22.csv

The spreadsheet's weakness is that it does not remember. Next week you export again, pivot again, chart again. It also chokes on large exports and quietly mangles anything that looks like a date. So it is the right rung for a single investigation, a report for a non-technical audience, or exploring a new question before you automate it. It is the wrong rung for anything on the calendar.


Rung three: a lightweight log platform


A log platform is software that receives log lines, often directly over syslog, so it can be your collector. It parses each line into named fields and stores them in an indexed form that makes searching fast. It provides a web interface for search, saved queries, dashboards, alerts, and scheduled reports. Several capable ones are free or inexpensive at small-team scale and run comfortably on a modest virtual machine. This rung is where the weekly review stops being a chore, because the five commands above become five saved searches on one dashboard that refreshes itself.


When choosing one, judge it on the things that matter for firewall logs rather than on feature lists:


  • Field extraction for your firewall. Does it parse your device's format into fields out of the box, or can you teach it to in an afternoon? This one property determines how pleasant every later day will be.
  • Retention controls per category. Can you keep denies longer than allows, and summaries longer than either? If retention is one number for everything, your tiers become wishful thinking.
  • Saved searches and dashboards that a colleague can use without learning the query language.
  • Scheduled reports that run a saved search and email the result, so the weekly review comes to you.
  • Honest resource needs. Indexing costs CPU, memory, and disk. Measure a day of your logs (the method is in the retention guide) and confirm the platform's footprint fits the hardware you actually have.

What a platform does not change is the analysis. The aggregations, the three bins, the verdicts, the notes: all identical. The platform makes them faster, shareable, and repeatable, which is enough to justify it, but it will not think for you.


Parsing and field extraction


Every rung above has a hidden dependency: turning a line of text into fields. At rung one you did it with sed patterns, one per field, per question, rewritten each time. That works, and it scales badly for a specific reason. Regular expressions written against a text format encode assumptions about the format — field order, separators, quoting. Those assumptions break silently when a firmware update rearranges the line. They also break silently when a second firewall from a different vendor joins the collector, or when a field appears that your pattern did not expect. The failure is a wrong count, not an error, and you will not notice until a review is quietly missing half its data.


Structured fields fix this by parsing each line once, at ingestion, into named values. Those are src, dst, dpt, action, rule. The values are stored, not the text. Every later question then works against names rather than patterns. Change the format and you change one parser, not fifty searches. Add a firewall from another vendor and you write one more parser that emits the same field names. Then every dashboard covers it automatically. The general craft of writing parsers for network devices is the subject of parsing device logs. For firewall logs, the practical advice is short:


  • Normalize names across devices. Decide once that the source address is called src everywhere, whatever each vendor calls it. Map every firewall to that vocabulary at ingestion.
  • Keep the raw line too. Parsers have bugs. Storing the original text beside the fields means you can re-parse later without losing anything.
  • Parse the rule name and the zone pair. These are the fields the other guides in this pillar lean on hardest, and they are the ones default parsers most often skip.
  • Watch for the unparsed. Count lines that failed to parse, per firewall, and put that count on the dashboard. A jump means the format changed.

If you are at rung one and expect to stay there a while, you can still borrow the idea. In that case, write one script that converts raw lines to a clean, consistent, comma-separated form with your chosen field names. Point all the other scripts at its output. You have built a parser; it just runs from the shell.


Dashboards worth building


Dashboards are where log platforms tempt you into building thirty panels nobody looks at. Resist. The point of a dashboard is to make the daily five-minute glance and the weekly review faster than the alternative. Six panels cover both:


  • Denies over time, per hour, for the past two weeks, with each firewall as a separate line. This is your baseline and your burst detector in one picture.
  • Top denied external sources, with distinct ports and days seen as extra columns. A table, not a chart; the columns are the point.
  • Top denied destination ports on the outside interface, for this week beside last week, so the change is visible without arithmetic.
  • Internal hosts hitting denies, by source, destination, and port. The most important panel and the shortest. If it is long, something is wrong or the rulebase needs work.
  • Administrative events: logins to the firewall (success and failure, with source address) and every rule or configuration change, as a chronological list. Should match what you know happened.
  • Lines received per firewall, per hour, plus unparsed lines. A zero here is the most urgent thing the dashboard can show.

Build the daily-glance version first: six panels on one screen with a two-day window. The weekly-review version is the same panels with a seven-day window and longer tables. Add a panel only when a review keeps asking a question the existing panels cannot answer. Remove any panel nobody has looked at for a month.


Automating the weekly report


A review that requires someone to open a dashboard will be skipped on busy weeks. A review that arrives by email on Monday morning, already containing the aggregations, is much harder to ignore. At rung three, this is a scheduled report attached to your saved searches. At rung one, it is the script you wrote earlier, run by the scheduler, with its output mailed. Here is the shape:


#!/bin/sh
# weekly-fw-report.sh - run early every Monday; emails the standard aggregations
LOGS=/var/log/firewall
DAYS=$(ls -d $LOGS/day-* | tail -7)
OUT=/var/tmp/fw-weekly-report.txt

{
  echo "== Denies received per firewall, per day =="
  for d in $DAYS; do echo "$d"; grep -c 'action=deny' $d/*.log; done

  echo; echo "== Top denied external sources (count, source) =="
  cat $DAYS/*.log | grep 'action=deny' | grep 'in=wan' \
    | sed -n 's/.*src=\([0-9.]*\).*/\1/p' | sort | uniq -c | sort -rn | head -20

  echo; echo "== Top denied destination ports (count, port) =="
  cat $DAYS/*.log | grep 'action=deny' | grep 'in=wan' \
    | sed -n 's/.*dpt=\([0-9]*\).*/\1/p' | sort | uniq -c | sort -rn | head -20

  echo; echo "== Internal hosts hitting denies (count, src, dst, port) =="
  cat $DAYS/*.log | grep 'action=deny' \
    | grep -E 'src=(10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.)' \
    | awk '{ for (i = 1; i <= NF; i++) { split($i, kv, "="); f[kv[1]] = kv[2] }
             print f["src"], f["dst"], f["dpt"] }' | sort | uniq -c | sort -rn | head -40

  echo; echo "== Hits on never-fire rules =="
  grep -hE 'rule=(dmz-to-lan-deny|guest-to-servers-deny)' $DAYS/*.log | head -50

  echo; echo "== Admin logins and configuration changes =="
  grep -hE 'event=(admin-login|config-change)' $DAYS/*.log
} > "$OUT"

mail -s "Firewall weekly review" it-team@company.example < "$OUT"

Schedule it with the system's scheduler so it runs before anyone arrives. A line such as 30 6 * * 1 /usr/local/bin/weekly-fw-report.sh in a crontab runs it at half past six every Monday. Then treat the email as the agenda for the review: read the sections top to bottom, write the verdicts and notes, and file it. Once the report exists, the review has a fixed cost of thirty minutes. It survives the week the helpdesk queue explodes, because the hard part already happened while you were asleep.


Two refinements are worth adding once the basic report runs. First, include last week's numbers beside this week's for the totals and top lists, so change is visible at a glance. Second, keep the reports. A folder of Monday emails is a crude but real archive of your environment's normal. It is exactly what you will want when someone asks "when did that source first show up?"


The ladder at a glance


RungEffort to set upEffort per reviewWhat you gainMove on when
Plain text and shellAn afternoon: collector, files, five commands in a scriptLow once scripted; high for any new questionThe full weekly review at zero cost; total understanding of your dataSearches take minutes, formats keep changing, or a colleague needs to run it
SpreadsheetMinutes per exportHigh and repeated every timeCharts and pivots for a one-off; a file for non-technical readersImmediately; it is a side tool, not a rung to live on
Lightweight log platformA few days: install, parsers, retention tiers, six dashboards, one scheduled reportLow; the report arrivesFast search over long windows, structured fields, dashboards anyone can use, scheduled reportsYou need firewall logs joined with authentication, endpoint, and server logs, and rules across all of them
SIEM-style correlationWeeks, plus ongoing tuningDepends on the rules you writeCross-source detection; alerts on patterns no single log showsTop of the ladder; go here only with a person to own it

One reading of the table is worth stating plainly: most small teams belong at rung three. But consider a team that stops at rung one with a scheduled script and a weekly habit. That team is doing more real analysis than many organizations with expensive platforms and no review.


Where to go next


Tooling for firewall logs is a ladder, not a purchase. Collect everything into one place. Start with text files and five shell commands. Use a spreadsheet for the one-off. Climb to a lightweight log platform when search gets slow or a colleague needs in. Parse into structured fields as early as you can. Build six dashboards and no more. And schedule the weekly report so the review comes to you. Each rung produces the same aggregations, which means the analysis skills you build at the bottom carry all the way up.


Those skills are the rest of this pillar. What a firewall log contains defines the fields your parsers extract. The guide to what to keep, review, and ignore sets the retention tiers and the cadence your report serves. And deny-log mining is what you do with the report once it arrives. The guide to closing the loop turns findings into a safer rulebase. All of it lives under Firewall Log Analysis Fundamentals.


Put this into practice with SysPeep Monitor

Log tooling tells you what the firewall decided. It goes quiet when the firewall, the collector, or the link between them is struggling. SysPeep Monitor watches those pieces from one web console. It watches firewall CPU, memory, and interface throughput over SNMP, and the collector's disk and services over WMI. It alerts you before a full disk or a saturated link leaves a hole in Monday's report. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep