Flow Logs: Your Cloud's Version of NetFlow
On-prem, when you need to know what is actually crossing the network, you have options you barely think about. Mirror a switch port and capture packets. Turn on NetFlow at the router. Read the firewall's deny log. In the cloud, all three reflexes reach for equipment that is not there. There is no switch to mirror and no router of yours to export flows. The "firewall" — the security group — drops traffic without writing a single line anywhere.
Flow logs are the cloud's answer, and they are a good one once you understand their shape. A flow log is the provider's record of connection summaries crossing your virtual network. It records who talked to whom, on which ports, and how much data moved. It also records whether the traffic was allowed or refused — the part with no on-prem equivalent. That last field turns hours of "why can't the app reach the database?" guesswork into a one-query answer.
This guide covers what flow logs record and how they differ from the flow data you may know from on-prem. It covers how to enable them without regretting the bill and how to actually read them. That includes a worked connectivity investigation, traffic accounting, and the silent-deny hunt that justifies the whole feature.
In this guide
- What a flow log actually records
- How cloud flow logs differ from NetFlow
- The killer use case: seeing the silent deny
- Turning them on without regret
- A worked investigation: app cannot reach database
- Traffic accounting: who talks to whom, and how much
- The honest cost note: visibility is metered
- Where to go next
What a flow log actually records
A flow log entry is a summary, not a recording. It describes a conversation without containing any of it — the envelope, never the letter. Each record covers traffic seen on one virtual network interface during a short window of time. And the core of every record is the 5-tuple: source address, destination address, source port, destination port, and protocol. Those five values are what makes one conversation distinct from another. Around the tuple, the record adds how much moved (packets and bytes) and when the window started and ended. It adds which interface saw it and the verdict: ACCEPT or REJECT.
Field names and ordering vary by provider, but a record always resembles this:
interface srcaddr dstaddr srcport dstport proto packets bytes action if-a7 10.0.1.25 10.0.2.14 49712 3306 6 18 4620 ACCEPT
Reading it left to right: the interface if-a7 saw a conversation from 10.0.1.25. The source used an ephemeral source port in the high range, meaning this side started the connection. The destination was 10.0.2.14 on port 3306, protocol 6 (TCP; UDP is 17, ICMP is 1). Eighteen packets and about four and a half kilobytes moved during the window, and the rules allowed it. One line, and you already know an app server opened a database connection and the security groups were happy about it. That is a remarkable amount of truth for eighty characters.
What is not in the record matters just as much: no payload, no query text, no URLs, no packet timing. Flow logs tell you that a conversation happened and how big it was — never what was said. For most network troubleshooting that is exactly enough, and it conveniently keeps the logs themselves from becoming a pile of sensitive data.
How cloud flow logs differ from NetFlow
If you have used NetFlow, sFlow, or IPFIX on-prem, flow logs will feel familiar — they are the same "summaries of conversations" idea. If flow data is new to you entirely, the guide on what flow data reveals covers the general craft. Here we focus on what changes in the cloud. The differences are worth knowing precisely, because two of them regularly surprise people mid-incident.
| Aspect | On-prem NetFlow-style data | Cloud flow logs |
|---|---|---|
| Where it is captured | At routers and switches you choose | Per interface, per subnet, or per VPC |
| Who did the blocking | Invisible — flows show traffic that passed; denies live in firewall logs | Visible — every record carries ACCEPT or REJECT |
| Timeliness | Near-real-time export | Delivered minutes after the fact |
| Time resolution | Flow start and end, often to the second | Aggregation windows — a record summarizes minutes |
| Escalating to packets | Mirror a port and capture | No default equivalent; mirroring is a separate paid feature where offered |
| Cost | Mostly your storage and CPU | Metered by the provider: volume in, storage, queries |
The two mid-incident surprises are the timing rows. First, aggregation windows: a flow log record is a rolled-up summary of an interval measured in minutes, not a per-packet timeline. Two short bursts inside one window can merge into one record; a connection's recorded "start" is really the start of a window. Do not try to reconstruct split-second sequences from flow logs; they answer "what happened around then," not "what happened at that exact instant."
Second, delivery lag: records show up minutes after the traffic they describe. During a live incident this feels like an eternity. The practical adjustment is to start the flow-log query early, keep working the problem with other evidence, and come back to the log. Do that rather than stare at a query that cannot yet contain the thing you just did.
One more structural difference: capture points. On-prem, flows are captured where you enabled them, usually the core router, and each conversation is exported once. Cloud flow logs are captured per interface, so a conversation between two instances inside your VPC appears twice — once from each side's interface. That doubling is a gift for troubleshooting (you can see both ends' verdicts, as the investigation below shows). It is a trap for accounting (naively summing bytes counts internal traffic twice).
The killer use case: seeing the silent deny
Here is the scenario that makes flow logs indispensable. On-prem, your firewall logged its denies; "check the firewall log" was step one of every connectivity ticket, and the blocked connection announced itself. Security groups and network ACLs do not log. When a rule drops traffic, nothing appears anywhere you can see by default: no counter, no syslog line, no console event. The connection just times out, and you are left interrogating an application team about whether they are sure the service is listening.
The flow log's REJECT record is the only witness to that drop. A record like this one is a security-group deny caught in the act:
interface srcaddr dstaddr srcport dstport proto packets bytes action if-d2 10.0.1.25 10.0.2.14 50104 8443 6 3 180 REJECT
Someone at 10.0.1.25 tried three times to open port 8443 on the database host, and a rule refused it. To be honest about the limits: the record says a rule said no, but not which rule. It could be the security group on the destination, the source's outbound rules, or a network ACL in between. You still have a short hunt ahead. But the question just shrank from "is it the network, the app, DNS, or the phase of the moon?" to "which of three rule sets is missing an entry for 8443?" That shrinkage, repeated across every future ticket, is the feature paying rent.
REJECT records earn their keep in security work too. A steady drizzle of rejects against unusual ports from unfamiliar addresses is your internet-facing subnets being scanned (normal, if unlovely). But rejects originating from your own instances toward odd destinations are worth a real look. That can be misconfigured software, or something on the instance trying to phone somewhere it should not.
Turning them on without regret
Enabling flow logs is a short checklist of decisions, and each one is easier to make deliberately now than to unpick later:
- 1. Pick the scope. A flow log can watch one interface, one subnet, or the whole VPC. Start with the whole VPC: complete coverage, one thing to manage, and every future investigation already has data. Add a per-interface log later when you need a surgical view of one busy resource.
- 2. Pick the destination. Providers offer roughly two kinds of home for the records. One is the logging service (queryable quickly, costs more per byte kept). The other is object storage (cheap and durable, but analysis needs a query tool over files). A common pattern is both — recent data queryable, older data archived cheap.
- 3. Pick what to capture. Many providers let a flow log record everything, only accepted traffic, or only rejected traffic. Everything is the right default. A rejects-only log is a nice cheap tripwire for security, but it cannot do accounting or troubleshooting, because the healthy traffic is invisible.
- 4. Pick retention. Troubleshooting rarely needs more than a few weeks hot; accounting and security reviews like having months in the cold archive. Decide, write it down, and set lifecycle rules so the decision enforces itself.
- 5. Name things. A flow log named
flowlog-vpc-prod-allwill be findable during an incident. The default generated name will not. Future-you is on the incident bridge; be kind.
Tip: enable flow logs before you need them. They start recording at the moment you turn them on and know nothing about yesterday. The worst time to discover this is the morning after the incident, staring at an empty query window. Flow logs on a quiet VPC cost very little and sit there accumulating exactly the history your next investigation will want.
A worked investigation: app cannot reach database
Time to use all of this. The ticket: the app server at 10.0.1.25 cannot reach the database at 10.0.2.14 on port 3306. Connections hang and time out. On-prem you would check the firewall log. Here, you query the flow logs for both sides of the conversation. Remember, each interface logs its own view, so a healthy conversation leaves records on the app's interface and the database's interface.
Filter for the tuple on both interfaces and compare what you find. The possible shapes, and what each one means:
Case A: rules problem at the destination if-app (10.0.1.25): 10.0.1.25 -> 10.0.2.14:3306 ACCEPT left the app fine if-db (10.0.2.14): 10.0.1.25 -> 10.0.2.14:3306 REJECT refused at the door Case B: routing or path problem if-app (10.0.1.25): 10.0.1.25 -> 10.0.2.14:3306 ACCEPT left the app fine if-db (10.0.2.14): (no record of this tuple at all) never arrived Case C: not the network if-app (10.0.1.25): 10.0.1.25 -> 10.0.2.14:3306 ACCEPT if-db (10.0.2.14): 10.0.1.25 -> 10.0.2.14:3306 ACCEPT delivered and allowed
Case A is the classic silent deny: the traffic left the app, arrived at the database's interface, and a rule refused it. The fix lives in the database's inbound security group rules, the app's outbound rules, or a network ACL between the subnets. In practice, check the destination's inbound rules first, because that is where the missing entry usually is.
Case B is subtler and is the one on-prem instincts misread. The app's side shows accepted outbound traffic — the rules were fine with it — but the far interface never saw it. The packets were allowed and then had nowhere to go. Possible causes include a route table missing the destination subnet or a peering connection between VPCs that was never linked into the routes. Or traffic matched a more specific wrong route and went somewhere else entirely. Accepted-but-never-arrived means stop reading rules and start reading route tables.
Case C clears the network. Traffic arrived, rules allowed it in both directions, bytes moved — and the app still fails. Now it is a listening-service problem, a database-permissions problem, or an application bug. Hand the evidence over: two ACCEPT records are a wonderfully unarguable way to end the "it must be the network" phase of an incident.
And if the app's own interface shows a REJECT on the outbound side, the app server's own security group or subnet ACL is blocking the departure. That is rarer, but it happens after well-meaning "tightening." If neither interface has any record of the tuple at all, the traffic was never sent. Possible causes are a wrong destination address in the app's config, a DNS name resolving somewhere unexpected, or the app not actually attempting the connection. Flow logs cannot record what never touched the network — and even that silence tells you where to look next.
Traffic accounting: who talks to whom, and how much
Beyond troubleshooting, flow logs are a ledger. Summing bytes by source, destination, and port answers questions you used to answer with NetFlow. Which subnets exchange the most data? Which services are the top talkers? Which instance suddenly started moving ten times its usual volume? And what actually depends on that server everyone forgot about? Group flows by destination port and you get a service-level map of the VPC. It shows who uses the database, who calls the internal API, and what still talks to the thing scheduled for decommissioning.
A concrete shape for this: once a week, run a summary of outbound bytes grouped by source, and make yourself explain the top entries. The first run is always educational.
Top sources by bytes to destinations outside 10.0.0.0/16, one day srcaddr bytes (approx) what it turned out to be 10.0.2.31 290 GB backup agent pushing nightly fulls off-cloud 10.0.1.25 31 GB app server calling an external payment API 10.0.1.44 12 GB OS updates — normal patch-day noise 10.0.2.9 8 GB log shipper forwarding to an outside service
Nobody had decided that the backup agent should push full copies out of the cloud every night; it was a default setting doing what defaults do. Flow logs did not fix it — they just made it impossible not to see, which is most of what monitoring is for.
Two accounting cautions. First, there is the double-count from earlier: internal conversations appear once per interface. So deduplicate (or sum only one side, or count each interface separately as "traffic seen by this resource") before quoting totals. Second, direction matters more in the cloud than it ever did on-prem, because traffic leaving is what shows up on the bill. Summing flows whose destinations fall outside your own address ranges, grouped by source, is exactly how you find out which workload is responsible for the egress line item. That is a big enough subject that watching egress gets its own guide in this pillar.
The honest cost note: visibility is metered
Here is the part the enthusiastic tutorials skip: flow logging itself costs money. The pattern is worth understanding even though the specific rates vary by provider and change over time. You generally pay along three dimensions. These are the volume of log data generated as it is ingested, the storage of whatever you retain, and sometimes the queries you run against it. The bill scales with how chatty your network is, not with how valuable the answers are. A quiet VPC generates trivial log volume; a busy microservices platform chattering internally can generate flow logs whose cost you will notice on the invoice.
Cost control is a set of deliberate scoping choices, not an afterthought. Scope captures to what you will actually use: a whole-VPC log at coarse settings for baseline coverage, per-interface logs only where you need detail. Prefer longer aggregation windows where the provider offers the choice — fewer, fatter records for the same traffic. Use sampling if it is available and your goal is accounting rather than forensics; a representative fraction of flows draws the same traffic map for less. Send hot data to the queryable service with a short retention, and archive the rest to cheap object storage. And put a size review on the calendar: flow log volume grows as quietly as the traffic it describes. And a monthly glance at "how big were the logs?" keeps the visibility bill a line item instead of a surprise. None of this is a reason to skip flow logs; it is the reason to enable them on purpose instead of on autopilot.
Where to go next
Flow logs are the traffic truth of your cloud network: the 5-tuple ledger of what moved, plus the ACCEPT/REJECT verdict that no on-prem flow format ever gave you. Enable them before the incident, learn the two-interface reading trick, and they will convert connectivity mysteries into short queries. They pair naturally with the rest of the Cloud Network Monitoring pillar. The guide the plain-language VPC tour explains the routes and rules the verdicts point at. And the provider metrics guide covers the gauges that tell you how the fabric itself is feeling.
Put this into practice with SysPeep Monitor
Flow logs tell you what moved inside the cloud — but half of every cloud conversation starts on the network your users sit on. SysPeep Monitor watches that half. It auto-discovers your switches, firewalls, and servers. It collects health and traffic metrics over SNMP and WMI, and raises alerts from one console. So the on-prem end of the story is never the blind spot. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep