Metadata Analysis: Fingerprints, Handshakes, and Flow Shapes
Once you accept that encryption locks the letter and leaves the envelope readable, the next question is practical. What do you actually do with the envelope? Staring at addresses and byte counts is not analysis. Analysis is knowing which patterns in those fields mean something, how to spot them, and — the part most guides skip — how often they will fool you.
This guide collects the techniques that work on encrypted traffic without decrypting a single byte. Some are about the handshake, the short and partly readable negotiation at the start of every TLS connection. Some are about the certificate the server presents. Some are about the shape of the whole conversation: how big, how long, how regular. And one is about noticing when a protocol turns up on a port where it does not belong.
None of them requires special hardware. Most can be done with the flow exporter already inside your firewall and a packet capture tool on a laptop. By the end you will know what each technique reveals, what it cannot tell you, and the innocent situations that trigger it anyway.
In this guide
- Why metadata carries meaning
- Handshake fingerprinting: the client's accent
- Certificate observation: reading the identity card
- Server-name observation
- Flow shapes: beacons, bulk uploads, and everything between
- Protocol on the wrong port
- The tools, generically
- Technique, reveal, and false positive: the table
- A first week of metadata analysis
- Where to go next
Why metadata carries meaning
If you have not yet read what encryption hides and what it still shows, start there. It lays out the visible fields one by one. This guide assumes you know that addresses, ports, sizes, timing, the server name, the certificate, and the handshake's shape all stay readable.
The reason those fields carry meaning comes down to one observation: software is consistent. A program is written once and behaves the same way every time it runs. The same browser produces the same handshake offer on every connection. The same background agent checks in at the same interval. The same service presents the same certificate to everyone. Humans are irregular; machines are not. Metadata analysis is the practice of learning your network's regularities and noticing when something breaks them. You are not looking for evil; you are looking for different, then deciding whether different matters.
Handshake fingerprinting: the client's accent
When a client opens a TLS connection, its first message is a proposal. The proposal says: here are the encryption methods I support, in my order of preference. Here are the extra features I understand. Here are the key-exchange options I can use. The server picks from that menu. The proposal is sent before any encryption exists. So it is readable on the wire, and here is the useful part: every client library writes its proposal differently. One browser lists its options in one order. A different browser uses another order. A scripting language's networking library uses a third. An embedded device's tiny TLS stack offers a short list nobody else offers.
Think of it as an accent. Two people can say the same sentence, and you can still tell one grew up somewhere else. The handshake proposal is the accent of a piece of software, and the combination of its choices forms a fingerprint. Tools that record handshake metadata reduce that combination to a short identifier so that identical proposals get identical labels. You do not need to understand the options themselves to use the labels; you need only notice which labels your network normally produces.
What fingerprinting reveals, in order of usefulness:
- Browser or not-a-browser. The most valuable distinction. A connection to a web service from a workstation is normally made by a browser, and browsers have well-known accents. A connection to the same service with a scripting-library accent means a program, not a person, made it. Sometimes it is an update agent or a sync client. Sometimes it is something nobody installed on purpose.
- New accent on the block. Your workstations collectively produce a small number of fingerprints. When a single host starts producing one that has never been seen anywhere on the network, that host has new software on it.
- Odd accent for the device type. A printer might suddenly speak with a desktop browser's accent. Or a server might start producing a fingerprint associated with a scripting library it has no business running.
And what it cannot tell you: a fingerprint is not an identity. Every program built on the same library shares the same accent, so a label says "this family of software", never "this exact program". Fingerprints also change when software updates, so a fleet-wide browser update produces a burst of new labels on the same day. Expect that noise. Treat the fingerprint as one attribute of a connection, combined with destination, timing, and certificate, rather than as a verdict on its own.
Certificate observation: reading the identity card
A certificate is an identity card for a server. It says who the server claims to be (the subject, including the list of names it covers). It says who vouches for that claim (the issuer), and for what period the card is valid. Observing certificates passively means noting the card each server presents during handshakes you can see. It gives you several checks that need no decryption at all.
Self-signed certificates
A self-signed certificate is one where the issuer and the subject are the same: the server vouches for itself. That is normal for lab gear, appliances fresh out of the box, and internal test systems. It is not normal for a production workstation to accept one from a server on the internet, because browsers would refuse it. That means whatever accepted it was not a browser behaving normally. A self-signed certificate on an outbound connection from a non-lab host deserves a look.
Expired and not-yet-valid certificates
Certificates carry not-before and not-after dates. An expired certificate on your own service is an operations problem you want to hear about before users do. An expired certificate on an external destination that your hosts keep connecting to anyway is stranger. Normal clients refuse expired cards, so persistent connections through one suggest software that does not check.
Name mismatch
The client asked for one name in its handshake; the certificate covers different names. Sometimes this is a misconfigured web server or a shared hosting quirk. Sometimes it is a service presenting a card that was never meant for that name. Mismatch plus a non-browser fingerprint is a strong pairing.
Unknown issuers and first-seen certificates
Over a few weeks your network settles into a set of issuers it sees constantly: a handful of public authorities and your own internal one. An issuer outside that set is worth a glance. Stronger still is a first-seen event on a service you know. For example, the certificate presented by your internal mail server today is not the one it presented yesterday, and nobody renewed it. Passive observation gives you the change; a certificate inventory tells you whether the change was expected. The certificate inventory guide covers building that list.
Server-name observation
The server name in the handshake tells you which service a client asked for, in words rather than an address. Two simple habits get most of its value. First, keep a per-host list of names each machine contacts, and flag a name that no host has ever contacted before. Second, compare the handshake name with your DNS logs: the name a client looked up a moment earlier should match the name it then asked for. When it does not, or when no lookup preceded the connection at all, the client had the address some other way. That is unusual for ordinary software and worth noting.
A missing server name is a weaker signal than beginners expect. Many non-browser clients simply do not send one, and some legitimate services deliberately protect it. Treat "no name" as a reason to fall back on the certificate and the DNS lookup, not as an alarm by itself. The false positives here are mostly about scale: new cloud services get onboarded without anyone telling IT. And a single web page can trigger connections to dozens of names you have never heard of. Judge name novelty per host and over time, not as an alert for every new name across the whole company.
Flow shapes: beacons, bulk uploads, and everything between
A flow is one connection summarized: addresses, ports, bytes each direction, packet counts, start and end. Put several flows from the same host side by side and a shape appears, the way a silhouette identifies a bird you cannot see in detail. A few shapes cover nearly everything on a business network:
Shape Duration Bytes out Bytes in Repeats? Typical cause page load seconds small medium irregular browsing, web apps bulk upload minutes+ very large small once, or nightly backup, sync, exfiltration bulk download minutes+ small very large once updates, installers, media interactive long small small steady trickle remote desktop, shell, chat streaming long tiny large steady video, audio beacon seconds tiny tiny fixed interval agents, updaters, implants
Two of those shapes matter most for security. A beacon looks like a series of small, regular envelopes to the same destination. The pattern is covered in depth in the beaconing patterns guide. So here it is enough to say that the giveaway is the regularity of the interval and the sameness of the sizes. It is not the destination. A bulk upload looks like one long, fat flow with almost everything going outward. Backups and cloud sync clients produce exactly that shape on purpose. So the question is never "is there an upload?" but "is there an upload to a destination and at a time I did not expect?".
Reading shapes takes three numbers per flow. They are the ratio of bytes out to bytes in, the duration, and, for repeated flows to the same destination, the interval between them. The in-versus-out ratio separates uploads from downloads and page loads. Duration separates transfers and interactive sessions from quick fetches. Interval regularity separates beacons from human-driven traffic, because people do not click at exactly the same second every five minutes. None of this requires the tool to understand the application; it requires you to know what shapes each host normally makes.
Protocol on the wrong port
Port numbers are conventions, not laws. The web port usually carries TLS, the plain-text management ports usually carry plain text, and a database port usually carries the database's own protocol. Metadata analysis catches two kinds of mismatch, and both are worth alerting on.
The first is TLS where you expected something else. A TLS handshake has a recognizable structure in its first bytes; capture tools and flow exporters identify it by content, not by port. A handshake may appear on a port that should carry an old plain-text management protocol, or on a port you never opened for anything. In that case, a program is wrapping itself in encryption to reach somewhere. Sometimes an admin moved a service to a custom port and forgot to tell anyone. Sometimes not.
The second is something else on the TLS port. Bytes on the web port that are not a TLS handshake at all. Firewalls tend to allow the web port outbound from everywhere, which makes it an attractive place for any protocol that wants through unchallenged. If your flow exporter or sensor labels the protocol it actually saw, a filter for "web port, but not TLS" is one of the cheapest useful alerts in this whole pillar.
False positives are real and mostly boring. Examples include administrative consoles on alternate web ports, and database servers that speak TLS on their own port by design. Another example is VPN products that deliberately use the web port so they work from hotel wireless. Build a list of the port-and-protocol pairs you expect on your network. Alert on pairs outside it. And add each confirmed-benign pair to the list with a note about who owns it.
The tools, generically
You need something that records handshake metadata, and somewhere to keep it. Three families of tool do the recording:
- Flow exporters with handshake fields. Many firewalls and some switches can export more than addresses and byte counts. They can also export the server name, the certificate's subject and issuer, and sometimes a handshake fingerprint label. Look for words like "application visibility" or "extended flow fields" in the device's documentation. This is the lowest-effort path.
- Passive network sensors. This software watches a mirror port (a switch port configured to copy traffic from other ports). It writes one log line per connection with all the metadata it can see, plus a separate log of certificates observed. Several free and commercial sensors do this; the output looks much like the sample below.
- Packet capture tools with protocol dissectors. A dissector is the part of a capture tool that decodes a protocol's fields for display. Open a capture and filter for TLS handshakes. The dissector shows you the server name, the offered options, and the certificate in readable form. This is the slowest route but the best one for learning. The packet capture fundamentals guide covers getting comfortable with a capture tool.
Whatever produces the records, ship them to a log collector that can search and count. A sensor's connection log with handshake metadata looks roughly like this:
time source destination server_name issuer subject client_fp 09:14:02 10.20.5.41:51834 203.0.113.17:443 files.example.test public-ca-1 *.example.test browser-a 09:14:05 10.20.5.41:51840 203.0.113.17:443 files.example.test public-ca-1 *.example.test browser-a 09:14:09 10.20.5.77:60122 198.51.100.8:443 (none) self-signed localhost unknown-1 09:19:09 10.20.5.77:60131 198.51.100.8:443 (none) self-signed localhost unknown-1 09:24:09 10.20.5.77:60140 198.51.100.8:443 (none) self-signed localhost unknown-1
Read it the way a detective reads a guest book. The first host is a browser fetching a file service twice, public issuer, matching names: nothing to see. The second host has no server name. It has a self-signed certificate claiming to be localhost, and an unknown fingerprint. And — look at the timestamps — it has a connection every five minutes to the second. Four techniques fired on five lines of log, and you decoded nothing.
Tip: the fingerprint labels in your tool will be opaque strings, not friendly names like browser-a. That is fine. Spend the first week building a lookup table of label to "what we think it is." To do that, check which processes on a known-good workstation produce each label. The table becomes the most valuable document in your metadata practice.
Technique, reveal, and false positive: the table
| Technique | What it reveals | Common false positives |
|---|---|---|
| Handshake fingerprint, new label on one host | New software on that host | Software updates; a user installing a legitimate tool; a rarely used application |
| Handshake fingerprint, non-browser to a web service | A program, not a person, made the connection | Update agents, sync clients, monitoring tools, scripts written by IT |
| Self-signed certificate, outbound | A client that does not verify identity, talking to an unvouched server | Lab and test systems; appliances calling home with factory certificates |
| Expired certificate, still used | Your own service needs renewal, or a client that does not check | Forgotten internal services; devices with wrong clocks |
| Name mismatch | Server presenting a card for a different name | Shared hosting; misconfigured internal web servers; wildcard quirks |
| Unknown issuer | A trust chain your network has never used | New cloud services; regional authorities; a partner's internal authority |
| First-seen certificate on a known service | The service's identity changed | Scheduled renewals; automated short-lived certificates; load balancer changes |
| Server name never seen before | First contact with a new service | Content delivery networks; new SaaS onboarding; advertising and analytics names |
| Missing server name plus missing DNS lookup | Client had the address without asking for it | Hard-coded addresses in appliances; direct-to-IP admin tools |
| Beacon shape | Automated, regular check-ins | Your own agents, update checkers, chat clients, mail polling |
| Bulk upload shape | Large one-way transfer out | Backups, cloud sync, video calls, log shipping |
| TLS on an unexpected port | Encryption wrapped around something on a non-standard port | Admin consoles on alternate ports; databases with TLS enabled |
| Non-TLS on the TLS port | Another protocol using the always-open port | VPN clients designed to use it; some conferencing tools |
A first week of metadata analysis
Doing all of this at once produces an alert queue nobody reads. A slower sequence works better:
- Day one: get the records flowing. Enable flow export with handshake fields on the edge firewall, or stand up a sensor on a mirror port of the switch that carries internet-bound traffic. Point it at a log collector. Alert on nothing yet.
- Days two to five: collect quietly. Let a full working week accumulate, including the overnight and weekend patterns that reveal scheduled jobs and agents.
- Day six: build the baselines. Three lists: fingerprint labels seen per device type, certificate issuers seen, and server names per host. Note the beacon-shaped flows and identify each one; most will be your own tools.
- Day seven: turn on two alerts. Self-signed certificate on an outbound connection from a non-lab host, and protocol-port mismatch. Both are rare, both are cheap to check, and both build confidence.
- Week two onward: add novelty alerts. New fingerprint label on a single host, new issuer, and beacon shape to a destination outside your baseline. Review weekly, and for every alert that turned out benign, add the case to a baseline list with an owner's name.
The instinct to skip the quiet week is strong. Resist it: every technique in this guide works by comparison against normal, and you cannot compare against a normal you have not measured.
Where to go next
Metadata analysis is a set of habits more than a set of tools. Know your fingerprints, issuers, names, shapes, and ports, and notice when any of them changes. Each technique has a reveal and a false positive. The table above is meant to be kept next to the alert queue until you no longer need it.
From here, the anomalies guide turns these techniques into specific alert rules with confirmation steps and a triage sequence. If someone in your organization is proposing to decrypt traffic instead, read the inspection tradeoffs guide before the meeting. And for the bigger design picture — where metadata fits alongside DNS logs, endpoint logs, and server logs — see visibility without breaking encryption. All of them are part of Monitoring Encrypted Traffic.
Put this into practice with SysPeep Monitor
Metadata analysis tells you what the traffic looks like. SysPeep Monitor tells you what it does to your network. It auto-discovers your switches, routers, firewalls, and servers. It collects utilization and health over SNMP, WMI, and SSH. And it alerts when a link fills up or a device misbehaves. That is the volume side of every shape in this guide. Free trial on any Windows machine.
Download Free TrialLearn About SysPeep