Home › Topics › Monitoring Encrypted Traffic › Encrypted Traffic Anomalies

Anomalies in Encrypted Traffic: Detection Without Decryption


You have the plumbing. Flow records are arriving from the firewall. Your resolver is logging DNS queries. A sensor or an enriched flow export is recording server names and certificates. You have a rough idea of what normal looks like. Now comes the question that separates a monitoring setup from a pile of logs: what, exactly, should page you? The tempting answer is "anything new", and it lasts about a week before the alert queue is ignored by everyone including you.


This guide narrows that down to six patterns in encrypted traffic that are worth an alert, each detectable from the envelope alone. For each one you get what it looks like and the detection logic in plain terms. You also get the part that keeps the queue short: a confirmation step. That step filters out the innocent cases before a human sees the alert. Then you get a triage sequence for when an alert does arrive, and a table to keep beside the queue.


One framing before we start. An anomaly is something that differs from your network's normal. It is not an attack. Most anomalies you catch will be colleagues doing new things, agents nobody remembered installing, and vendors changing certificates. The goal is not a queue of zero. It is a short queue where each item deserves ten minutes. Those ten minutes usually end with a note in a baseline list and occasionally with a phone call.


In this guide

  • Anomaly without content: what you are really detecting
  • Pattern one: untrusted certificates inside the network
  • Pattern two: TLS on unexpected ports, and the reverse
  • Pattern three: long sessions to brand-new destinations
  • Pattern four: regular small sessions
  • Pattern five: a certificate that changed when you did not change it
  • Pattern six: volume to a single encrypted destination
  • The triage sequence
  • Benign or worrying? The table
  • Where the findings go next

Anomaly without content: what you are really detecting


With the payload locked, you cannot detect what was said. You can detect four kinds of change. You can detect a change in identity (a certificate or issuer that is wrong or new). You can detect a change in place (a port or destination that is unexpected). You can detect a change in rhythm (sessions that are too regular, too long, or at the wrong hour). And you can detect a change in quantity (more bytes than that host or destination ever moves). Every pattern below is one of those four, or a combination. Keeping that in mind stops you from inventing alerts that secretly require content you do not have.


Every pattern also needs the same three ingredients. It needs a record source (flow records with handshake metadata, plus DNS logs). It needs a baseline (a list of what is known and expected, built from a quiet week or two of records). And it needs a confirmation step that checks the baseline before alerting. Detection without confirmation is how alert queues die. If you have not yet built the baselines, the metadata analysis guide ends with a one-week plan for doing so.


Pattern one: untrusted certificates inside the network


What it looks like

A host that is not a lab machine completes an encrypted session with a server. That server's certificate is self-signed (issuer and subject are the same). Or it is signed by an issuer that is neither a public authority nor your internal one. Inside the network this can be either direction. It could be a workstation connecting out to an internet server with an untrusted certificate. Or it could be one internal host connecting to another that presents one. Normal browsers refuse untrusted certificates, so a completed session means the client software did not check. Software that does not check identity is either an appliance, a badly written agent, or something built to talk to a server its author controls.


Detection logic and confirmation

FOR each completed TLS session where the client is an internal host
  IF   certificate.issuer == certificate.subject             (self-signed)
    OR certificate.issuer NOT IN known_issuers
  AND  client_host NOT IN lab_hosts
  AND  (client_host, server_address) NOT IN accepted_untrusted_pairs
  THEN raise "untrusted certificate" for review

The two lists do the noise control. lab_hosts holds machines where self-signed is normal: test servers, developer workstations, network appliances that call home with factory certificates. accepted_untrusted_pairs holds specific host-to-server pairs someone reviewed and accepted, with an owner's name beside each. Confirm before alerting by checking that the session actually carried data after the handshake. A client that connected, saw the certificate, and disconnected immediately was probably a browser doing the right thing.


Pattern two: TLS on unexpected ports, and the reverse


What it looks like

Two mirror-image cases. First, there is a TLS handshake on a port where you expect a plain-text protocol or nothing at all. That is encryption wrapped around a service that was never meant to be there. Second, there is traffic on the standard TLS port that is not a TLS handshake: some other protocol borrowing the port everyone leaves open. Both are detected by protocol identification based on content rather than port number, which flow exporters with application identification and mirror-port sensors both provide.


Detection logic and confirmation

Maintain a list of expected (server_port, protocol) pairs for your network. Alert when a session's identified protocol does not match the list for its port. Confirm by checking whether the server address is one of yours. An internal server running an admin console with TLS on an unusual port is an inventory update, not an incident. It gets added to the list with an owner. An external server receiving TLS on an odd port from a workstation stays in the queue. So do non-TLS bytes on the TLS port from a host that is not running a known VPN client. The most common benign cases are administrative web consoles on alternate ports and databases with encryption enabled on their own port. Other common benign cases are VPN and conferencing clients that use the web port by design.


Pattern three: long sessions to brand-new destinations


What it looks like

An encrypted session lasts far longer than sessions to that kind of port normally do. It goes to a destination that no host on your network had contacted before today. Web sessions are usually seconds; remote-access sessions can be hours but go to known places. A multi-hour session to somewhere new combines a change in place with a change in rhythm. That pairing is what makes it worth an alert where either alone would be noise.


Detection logic and confirmation

FOR each session from an internal host to an external server
  new  := server_name (or server_address) NOT IN first_seen_table
          before today
  long := duration > long_threshold_for(server_port)
  IF new AND long
  THEN look up DNS name, certificate issuer, and client fingerprint
       IF issuer is public AND fingerprint is a known browser
          -> log only (probably a new web application in use)
       ELSE raise "long session to new destination"

The first_seen_table records the first date any host contacted each name and address; keep it forever, it is small. The long threshold depends on the port: a few minutes for the web port, longer for remote-access ports. The confirmation branch is what keeps this useful. A new destination reached by a normal browser with a public certificate is nearly always someone using a new website. So it is logged for the weekly review rather than paged. The alert fires for the combination of new, long, and either a non-browser client or a non-public certificate.


Pattern four: regular small sessions


What it looks like

The beaconing shape is a host opening a small encrypted session to the same destination at a fixed interval. Each session is about the same size. They continue regardless of whether anyone is using the machine. The beaconing patterns guide covers the shape in depth, so this section stays on the alert itself.


Detection logic and confirmation

Group sessions by client host and destination over a window of several hours. For each group with more than a handful of sessions, compute how much the interval between sessions varies and how much the byte counts vary. Low variation in both, over a window that includes hours when the host's user is away, is the signal. Confirm against a known_agents list. Your monitoring agents, update checkers, mail clients polling for messages, chat clients, and backup software all beacon by design. They are the bulk of what this rule finds. Two refinements cut the remainder. Weight destinations with no readable server name and a non-public certificate more heavily. Treat a beacon that started on the same day the host first contacted that destination as more interesting than one that has run for months.


Pattern five: a certificate that changed when you did not change it


What it looks like

Clients inside your network are being shown a certificate for one of your well-known internal services. The service might be the mail server, the intranet, the directory, or the VPN. The certificate differs from the one in your certificate inventory. Sometimes the subject is the same but the issuer changed. Sometimes the fingerprint changed and nothing else. Sometimes only some clients see the new certificate while others still see the old one. That is the most interesting variant, because a real renewal shows the new certificate to everyone at once.


Detection logic and confirmation

FOR each observed certificate on a session to an internal service in the inventory
  IF certificate.fingerprint != inventory[service].fingerprint
  THEN check the change log for a renewal of that service
       IF renewal recorded today  -> update the inventory, close
       ELSE raise "unexpected certificate change" (high priority)
            note which clients see the new one and which see the old

This is the pattern with the fewest false positives, provided the inventory is maintained. The benign causes are a renewal someone forgot to record, a load balancer with one member holding a stale certificate, and automated short-lived certificates renewing on schedule. The worrying cause is something between the client and the service presenting its own certificate, which is exactly what this rule was built to catch. Because the detection depends entirely on a good inventory, the certificate inventory guide is a prerequisite rather than a suggestion.


Pattern six: volume to a single encrypted destination


What it looks like

One host sends far more bytes to one encrypted destination than it ever has. This stands out especially when the ratio is heavily outbound and the hour is one when the host is normally quiet. This is the bulk-upload shape, and it is a change in quantity first and rhythm second. It is the pattern most likely to be benign — backups and cloud sync produce it every night on purpose. It is also the one most people most want to catch. A large one-way transfer is what data leaving the building looks like from the envelope.


Detection logic and confirmation

Track, per client host and destination, the bytes sent per day. Alert when today's total to a destination exceeds several times that host's previous daily maximum to any destination, and the outbound-to-inbound ratio is strongly outbound. Confirm against a known_bulk_destinations list — backup targets, sync services, log shipping endpoints, video conferencing. Confirm against the schedule too. A backup that runs at two in the morning to the same target every night is not an anomaly. The same volume at two in the morning to a first-seen destination is. Cross-check the DNS name and certificate. A public certificate for a well-known sync service downgrades the alert to a review item. A self-signed certificate on a first-seen address upgrades it to a phone call.


Tip: combine patterns before paging. Any one of the six on its own is usually a review item. Two of them on the same host and destination within the same hour is what should wake someone up. That could be a self-signed certificate and a bulk upload, or a new long session and a non-browser fingerprint. Most alert tools can express "two rules matched the same host in the same window". Use that as your paging rule and keep the single-pattern alerts for the daily review.


The triage sequence


An alert has fired. The same four questions, in the same order, resolve nearly every one in ten minutes. Resist the urge to jump to the fourth.


  • 1. Which host? Turn the address into a device and a person. The DHCP lease says which machine held the address at that time; the inventory says whose it is and what it is for. A workstation, a server, a printer, and a lab box are four completely different conversations. If you cannot identify the host, that is itself a finding: an unknown device on your network is a problem regardless of what it was doing.
  • 2. Which destination? Gather the DNS name that was looked up, the server name from the handshake, the certificate subject and issuer, and the gateway's category. Then ask the most useful question in the whole sequence: who else talks to it? A destination that half the company contacts is a service; one that only this host contacts, starting today, is not.
  • 3. Since when? Look up the first-seen date for this host-and-destination pair. Then check what changed on the host around that date: a software install, an update, a new user, an email with an attachment, a ticket. The anomaly's birthday is usually the explanation.
  • 4. What else does the host do? Now widen the search to everything the host did in the same window. Look for other new destinations, other patterns from this guide, and, if you have endpoint logs, the process name behind the connection. One anomaly on a host that is otherwise ordinary is a review item. Three anomalies in an hour on a host that has never had one is an incident.

Then decide, and write the decision down. Benign: add the case to the relevant baseline list with an owner and a date. That way, the same alert never fires again for the same reason. Unclear: leave it in a watch list and set a reminder to look at the host again tomorrow. Anomalies that matter tend to repeat. Worrying: hand it to whatever incident process your organization has, with the timeline you just built attached. The four answers above are exactly what the next person will ask for.


Here is the sequence applied to a real-looking alert, the kind you will see most often. Pattern four fires: host 10.20.7.115 has contacted 198.51.100.40 every ten minutes for six hours. Which host? The DHCP lease points to a laptop in the finance team. Which destination? DNS shows a lookup for a name belonging to a well-known accounting service. The certificate is from a public issuer and matches the name. And eleven other hosts contact the same destination. Since when? The pair first appeared three weeks ago, the same day the finance team rolled out a new expenses tool. What else? Nothing; the laptop is otherwise ordinary. Decision: benign; the tool polls for updates. Add the destination to known_agents with the finance manager's name as owner. Total time, six minutes, and that alert never fires again.


Benign or worrying? The table


PatternLikely benignLikely worryingFirst check
Untrusted certificate inside the networkLab host; appliance calling home; internal test serverWorkstation to external self-signed server with a generic subjectIs the client in the lab list? Does the session carry data past the handshake?
TLS on an unexpected portAdmin console or database with encryption on an alternate port, on your own serverExternal server on an odd port from a workstation; non-TLS bytes on the TLS portIs the server address yours? Is a known VPN or conferencing client involved?
Long session to a new destinationNew web application in use; browser fingerprint; public certificateNon-browser fingerprint; no server name; non-public certificate; overnightWho else on the network contacts this destination?
Regular small sessionsYour own agents, updaters, mail polling, chat clientsBeacon that began the day the destination was first seen; no readable nameIs the destination in the known-agents list? When did the pair first appear?
Changed certificate on an internal serviceRecorded renewal; stale load-balancer member; automated short-lived certificatesNo renewal recorded; different issuer; only some clients see the new oneIs there a change record? Do all clients see the same certificate?
Volume to a single destinationBackup or sync to a known target on its normal scheduleFirst-seen destination; self-signed certificate; off-hours; host not a serverIs the destination in the bulk list? Has this host ever sent this much before?

Where the findings go next


Six patterns, one method: detect a change in identity, place, rhythm, or quantity. Confirm against a baseline list before anyone is interrupted. Triage with the same four questions every time. Feed each decision back into the lists so the queue gets shorter as the months pass. None of it needed a byte of decrypted payload, and none of it required you to break the protection your users rely on.


When a finding turns out to be worrying, two neighbouring pillars pick up the story. If the pattern was volume or a long upload, the data exfiltration patterns guides pick up the story. They cover what data leaving the building looks like across every channel, not just encrypted ones. If it was rhythm or place, the traffic anomaly detection guides go deeper on baselining and on the statistics behind "unusual". And if you want the full design that these alerts sit inside, visibility without breaking encryption shows how the sources combine. All of it lives under Monitoring Encrypted Traffic.


Put this into practice with SysPeep Monitor

Every pattern in this guide starts with knowing what normal looks like, and normal starts with graphs. SysPeep Monitor tracks utilization per interface and health per device over SNMP, WMI, and SSH. It has threshold and dependency-aware alert rules built to keep noise down. When a link's overnight traffic doubles, that is the kind of change it flags. The free trial installs on one Windows machine.

Download Free TrialLearn About SysPeep