Home › Topics › Monitoring Encrypted Traffic › Visibility Without Breaking Crypto

Getting Visibility Without Breaking Encryption


You have read about TLS inspection and decided against it. Or someone decided for you, or the budget did. Or the mix of personal phones and building controllers on your network made it impossible. Now you are standing in front of a wall of encrypted traffic with no gateway to open it. And you have a manager who still expects you to know what is going on. The temptation is to conclude that visibility and encryption are opposites, and that you chose encryption.


They are not opposites. The mistake is looking for visibility in the one place encryption was designed to deny it: the middle of the wire. Every encrypted connection has two ends that know exactly what happened, because they are the ones doing it. It also casts a shadow. That includes the DNS lookup that came just before it and the flow record the firewall wrote about it. The shadow also includes the certificate the server showed during the connection, and the policy decision that allowed it. Go around the encryption, not through it, and most of what you wanted is still there.


This guide is the design for doing exactly that. It walks through six sources of visibility that need no decryption. It shows how to combine them into a single timeline that answers real questions. It is honest about the one gap the approach leaves open. And it ends with a blueprint sized for a small team with no dedicated security staff.


In this guide

  • The design principle: around, not through
  • Source one: endpoint-side logs
  • Source two: DNS query logs
  • Source three: flow metadata and handshake observation
  • Source four: server-side logs for your own services
  • Source five: gateway policy by destination category
  • Source six: certificate inventory
  • Combining sources into a timeline
  • Where this approach falls short
  • Sources at a glance: the table
  • A small-team blueprint
  • Where to go next

The design principle: around, not through


Picture an encrypted connection as an opaque tube between two rooms. You cannot see into the tube. But each room has a window, and the tube itself casts a shadow on the floor. The shadow shows how thick the tube is, how long it stayed connected, and which rooms it joined. The design question for every connection you care about is simple: do I have a window into at least one end, or a clear view of the shadow, or both?


Sort your connections into three groups. You own the client: a managed workstation or server talking to something on the internet. You have a window at the client end. You own the server: someone, inside or outside, talking to your web application, your mail system, your VPN. You have a window at the server end, and you hold the keys there, so nothing is hidden from you at all. You own neither: a personal phone on the guest network talking to the internet. All you have is the shadow and whatever policy the gateway applied. Most networks are dominated by the first two groups, which is why this design works. The third group is where you rely on segmentation and policy rather than visibility, and that is a reasonable place to draw the line.


Source one: endpoint-side logs


The client knows everything about the connection, because it made it. The question is only whether it wrote that knowledge down and whether you collect it. Four kinds of endpoint logs matter here:


  • Operating-system connection logs. Most operating systems can record every outbound connection with the name of the process that opened it. That process name is gold. It turns "the workstation connected to an unknown address" into "a program called updater.exe in a temporary folder connected to an unknown address". Usually this needs a policy setting or the host firewall's logging turned on.
  • Managed browser logs. Browsers under central management can report the URLs they visit to a collection point. This is the only clean way to see full URLs without inspection. It comes with the same privacy questions, so the same category exemptions apply.
  • Application and agent logs. Mail clients, sync clients, backup agents, and your own monitoring agents all log what they did. Those logs explain a large share of the traffic that would otherwise look mysterious from the wire.
  • Endpoint protection logs. As a concept: software on the device that watches processes, files, and connections together, and records what it saw. Its logs pair a connection with a file hash and a process tree, which the wire can never do.

The catch is coverage and trust. Endpoint logs exist only on devices you manage. They are useful only if shipped somewhere central before the device is lost or wiped. A compromised endpoint can lie. That last point is why the wire-side sources below are not optional. The shadow on the floor is the cross-check that keeps the window honest.


Source two: DNS query logs


Before nearly every connection, the client asks a question: what is the address of files.example.test? That question goes to your DNS resolver in plain text. Your resolver can log it: which client asked, for which name, at what time, and what answer came back. DNS logs are tiny, cheap, and astonishingly useful. They give you the name even when the handshake hides it. They give you a first-seen record for every new name any host has ever asked about. The answer they record — the address — is the join key that links the lookup to the flow record that follows a moment later. The DNS as a threat sensor pillar covers the detection side of this in depth.


Two design rules make DNS logs trustworthy. First, make your resolver the only path. Block or redirect outbound DNS from clients to anything other than your resolver at the gateway. That way, a device cannot quietly ask someone else. Second, remember caching. A client looks a name up once and may reuse the answer for many connections. So the absence of a lookup right before a flow is not automatically suspicious. Look for the lookup within the cache lifetime, not within the same second.


Source three: flow metadata and handshake observation


This is the shadow itself. A flow record summarizes one connection: addresses, ports, bytes each way, packet counts, start, end. A record enriched with handshake metadata adds the server name the client asked for, the certificate's subject and issuer, and a label for the client's handshake style. Routers and firewalls export flow records; passive sensors on a mirror port produce the enriched kind. The metadata analysis guide covers how to read all of it, so this section stays short.


In the design, flow metadata has two jobs. It is the source that covers every device, managed or not, because it is collected from the network rather than the endpoint. And it is the independent witness. Suppose an endpoint log says a process connected to one place and the flow record says the machine connected somewhere else. In that case, the flow record is the one to believe.


Source four: server-side logs for your own services


Here is the part of the design that people overlook while worrying about the wire. For every service you run, you hold the keys. Your web application, your mail server, your VPN concentrator, your file service, your directory — each one decrypts the traffic legitimately, as the intended recipient. Each one can log the full request in plain text. User name, URL path, bytes transferred, status, source address, time. Everything inspection would have shown you, for the connections that matter most to the business, with no trusted middle and no substitute certificates.


A practical pattern makes this easier. Put a reverse proxy or load balancer in front of your internal web applications, let it terminate TLS, and log there. One place, one log format, every internal web application covered, and the applications behind it need not be touched. Authentication systems deserve the same attention. A log of who logged in from where, and who failed to, answers many of the questions that start with "who accessed".


Source five: gateway policy by destination category


Even without decrypting, the gateway sees the server name and the destination address. That is enough to sort connections into categories — file sharing, webmail, business software, uncategorized, newly seen. It is also enough to allow or deny by category. The log of those decisions is a visibility source in its own right. It records intent: this user's device tried to reach that class of site at that time, and was allowed or was not. Denied attempts are especially informative, because a host repeatedly trying to reach a blocked category is telling you something regardless of what it would have sent.


Category filtering at the DNS layer is a variant of the same idea. The resolver refuses to answer for names in blocked categories, and logs the refusal. DNS category filtering is simpler to deploy than gateway filtering, works for any device that uses your resolver, and pairs naturally with the DNS logs you already keep.


Source six: certificate inventory


Several detections in this pillar depend on noticing a certificate that changed, or one you have never seen. Both require a baseline: a list of the certificates your own services present, with subject, issuer, expiry, and a fingerprint of each. That list is a certificate inventory, and the certificate inventory guide walks through building one. In this design, the inventory is the reference the flow-metadata source is compared against. Suppose the certificate observed on the wire for your mail server does not match the inventory. Either someone renewed it without updating the record or something is presenting itself as your mail server. Either way you want to know.


Tip: the inventory also does the boring, valuable thing of warning you before your own certificates expire. For many small teams, certificate expiry is the most common encrypted-traffic "incident" of the year. Build it for the security case; enjoy it for the expiry alerts.


Combining sources into a timeline


No single source answers a real question on its own. The design works when you can line them up by time and by address. Here is an ordinary connection viewed across five sources, in the layout a search across a central log collector might give you:


09:13:58  DNS       10.20.5.41 asked files.example.test  -> 203.0.113.17
09:14:02  FLOW      10.20.5.41:51834 -> 203.0.113.17:443  41 KB  1.9 s
                    name=files.example.test  issuer=public-ca-1  fp=browser-a
09:14:02  GATEWAY   user=jsmith  category=file-sharing  action=allow
09:14:02  ENDPOINT  host=WS-041  process=browser  url=https://files.example.test/shared/q3-report
09:14:03  CERT-INV  *.example.test  public-ca-1  known, expires in 61 days

Read the story. The workstation looked up the name and connected four seconds later. The gateway allowed it as file sharing for a named user. The endpoint says a browser opened a specific report. And the certificate is one you already know. Five lines, five sources, one boring and fully explained event. The join keys were the client address, the destination address, the name, and a window of a few seconds. Now the same search for something less boring:


02:41:10  DNS       10.20.5.41 asked cdn-sync-7731.example.invalid  -> 198.51.100.8
02:41:17  FLOW      10.20.5.41:60122 -> 198.51.100.8:443  2.1 GB out  42 KB in  1 h 47 m
                    name=(none)  issuer=self-signed  subject=localhost  fp=unknown-1
02:41:17  GATEWAY   user=(no session)  category=uncategorized  action=allow
02:41:17  ENDPOINT  host=WS-041  process=svchelper.exe  path=C:\Users\jsmith\AppData\Local\Temp\
                    (no CERT-INV record for this destination)

Same workstation, middle of the night. A name that was never looked up before, resolving to an address nobody else talks to. A two-hour upload of two gigabytes, no server name, a self-signed certificate claiming to be localhost, an unknown handshake label. The gateway allowed it because uncategorized was allowed. And the endpoint says the process was a program in a temporary folder, not a browser. You decrypted nothing. You know which host, which user's profile, which process, how much left, where it went, and when it started. You learned all that in the time it took to run one search. That is what "visibility without breaking encryption" means in practice.


Where this approach falls short


Be clear about the gap, because someone will ask. This design cannot see content-level threats inside allowed connections. Consider a malicious file downloaded from a popular, correctly categorized, publicly certified file-sharing service. From every source above, it looks exactly like a colleague downloading a spreadsheet: known name, known issuer, browser handshake, allowed category, ordinary size. The wire cannot tell the two apart, and neither can DNS or the gateway.


What covers that gap is endpoint protection, as a concept. That is software on the device that examines files when they arrive and processes when they run, regardless of how they got there. In this design the endpoint is the other end of the tube. It is where content is examined. The endpoint is the one place content is legitimately in plain text on a device you control. Inspection at the gateway would give you a second look at the same content. For some organizations that second look is worth its cost. The inspection tradeoffs guide is the place to weigh it. For most small teams, the honest answer is that endpoint protection is not optional under any design, and this one simply makes that explicit.


There are two smaller gaps. Unmanaged devices give you only the shadow, so keep them on their own segment with their own policy. A fully compromised endpoint can falsify its own logs. That is why the wire-side sources are the independent witness you never switch off.


Sources at a glance: the table


SourceWhat it gives youWhere it livesBlind spot
Endpoint connection logsProcess name, user, destination, timeOn each managed device; ship to a collectorUnmanaged devices; a compromised host can lie
Managed browser logsFull URLs visitedBrowser policy reporting to a collectorNon-browser traffic; privacy exemptions apply
DNS query logsName looked up, by whom, when, and the answerYour resolverClients using other resolvers; cached answers
Flow recordsAddresses, ports, bytes, duration for every deviceFirewall or router exportNo names or certificates unless enriched
Handshake metadataServer name, certificate, client fingerprintEnriched flow export or a mirror-port sensorConnections that hide the name
Server-side logsEverything, for services you runWeb servers, reverse proxy, mail, VPN, directoryOnly your own services
Gateway category logsIntent: who tried to reach what class of site; denialsWeb gateway or DNS filterCategories are approximate; uncategorized is a bucket
Certificate inventoryWhat normal looks like for your services; expiry warningsA maintained list plus scheduled checksOnly covers services you listed

A small-team blueprint


Here is the order that gets the most visibility for the least effort, assuming one or two people, existing equipment, and no security budget line. Each step stands on its own; stop wherever time runs out and you are still better off than when you started.


  • Step 1: Own DNS. Turn on query logging on your resolver. At the gateway, block outbound DNS from clients to anything except that resolver. One afternoon, and the single highest-value change in this list.
  • Step 2: Export flows. Enable flow export on the edge firewall, with handshake fields if the device supports them, pointed at a collector. If it does not support enrichment, note that a mirror-port sensor is the upgrade path.
  • Step 3: Log your own services. Ship web, mail, VPN, and authentication logs to the same collector. Put a reverse proxy in front of internal web applications so one log covers them all.
  • Step 4: Turn on category logging. Whether at the gateway or the DNS filter, log allow and deny decisions per category. Make sure uncategorized and newly seen names are at least logged even if allowed.
  • Step 5: Build the certificate inventory. List every service you run with its certificate details, and schedule a check that compares the live certificate to the list.
  • Step 6: Enable endpoint connection logging on managed devices, with process names, and ship it. This is the step that turns "which host" into "which program".
  • Step 7: Get one search tool. Anything that can search all of the above by address and time window. Fancy correlation is optional; a text search across sources with a shared clock is what solves cases.
  • Step 8: Write down three questions you can now answer. "Which hosts contacted this name?", "What process made this connection?", "Has this certificate changed?" Practise answering them on quiet days, so the search is familiar on loud ones.

Notice that nothing in the blueprint required a new appliance, a substitute certificate, or an exemption list. It required turning on logs that already existed, pointing them at one place, and synchronizing the clocks. That last item is not a joke. A timeline built from sources with clocks a few minutes apart is a puzzle, not a timeline. Make time synchronization part of step one.


Where to go next


Visibility without decryption is a design, not a compromise. You collect from the two ends of every connection you own. You watch the shadow for everything else. You compare what you see against a baseline of names, categories, and certificates. The connections that matter most — to your own services — are the ones where you have the most visibility, because you hold the keys. The gap that remains is content inside allowed connections, and the endpoint covers it.


To turn these sources into alerts, continue with anomalies in encrypted traffic, which gives detection logic and a triage sequence for each pattern. If you still need to explain to someone why the envelope is enough, what encryption hides and what it still shows makes that case. Everything in this series lives under Monitoring Encrypted Traffic.


Put this into practice with SysPeep Monitor

Most of the sources in this guide are logs; the one that is not is the network itself. SysPeep Monitor watches the switches, routers, firewalls, and servers that carry the traffic. It watches utilization, health, and availability over SNMP, WMI, and SSH, and alerts when they change. Pair it with the log sources above and your timeline gains a network layer. Free trial on a single Windows machine.

Download Free TrialLearn About SysPeep