Home › Topics › IoT, Cameras & Building Systems › Cameras and NVRs

Cameras and NVRs: Watching the Watchers


Nobody checks a security camera until the footage is needed. A camera can be dead for months, unplugged during a renovation, killed by a power blip, or simply failed. And every day it stays dead is invisible, because the one thing a camera does happens in a closet on a recorder nobody opens. Then something goes wrong in the parking lot and someone asks for the video. The discovery is made at the worst possible moment: the camera covering that exact corner stopped recording a long time ago.


This is the backup problem in a different costume. A backup that has been silently failing is discovered on restore day. A camera that has been silently dead is discovered on evidence day. I have sat in that meeting. Nobody in it was interested in why. The cure is the same as for backups: make the failure loud on day one, with monitoring, instead of letting silence pass for success.


A camera system is unusually monitorable for something you mostly cannot log into. This guide covers the whole chain. That means proving each camera is alive and proving the recorder is actually keeping footage as long as policy says. It also means respecting the power and bandwidth physics underneath it all. And the guide covers the one low-tech habit that catches what no network check can. By the end, "are the cameras okay?" becomes a question your dashboard answers — not the parking lot.


In this guide

  • Why dead cameras stay dead
  • Know the fleet before you watch it
  • Camera availability: three levels of proof
  • The NVR: where footage lives or dies
  • Retention: days-of-footage is the honest metric
  • The bandwidth reality
  • PoE dependency chains
  • The monthly footage spot-check
  • Ownership and privacy, honestly

Why dead cameras stay dead


A camera fails silently for a structural reason: its normal operation produces nothing a human sees. A file server that dies generates complaints within minutes because people touch it all day. A camera that dies generates nothing at all. The live wall in the security office (if anyone watches it) has a dozen tiles. The recorder keeps humming. And one black or frozen tile among many is exactly the kind of detail human attention filters out by week two. The failure modes are mundane: a PoE injector unplugged to charge a vacuum, a switch port that died, and a camera that hung and never rebooted. Other mundane failures include water in a housing and a firmware update that half-completed. None of them announce themselves.


The stakes are asymmetric, which is what makes this worth engineering properly. The cost of monitoring a camera is a check that runs every few minutes and bothers nobody. The cost of not monitoring it is discovered exactly once, during an incident investigation, an insurance claim, or a legal request. Those are audiences with no patience for "it seems that camera was down." You do not get to fix it retroactively. Footage that was never recorded cannot be recovered by any amount of after-the-fact effort, and everyone in that meeting knows it.


So treat the camera fleet the way you treat backups: assume silent failure is the default state of the universe. And build checks that force the system to prove, continuously, that it is doing its job.


Know the fleet before you watch it


Before any checks, you need a list. Camera fleets are notorious for not having one, because they arrive via installers and grow one camera at a time. Walk the recorder's channel list and the network. Build an inventory with the fields that matter during an incident. Include a name that describes the view, not the hardware (cam-loading-dock-east, never camera-07). Include the IP address and hardware address, the switch and port feeding it, and whether it is PoE-powered. Include which recorder and channel it records to, and who installed and services it. The general discipline of discovering and recording devices is covered in building a device inventory. Cameras just add the view-name and recorder-channel columns.


The view-based name is not cosmetic. During an incident, nobody asks "is camera-07 up?" They ask "do we have the loading dock?" (Nobody has ever known which one camera-07 is.) An alert that says cam-loading-dock-east down answers the real question instantly and tells the responder where to walk. While you are at it, set each camera's switch-port description to the same name. Your switch becomes a second, independent copy of the map.


Camera availability: three levels of proof


Most cameras will not accept an agent and many speak no useful SNMP. They are classic sealed devices, and the outside-in approach from monitoring the unmanageable applies directly. For cameras specifically, availability checks come in three strengths, and it is worth knowing exactly what each one proves:


LevelCheckWhat it provesWhat it misses
1PingThe camera is powered and its network stack answersA hung camera can still answer; says nothing about video
2TCP port checkThe video or web service is accepting connectionsThe service can accept connections and still serve no frames
3Stream checkActual video frames are being delivered right nowFrames can flow while the image is misaimed, blocked, or dark

Level 1 is universal and free; every camera should have it. Level 2 is nearly as cheap. Almost all network cameras expose at least one service port: a web interface, or the port their streaming protocol lives on. A connection check against it catches the "answers ping, serves nothing" failure mode. Level 3 is the strong claim: a check that actually requests the stream the way a recorder would and confirms that video data arrives. Some monitoring setups can do this generically. Where yours cannot, the recorder itself is your level-3 proxy, because it is requesting every stream constantly anyway. That is the next section's job.


Alert conservatively on cameras: a camera that misses one ping is noise, and cameras reboot themselves more than servers do. A sensible starting rule is "unreachable for five minutes" for the alert, with the port or stream check as the escalation. What you must not do is skip the check because the camera "belongs to security." A device on your network with no availability check belongs to nobody at exactly the moment it matters.


The NVR: where footage lives or dies


The NVR (network video recorder) is the box that receives every camera's stream and writes it to disk, around the clock. It is the single most important device in the entire system, and the most abused. It performs constant heavy disk writes for years in a warm closet. That is roughly a torture test for hard drives. A dead camera loses one view; a sick NVR loses all of them, including the past.


Monitor the NVR as a device first: reachability. Where it exposes them via SNMP, an API, or its web interface, monitor CPU, temperature, and above all disk health. An API is an interface that lets another program ask it questions. Many recorders surface drive warnings (failing drives, degraded arrays) somewhere in their interface. A warning light in a closet has an audience of coats. Get that signal out of the closet. Use an SNMP query, an email the NVR sends that your monitoring watches for, or a periodic human check written into a runbook. Whatever the box supports, choose loud over elegant.


Rank it accordingly in your alerting. A single dead camera is a next-business-day problem. A dead or wedged NVR is fleet-wide blindness that keeps getting worse by the minute. That is true even while every camera on the dashboard glows green. They are all faithfully streaming into a void. The NVR deserves the same alert urgency as a core server. It also deserves a spot on the UPS (uninterruptible power supply, the battery that rides out short outages). A recorder that loses power mid-write is how arrays get corrupted. And the NVR deserves a place in your dependency tree as the parent of every camera check. Then an NVR outage pages you once with the right headline instead of burying it under a pile of per-camera noise.


Then monitor what the NVR is doing: per-camera record state, where exposed. Most recorders know, per channel, whether they are currently receiving and writing video, and many will show or report it. This is the closest thing to level-3 proof for the whole fleet at once. A camera can be up, streaming, and still not be recorded. That can happen because someone's configuration change quietly disabled that channel, or a motion-only recording rule stopped matching. "Camera up" and "camera recorded" are separate facts. Monitor whichever of the two the equipment will give you, and both where you can.


Retention: days-of-footage is the honest metric


Somewhere there is a policy — written or assumed — that footage is kept for a certain number of days. The NVR does not enforce that policy; it enforces disk space. Recorders overwrite the oldest footage when the disk fills. So actual retention is whatever the disk size, camera count, and per-camera bitrate happen to produce. Bitrate is the amount of data each camera sends per second. Retention shrinks every time someone adds a camera, raises a resolution, or a disk in the array dies and capacity drops. Nobody notices, because the recorder keeps recording. It just remembers less.


So measure the real thing: days of footage actually on disk. Open the recorder and check the timestamp of the oldest available recording, per camera or per group. That number against the policy number is the entire honest conversation about retention:


retention policy:        30 days
oldest footage on disk:  11 days   (checked monthly, recorded in the log)

verdict: system does not meet policy — capacity or bitrate must change,
or the policy must be rewritten to say what the system actually does

Track it monthly and watch the trend: oldest-footage-age drifting downward is capacity quietly running out ahead of an uncomfortable question. And when the number and the policy disagree, force the conversation with the system's owner. Either the hardware grows or the policy shrinks, but the two must match, in writing. A policy the equipment cannot meet is a liability document waiting for its moment.


Tip: when a camera is added or its quality settings raised, recalculate retention that week — do not wait for the monthly check. One new high-bitrate camera on a full recorder shortens everyone else's history immediately. And "we added a camera and lost a week of retention" is a trade someone should get to approve rather than discover.


The bandwidth reality


A camera network has a traffic shape unlike anything else you run. Every camera pushes a constant upstream flow toward the recorder, all day, every day. There is no idle time and no burstiness to average away. Add up the per-camera bitrates and that sum flows continuously through every link between cameras and NVR. A handful of cameras is a rounding error. A serious fleet is a permanently occupied slice of every uplink it crosses. The uplink is the cable from an edge switch back toward the core. And the slice grows with each resolution upgrade.


The planning math is mercifully simple, and worth doing before the switch uplink does it for you. On the slower uplink in the worked example below, the cameras have taken more than half the link before any other traffic on that switch has said a word:


per-camera stream:            4 Mbps (typical mid-quality; check your actual settings)
cameras on one edge switch:   14
constant load on that uplink: 14 x 4 = 56 Mbps, all day, every day

uplink: 1 Gbps  → comfortable
uplink: 100 Mbps → more than half consumed by cameras alone, before
                    any other traffic on that switch is counted

Recording mode changes the math, so know which one you are running. Continuous recording is the sum above, all day. Motion-triggered recording averages lower — but it peaks at exactly the busy moments when every camera sees movement at once. Motion-triggered recording trades bandwidth for a different risk. A mistuned motion rule that records nothing is a silent failure your retention check will not catch until the gap is discovered. If you run motion-based recording, the monthly spot-check below stops being optional and becomes the main line of defense.


Two design consequences follow. First, keep camera traffic's path short and known. Ideally cameras and NVR share a switch or a dedicated segment, so the constant load crosses as few shared links as possible. Second, monitor utilization on every link the camera flow crosses. A link quietly running at a high constant floor has little headroom left for everything else. Judging how full is too full for a given link is its own subject, covered in bandwidth utilization guidance. When cameras saturate a link, the symptom is ugly in a specific way: dropped frames and gaps in the recording. These are discovered — as always with cameras — later, when the footage is needed.


At Northgate Retail the branch recorders had been set up by the installer to copy each day's footage to a central archive overnight. Nobody on the network side had been told. For a little over a year the camera system was the largest talker on the WAN between midnight and five, every night. The WAN means the wide-area links between the branches and head office. The overnight stock-file sync to the branches failed often enough to be investigated as a server problem for three months. The first flow report anyone ran for that window listed the recorders at the top, in order of branch size. The review called the archive job "a known process", which was true of one person. It now runs under a rate cap, and the stock files arrive.


PoE dependency chains


Most cameras take their power from the switch through the network cable: PoE (Power over Ethernet). This is wonderful for installation and sneaky for troubleshooting, because it welds two failure domains together. A camera can die for camera reasons, or because the switch stopped powering it. For example, the port failed, someone disabled PoE, or — the classic — the switch's shared power budget ran out and it shed the port. Power budgets fill up gradually as devices are added, then tip over when something asks for a little more on a cold morning. The mechanics and the monitoring are covered properly in PoE power budgets.


For the camera fleet, draw the dependency explicitly in your monitoring: these twelve cameras hang off this switch, which powers them. Then two good things happen. During an outage, one glance separates "a camera died" from "a switch took a dozen cameras with it" — radically different responses. And with dependencies declared, your monitoring can alert once on the switch instead of twelve times on its children. That is the difference between a page you act on and a page storm you dig through. Per-port PoE draw doubles as the best liveness signal a camera has. A camera's normal draw is a heartbeat. Zero watts on a camera port is a dead camera announced by the switch, whether or not the camera ever answered pings.


The monthly footage spot-check


Every check so far watches the plumbing. None of them can see the one thing the system exists to produce: a usable image. A camera can be up, streaming, recording. Yet it can be nudged toward the ceiling by a ladder, blinded by a new lamp, fogged by a failed seal, or wearing a spider. Machines are bad at noticing this; eyes are excellent. So close the loop with a deliberately boring human ritual, once a month, fifteen minutes:


  • 1. Pull up each camera's live view — every camera, not a sample. Confirm the image shows the view its name promises, in focus, correctly aimed, well lit.
  • 2. Pull one recorded clip per recorder from a random recent day and hour, and actually play it. This proves the write-read path, not just the write.
  • 3. Check the timestamp overlay against a known clock. Footage with a wrong clock has caused more evidentiary grief than footage missing entirely. Recorders drift when nobody points them at a time source.
  • 4. Note oldest footage age per recorder (the retention number from above) in the same log.
  • 5. Record the check — date, who looked, exceptions found — somewhere findable. The log itself becomes evidence that the system is cared for.

This is the camera equivalent of the quarterly test restore for backups. It is a small ritual that catches the entire class of failures your network checks are structurally blind to. Put it on the calendar with an owner's name on it, or it will be enthusiastically done twice and never again.


Ownership and privacy, honestly


Cameras watch people, and that makes them policy objects, not just devices. Somewhere, explicitly, it should be written down who owns the camera system. The record should say who is allowed to view live video and recorded footage and under what process. It should say how long footage is kept (the policy your retention number must match). And it should say what notice is given to the people being recorded. The specifics are jurisdiction- and organization-dependent and belong to management, HR, and legal — not to whoever happens to hold the admin password. Your job as the network person is narrower and important. Know who the owner is. Route every "can I see the footage?" request to them rather than answering it yourself. And keep your monitoring pointed at availability and health — not at content. A camera system with clear ownership and boring, documented process is an asset. The same hardware run informally is a liability with a lens.


Where to go next


Here is the pattern to take away. Prove each camera is alive (three levels of proof). Prove the recorder is healthy and actually keeping footage (days-on-disk versus policy). Respect the power and bandwidth physics underneath, and let a human look at actual images once a month. Do that, and "are the cameras okay?" gets answered from a desk on an ordinary Tuesday, which beats answering it in the parking lot. Camera fleets are also how most of us first meet a bigger subject: the rest of the sealed, vendor-controlled gear accumulating on your network. The other guides in IoT, Cameras & Building Systems cover it: the general outside-in toolkit, building systems landing on IP, and holding vendor boxes accountable.


Put this into practice with SysPeep Monitor

A camera fleet is exactly the kind of thing that fails silently until the day it matters. SysPeep Monitor discovers the cameras and switches on your network. It runs ping and port checks against every camera. It reads PoE and port state from your switches over SNMP. It understands dependencies — so a dead switch pages you once, not twelve times. Free trial, installs on any Windows machine.

Download Free TrialLearn About SysPeep