Pushing Events In: Feeding External Signals into Monitoring
Your monitoring system is very good at asking questions. Every few minutes it asks each switch about its ports, each server about its disks, each service whether it is answering. But there is a whole class of things it cannot ask about, because nobody is there to answer. Those include whether last night's backup finished or whether the deployment at four o'clock succeeded. They also include whether the cloud provider is warning you about something or whether the temperature sensor in the server room is still reporting. Those facts exist, but they live outside anything the monitor can poll.
The fix is to turn the conversation around. Instead of the monitor asking, the outside world tells. The backup script reports its own result, and the deployment tool announces what it did. The cloud alert lands as an event, and the sensor gateway posts a reading. Once those signals are inside the monitor, they get everything the monitor already does well: alert rules, escalation, dashboards, history, and one place to look. This guide explains the idea behind inbound events and the doors a monitor might expose for them. It explains how to design a payload that is actually useful and how to stop the same event paging you twice. It also explains how to keep the door from becoming a hole. The examples use invented endpoints; your product's documentation is the map to the real ones.
In this guide
- Why push signals into the monitor at all
- The passive check: the monitor accepts a result instead of asking
- The doors a monitor may expose
- Designing a payload that is useful at three in the morning
- Deduplication and idempotency
- A worked example: a backup job reports itself
- Securing the inbound door
- Sources, methods, and payloads at a glance
- Where to go next
Why push signals into the monitor at all
You could leave those outside facts where they are. The backup software has its own console; the deployment tool has its own log; the cloud provider has its own alert emails. The trouble is that each of them is a separate place to look, with its own idea of severity and its own notification settings. Each also has its own way of being ignored. The morning after a bad night, the story is scattered across five inboxes and three dashboards. The person on call was paged by none of them.
Pushing those signals into the monitoring system buys you one thing above all: reuse. You reuse the alert rules you have already tuned and the escalation paths you have already agreed. You reuse the dependency logic that already knows the backup server sits behind a particular switch. You reuse the history that already lets you say "this has happened four times this quarter." And you reuse the dashboard the whole team already watches. Each external signal that becomes a monitoring event inherits all of that for free. The sources that most often earn a place:
- Backup and batch jobs. Nightly backups, database maintenance, report generation, data transfers. Their success or failure is the single most common thing admins want in the monitor and the single most common thing left out.
- Deployment and change events. "Version X went to server Y at time Z" is priceless context next to the alert that fired two minutes later.
- Cloud provider and hosted-service alerts. Warnings about quotas, billing thresholds, or a degraded region that arrive as email or webhook and otherwise go unread.
- A script's own health. Every scheduled integration you build should report "I ran, and here is how it went," so that a script that silently stops is noticed.
- Physical sensors and gateways. Temperature, humidity, door contacts, power, usually reachable only through a gateway that can post a reading or send a message.
The passive check: the monitor accepts a result instead of asking
Monitoring people call the normal kind of check active: the monitor initiates, the target answers. An inbound event is the mirror image, a passive check: the monitor sits still and accepts a result that something else delivers. Think of a school register. In an active check, the teacher calls each name and waits for "present." In a passive check, students sign a sheet on the way in, and the teacher reads the sheet. Both produce an attendance record; the difference is who does the work and when.
The sign-in sheet has one weakness the roll call does not. If a student never signs in, the sheet says nothing about them. It does not say "absent". It simply has no entry. A passive check has the same gap: when results stop arriving, the monitor has no new information. And "no new information" is easy to mistake for "nothing wrong." So every passive check needs a second rule alongside it, a freshness or expected-result rule: "I expect a backup result every twenty-four hours; if none arrives within twenty-six, raise an alert." Silence becomes a signal. That is the same reasoning that protects you when the monitor itself dies, which the guide on monitoring system failure modes covers. Here it protects you from a backup script that was never scheduled after the server was rebuilt.
Many products let you configure that freshness window directly on a passive check. If yours does not, the same effect comes from a small active check that reads the age of the last received result. That check complains when the result is too old. Either way, the rule is the half of the design people forget, and it is the half that catches real failures.
The doors a monitor may expose
Products differ widely in how they accept inbound signals. These are the common doors, roughly from most to least structured. Check which ones yours actually has before designing around one.
An inbound API call
The cleanest door. The monitor's API has an endpoint that accepts a result or event. Typically this is a POST with a JSON body naming the object, its state, and a message. Your script makes one HTTP request at the end of its work. Everything about the payload is under your control, which is both the strength and the responsibility.
A webhook receiver the monitor exposes
Some monitors publish a URL that other systems can call: the cloud provider, a hosted service, a deployment tool. The monitor receives the sender's payload and maps it to an event. The shape is decided by the sender. So the work here is configuring the mapping: which field is the host, which is the severity, which is the message.
Email-to-event
The monitor watches a mailbox and turns messages into events, matching on sender, subject, or body. It is the least structured door and the most universal: every backup product, every cloud service, and every appliance can send email. The cost is fragility. A vendor rewording a subject line can silently break the match. So build the rules on the most stable fields you can find and alert on "expected email never arrived."
A file drop, a syslog message, or an SNMP trap
The oldest doors. A script writes a status file the monitor reads on its next poll. A device or script sends a syslog line the monitor parses. A device emits an SNMP trap the monitor receives. These are not APIs in the HTTP sense, but they serve the same purpose. On a monitor without an inbound API they are often the practical route. For example, a script writes backup-status.txt containing OK or FAIL plus a timestamp, and a file-content check reads it every few minutes.
Designing a payload that is useful at three in the morning
Whatever the door, the event you send should answer the questions the on-call person will ask when it wakes them. A minimal but complete inbound event carries these fields:
| Field | Why it matters | Example |
|---|---|---|
source | Which integration sent this, so you know where to look when it is wrong | backup-wrapper |
host or object | The thing this is about; must match how the monitor names it, or the event will not attach | fs-01 |
check | The named check on that host, so repeated events update one thing rather than making many | nightly-backup |
state | A small fixed set the monitor understands: ok, warning, critical, unknown | critical |
message | One human sentence with the fact and the number | "Backup failed: 3 of 12 volumes skipped, disk full on target" |
timestamp | When it happened, not when it was sent, in a standard format with a time zone | <timestamp> |
event_id | A unique identifier so a repeated delivery is recognised as the same event | fs-01-nightly-backup-<run id> |
details | Optional extras: a log path, a job id, a link back to the source system | "log": "\\\\fs-01\\logs\\backup.log" |
Put together, against an invented endpoint:
POST https://monitor.example.local/api/events
Authorization: Bearer <TOKEN>
Content-Type: application/json
{
"source": "backup-wrapper",
"host": "fs-01",
"check": "nightly-backup",
"state": "critical",
"message": "Backup failed: 3 of 12 volumes skipped, disk full on target",
"timestamp": "<timestamp>",
"event_id": "fs-01-nightly-backup-<run id>",
"details": { "job_id": "<run id>", "log": "\\\\fs-01\\logs\\backup.log" }
}
Three design rules follow from that shape. Keep state to the monitor's own vocabulary; do not invent "sort-of-failed." Keep host and check stable across runs. They are the key the monitor uses to say "this is the same thing as last time." A check named after today's date is a new check every day. And write the message for the reader, with the fact first and the number in it. The message "job finished with errors" is useless. The message "3 of 12 volumes skipped, disk full" tells the on-call person whether to get up.
Deduplication and idempotency
Networks retry. Scripts get run twice by an over-eager scheduler. A webhook sender that did not get a fast enough reply sends again. Sooner or later the same event arrives twice. If each arrival creates a fresh alert, the on-call phone rings twice for one problem. In that case, trust in the system erodes a little more. Two ideas prevent that.
Deduplication is the monitor's job of recognising a repeat. It works on the key you gave it. If two events share the same event_id, or the same host and check and state within a short window, the second updates the first rather than creating a new one. This is why the payload has an id, and why host and check names must be stable.
Idempotency is your side of the same bargain: design the sending so that delivering it twice has the same effect as delivering it once. The practical technique is to send state, not news. "The nightly backup on fs-01 is critical" can be sent a hundred times and the world is the same after each. "A backup failed!" sent twice looks like two failures. Most monitors think in terms of state per check. So send the current state on every run, including ok on success. Let the monitor decide when a change is worth an alert. That also gives you the clear for free. The next successful run sets the state back to ok, and the alert resolves without anyone clicking anything.
Tip: if your door is email or a file drop rather than an API, the same principle applies. Overwrite one status file rather than appending to it; send one email per run with a stable subject rather than one per error line. The monitor cannot deduplicate what it cannot recognise as the same thing.
A worked example: a backup job reports itself
Here is the whole idea in one script. Suppose the backup runs from a scheduled task that calls backup.exe. Wrap it: run the backup, capture its exit code, and post the result. In PowerShell:
$token = Get-Content "$env:ProgramData\monitor-push\token.txt" -Raw
$headers = @{ Authorization = "Bearer $token"; "Content-Type" = "application/json" }
$runId = [guid]::NewGuid().ToString()
& "C:\Backup\backup.exe" --profile nightly *> "C:\Backup\logs\$runId.log"
$code = $LASTEXITCODE
$state = if ($code -eq 0) { "ok" } else { "critical" }
$body = @{
source = "backup-wrapper"
host = $env:COMPUTERNAME.ToLower()
check = "nightly-backup"
state = $state
message = "Backup exit code $code; log $runId.log"
timestamp = (Get-Date).ToUniversalTime().ToString("o")
event_id = "$($env:COMPUTERNAME)-nightly-backup-$runId"
} | ConvertTo-Json
try {
Invoke-RestMethod -Uri "https://monitor.example.local/api/events" `
-Method Post -Headers $headers -Body $body -TimeoutSec 30
} catch {
"$(Get-Date -Format o) push failed: $($_.Exception.Message)" |
Add-Content "C:\Backup\logs\push-failures.log"
}
exit $code
And the same in shell with curl, for a Linux host:
#!/bin/sh
run_id=$(date +%s)-$$
/usr/local/bin/backup --profile nightly > "/var/log/backup/$run_id.log" 2>&1
code=$?
state=ok; [ "$code" -ne 0 ] && state=critical
curl -s --fail-with-body -X POST "https://monitor.example.local/api/events" \
-H "Authorization: Bearer $(cat /etc/monitor-push/token)" \
-H "Content-Type: application/json" \
-d "{\"source\":\"backup-wrapper\",\"host\":\"$(hostname -s)\",\"check\":\"nightly-backup\",\"state\":\"$state\",\"message\":\"Backup exit code $code\",\"event_id\":\"$(hostname -s)-nightly-backup-$run_id\"}" \
|| echo "$(date -Is) push failed" >> /var/log/backup/push-failures.log
exit $code
Walk through what this buys. The monitor now has a check called nightly-backup on each backup host. On a good night it goes ok. On a bad one it goes critical with the exit code and a log filename in the message. On that bad night, the ordinary alert rules page whoever should be paged. Then add the freshness rule: "if nightly-backup on any host has not received a result in twenty-six hours, warn." That single rule catches the disabled scheduled task, the rebuilt server nobody re-registered, and the wrapper that crashed before it could post.
Notice the two small courtesies in the script. A failure to reach the monitor is logged locally rather than allowed to break the backup. That is because a push that cannot be delivered should never make the underlying job worse. And the wrapper exits with the backup's own exit code, so anything else watching the task still sees the truth.
Securing the inbound door
An inbound endpoint is a door you have opened in a system that decides who gets woken up. Anyone who can post to it can create alerts. Worse, they can post ok for a check that is actually failing, silencing a real problem. The minimum precautions:
- A token per source. The backup wrapper has its own token; the deployment tool has its own; the sensor gateway has its own. When one leaks or misbehaves, you revoke one door, not all of them.
- Write-only, narrow scope. A token that may post events should not also be able to read the inventory or acknowledge alerts, if the product lets you separate those.
- An allowlist. Accept posts only from the addresses that should be sending them, at the firewall or in the monitor's own settings.
- TLS, always. The token travels in every request; without encryption it can be read off the wire.
- Validate the payload. Reject events for hosts the monitor does not know, states it does not recognise, and messages of absurd length. Garbage in should produce a log line, not an alert.
That is the short version. Signature verification for webhooks, rotation, logging who posted what, and what to do when a token leaks are in API security and hygiene. It is worth reading before the first inbound door goes live.
Sources, methods, and payloads at a glance
| Source | Likely method | What the payload should carry | Freshness rule |
|---|---|---|---|
| Backup or batch job | Wrapper script posts to the inbound API, or writes a status file | host, check name, ok/critical, exit code, log path | Expect one result per scheduled run, plus margin |
| Deployment tool | Its outbound webhook to the monitor's receiver, or a post at the end of the pipeline | target hosts, what changed, who triggered it, success or failure | None; deployments are irregular. Use as context, and optionally open a maintenance window |
| Cloud or hosted-service alert | Email-to-event, or the provider's webhook to the monitor's receiver | service name, severity mapped to the monitor's states, the provider's message and link | Optional heartbeat email if the provider offers one |
| Integration script's own health | Post ok/critical at the end of every run | script name as check, run duration, rows processed | Expect a result every interval; this is the heartbeat |
| Sensor gateway | SNMP trap, syslog, or a post with a numeric value | sensor id as check, the reading, units | Expect a reading every few minutes; missing readings mean a dead gateway |
| Appliance with email only | Email-to-event on a stable subject line | whatever the appliance sends; map sender to host | Expect the daily summary email; alert if absent |
Where to go next
The monitor can only alert on what it knows, and a great deal of what matters happens where it cannot poll. Pushing events in closes that gap. A script or a system delivers a result, and the monitor treats it as a passive check. A freshness rule turns silence into a signal. A stable host-and-check key lets the monitor deduplicate. Sending state rather than news makes retries harmless. And a narrow token behind an allowlist keeps the door from becoming a hole. The backup wrapper above is a good first project. The second is usually your own integration scripts reporting their health, so that the glue never fails silently.
The read side of the same API is in pulling data out. Turning inbound and outbound events into tickets, chat messages, and remediation is the subject of gluing systems together. The whole set lives under Monitoring APIs and Webhooks.
Put this into practice with SysPeep Monitor
A monitor's collectors are its inbound doors. SysPeep Monitor gathers health and performance data over SNMP, WMI, SSH, and similar collectors. So a status a script leaves where those collectors can see it becomes a check like any other. It has thresholds, dependencies, and email and other notifications on top. Check the product documentation for exactly what each collector can read. The free trial is the quickest way to find out.
Download Free TrialLearn About SysPeep