The alert you always ignore

The notification you swipe away without reading

Most home servers do not fail from a lack of monitoring. They fail with a phone full of alerts that nobody reads any more.

The pattern is familiar. You install something that watches the machine, it ships with a generous set of default alarms, and for a fortnight you inspect every one. Then a disk sits at 82% for a month, a container restarts nightly and recovers on its own, and a sensor crosses its threshold every afternoon in July. You learn, quite rationally, to swipe them away — and when something genuinely breaks, that alert looks exactly like the noise you have been dismissing.

This is not a discipline problem. It is a design problem, and it has a well-documented shape.

An alert is a request for a decision

Google's site reliability engineering chapter on monitoring, written by Rob Ewaschuk, sets a deliberately hard bar: "Every page should be actionable." It goes further — "Every page response should require intelligence. If a page merely merits a robotic response, it shouldn't be a page."

That second sentence is the useful one at home. If your response to an alert is always the same three commands, that is not a notification, it is an automation you have not written yet. If your response is nothing at all, it is not a notification either.

The chapter also names the cost of getting this wrong: when pages occur too frequently, people "second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page." The noisy alert does not just waste your attention. It quietly disables the alert next to it.

Symptoms interrupt you, causes explain themselves

The same chapter frames monitoring as answering two questions — what is broken, and why — and is explicit that the first deserves far more effort. The Prometheus project's alerting guidance puts it operationally: aim to have as few alerts as possible by alerting on symptoms associated with end-user pain, rather than trying to catch every possible way that pain could be caused. Keep alerting simple, it advises, and use good consoles to pinpoint causes.

Translate "end-user pain" into a household and it becomes concrete. Photos will not sync. The media library is empty. The door sensor stopped reporting. Those are symptoms, and there are far fewer of them than there are causes.

The causes — CPU load, memory pressure, a failed cron, a saturated link — still belong on a dashboard, where you go looking once a symptom has your attention. They just do not deserve a notification each. Prometheus makes a narrower version of the same point: page on latency at one point in a stack, because if a lower-level component is slow while the overall experience is fine, there is nothing to do.

Four numbers to argue with instead of feelings

The Site Reliability Workbook gives a vocabulary for tuning an alert, which beats debating whether a threshold "feels" right. It evaluates an alerting strategy on four attributes:

They trade off, and the workbook is candid about it: widening the window an alert evaluates improves precision but worsens reset time, so a resolved outage keeps notifying you long after it ended. Long reset times, it notes, can lead to confusion or to issues being ignored — the same failure we started with, arriving from the other direction.

Most household alert fatigue is a precision problem being solved with willpower. Naming it as precision tells you what to change.

• Precision — "the proportion of events detected that were significant". It is 100% if every alert corresponds to a significant event.
• Recall — "the proportion of significant events detected". It is 100% if every significant event results in an alert.
• Detection time — how long it takes to notify you once something is genuinely wrong.
• Reset time — how long alerts keep firing after the issue is resolved.

Before you delete the alert, try the levers

Monitoring tools generally give you more than on and off. Netdata's health configuration is a useful example of what to look for in whatever you run — an illustration, not a recommendation.

Tune in that order. Delete only the alerts that survive tuning and still have no action attached.

• Sticky thresholds. Netdata documents a hysteresis pattern where the level that raises an alert is higher than the level that clears it, explicitly to prevent alert spam when a value fluctuates around a boundary. A metric hovering at the line then produces one alert, not forty.
• A delay before notifying. Its delay setting takes separate up and down delays, a multiplier for repeated state changes and a maximum ceiling, to prevent notification floods. This is Prometheus's advice to allow slack for small blips, expressed as configuration.
• A deliberate repeat interval. A repeat setting controls how often an unresolved alert nags you. Silence is a decision; so is every fifteen minutes.
• Routing. Alerts go to named roles rather than to everyone — usually a choice between a channel you check daily and one that wakes you.

A short checklist for every alert you own

Six questions, one alert at a time. An alert that cannot answer them is a candidate for removal, not for a stricter threshold.

• Which symptom does this represent, and who notices it?
• What will I actually do at 23:00 when it fires?
• If the answer is always the same commands, can I automate the response instead?
• How long may this condition persist before it matters — and does the delay reflect that?
• When the problem clears, does the alert clear, and how quickly?
• If this fired ten times last month and I acted zero times, why does it still exist?

What this does not fix

Silence is not health. Everything above tunes alerts that fire. It does nothing about the failure that never announces itself — a backup job that stopped running, a sensor that went quiet. Those need a check that fails when something is absent, which is a different design. A backup is not a restore plan covers the version of this problem we see most often.

A household is not an on-call rotation. Google's guidance assumes a team, a pager and someone awake. You have one person and a phone on a bedside table. "Actionable" has to mean actionable by you, tired, without a runbook you never wrote.

Thresholds are not a substitute for design. If an automation fails badly whenever it fails, no threshold makes it safe — the recovery behaviour has to change. Home Assistant automations that fail clearly is the companion piece.

What we would do next

Open your alert history for the last month. For each alert that fired, write down what you did. Every row where the answer is "nothing" is a candidate: tune it with hysteresis and a delay, convert it into a scheduled check you read on your own schedule, or remove it.

The goal is not fewer notifications for their own sake. It is a small set of alerts you still trust at three in the morning.

Sources and limits

This article synthesises the sources below into a practical explanation. It is not a security standard, legal advice, or a guarantee that guidance current at review time still applies — check the review date above against your own situation.

1. Monitoring Distributed Systems, Site Reliability Engineering — Google. Rob Ewaschuk's chapter states that every page should be actionable and should require intelligence rather than a robotic response, that monitoring should answer what is broken (the symptom) and why (the cause) with much more effort spent on symptoms, and that when pages occur too frequently people second-guess, skim or ignore alerts including real ones.
2. Alerting on SLOs, The Site Reliability Workbook — Google. Defines the four attributes used to evaluate an alerting strategy — precision, recall, detection time and reset time — and shows that they trade off against each other, for example that lengthening an alerting window to improve precision worsens reset time because alerts keep firing after the problem is resolved.
3. Alerting, Practices — Prometheus. Advises aiming to have as few alerts as possible by alerting on symptoms associated with end-user pain rather than every possible cause, keeping alerting simple with good consoles for pinpointing causes, avoiding pages where there is nothing to do, allowing slack in alerting to accommodate small blips, and paging on latency at only one point in a stack.
4. Configure Health Alerts — Netdata. Documents the warn and crit expressions that trigger each severity, a hysteresis pattern that makes thresholds sticky to prevent alert spam when values fluctuate around a threshold, a delay line with up, down, multiplier and max settings to prevent notification floods, a repeat interval for alerts held in warning or critical, and a to line naming the roles that receive the notification.

More from Server Attic

• 2026-07-27

  What AdGuard Home can and cannot fix

  What network-wide DNS filtering genuinely solves, the ceiling AdGuard's own documentation sets out, and how encrypted DNS and Encrypted Client Hello change the picture.

• 2026-07-22

  Make home automations fail safe first

  A practical checklist for designing Home Assistant automations that recover clearly, leave evidence and avoid doing the wrong thing twice.

• 2026-07-18

  A backup is not a restore plan

  A practical home-server guide to turning backup jobs into evidence you can actually recover from, using restore drills, integrity checks and clear limits.