A health check is not a recovery plan

The green endpoint is not the green light

When a home server has a health endpoint, a dashboard tile or a container probe, it is tempting to treat a passing response as a small promise: the service is fine. A failing response then feels like the opposite promise: restart it and move on.

Neither interpretation is complete. A health check is an observation made from one place, at one time, using one definition of “working”. A recovery plan is a sequence of decisions for when that observation is wrong, incomplete or noisy. The first can be automated. The second still needs an owner, limits and evidence.

Name the question before naming the endpoint

Different checks answer different questions. Calling them all `/health` makes it easy to attach a dangerous action to a weak signal.

Docker's `HEALTHCHECK` gives a container a health state that begins as `starting`, becomes `healthy` after a passing check and becomes `unhealthy` after the configured number of consecutive failures. That is valuable operational information, but it is still one health signal. Docker's restart policies are configured separately and are described in terms of containers stopping or exiting. Do not assume that a red health state contains the recovery policy you meant to have.

Kubernetes makes the distinction more explicit. A failed liveness or startup probe can cause the kubelet to restart a container; a failed readiness probe marks it not ready and removes it from matching Service endpoints. That is a useful separation between “try to recover this process” and “stop sending work here”. It is not a complete incident response. Neither action repairs corrupted data, explains a bad configuration, chooses a rollback, or tells a person that the dependency is down.

• **Liveness:** is the process alive and able to make progress? A cheap local check might answer this without touching a database or a remote API.
• **Readiness:** should this instance receive normal traffic? This can include a required local dependency, a loaded configuration or a temporary maintenance state.
• **Startup:** has a slow application finished its initial work? It prevents a normal liveness check from declaring a still-starting service dead.

A restart is one possible action, not the definition of recovery

Restarting is attractive because it is quick and sometimes exactly right. A stuck process may become useful again after a clean start. A temporary connection problem may clear while the service is coming back.

But a restart can also hide the cause, discard useful in-memory evidence or make an overloaded machine work harder. Kubernetes explicitly warns that an incorrectly designed liveness probe can create cascading failures by restarting containers under load. A check that includes every dependency can make a healthy process look dead whenever one dependency is slow. A check that is too shallow can stay green while every useful request fails.

The control should therefore map each signal to a bounded action:

1. Observe the failure and record the time, service version and recent check history. 2. Decide whether to keep serving, remove the service from traffic, restart it, or leave it alone while investigating. 3. Limit automatic attempts. A retry budget is a safety control, not an admission of defeat. 4. Preserve enough evidence to distinguish a process fault from a dependency, capacity, storage or configuration fault. 5. Escalate to a named person when the budget is exhausted, and state the next safe action.

If those five decisions exist only in somebody's memory, the health check is carrying authority it cannot safely hold.

A worked example: a small household dashboard

Imagine a fictional home dashboard that reads a local data store and displays energy history. It is not a public service, and its most important property is that it remains understandable when it fails.

The operator could define the checks like this:

Then write the recovery card beside the service definition:

The exact thresholds are an example, not a universal home-server setting. The important part is that the threshold, action, evidence and stop condition are written down before the outage.

• `/health/live` returns success only when the application process can accept a small request and complete it promptly. It does not run a write or depend on the data store.
• `/health/ready` returns success only when the application has the configuration and data-store connection required for a useful page. A failure keeps the service out of normal traffic but does not automatically prove the process needs a restart.
• A startup allowance covers the known time needed to load the application. The check is not tuned so tightly that a cold start looks like a deadlock.
• One failed readiness check: record it and let the next check decide whether the condition persists.
• Repeated liveness failures: allow a small, documented number of restarts.
• Repeated restarts or a readiness failure that lasts beyond the agreed window: stop automatic retries and notify the owner with the last known version, dependency result and recent logs.
• If the failure began immediately after a change, compare the current version with the last known good version. Roll back only through a tested, reversible procedure; do not make “rollback” a reflex for every dependency outage.
• Recovery is proved by a real read-only user path, not only by a green process endpoint.

Keep checks small and recovery evidence useful

A probe should be cheap enough to run often and safe enough to run when the system is stressed. It should not mutate data, print secrets or depend on a long chain of services unless that dependency is part of the question being asked. Server Attic's preference is to keep HTTP health endpoints dedicated and minimal; that is a useful habit outside Kubernetes too. Kubernetes' documentation separately notes a common liveness pattern that reuses the same low-cost HTTP endpoint as readiness, with a higher `failureThreshold`.

Keep the evidence human-sized:

This is more useful than a dashboard full of red dots with no explanation of what anyone is expected to do. A log line can tell you that a restart happened. A recovery record tells you whether the service became useful again and what changed in the plan.

• the first and most recent failure time;
• which check failed and from where;
• the running version or configuration revision;
• whether the process was restarted, drained or left untouched;
• the last known good user-level verification;
• the next action and its owner.

Know where this advice stops

This is an operating pattern, not a promise that probes prevent outages. It does not choose the right uptime target, backup retention, alert channel or data-repair procedure for every household. It also does not mean every service needs three endpoints or an orchestration platform. A single local process can still have a clear distinction between “alive”, “useful” and “what we do next”.

At Server Attic, the next practical step would be to choose one existing service and write its recovery card before adding another monitor: what does the check prove, what action follows, how many automatic attempts are safe, what evidence is retained, who owns the next decision, and how will usefulness be verified? A health check can tell you that something changed. A recovery plan turns that signal into a safe response.

Sources and limits

This article synthesises the sources below into a practical explanation. It is not a security standard, legal advice, or a guarantee that guidance current at review time still applies — check the review date above against your own situation.

1. Dockerfile reference — Docker. Describes HEALTHCHECK status, consecutive failures, startup timing and health-status events.
2. Start containers automatically — Docker. Describes restart policies as actions for containers that exit or stop, including their limits and interaction with process managers.
3. Liveness, Readiness, and Startup Probes — Kubernetes. Distinguishes startup, liveness and readiness probes, their actions and the risk of cascading failures from incorrect liveness checks.

More from Server Attic

• 2026-08-07

  A firewall rule is not a network boundary

  A practical OPNsense-inspired checklist for separating trust zones, writing understandable rules, testing the path and keeping a recovery route when a firewall change goes wrong.

• 2026-07-18

  A backup is not a restore plan

  A practical home-server guide to turning backup jobs into evidence you can actually recover from, using restore drills, integrity checks and clear limits.

• 2026-08-28

  Automate disk cleanup only after you can explain the growth

  A full disk invites a hurried deletion. A safer home-server workflow identifies the owner, growth rate, reclaimable data, retention need and recovery path before cleanup becomes automatic.