Why we check every thirty seconds

Ask a monitoring vendor why they check every five minutes and you will hear a story about noise. Shorter intervals, the story goes, produce false alarms. That story was true in 2012, when a single probe in a single data centre decided whether your site was up. It is not true now, and the interval survives because five-minute checks are cheap to run and expensive to explain away.

Carbon checks every thirty seconds. This post is about what that costs us, what it buys you, and why the false-alarm argument has it backwards.

The arithmetic of a five-minute check

A five-minute interval means an outage can run for four minutes and fifty-nine seconds before the first probe notices. Most vendors then require a second failed check before paging, which is another five minutes. Add the time for a human to wake up and you have a mean time to acknowledge of twelve minutes for an outage that began the instant the previous check passed.

Twelve minutes is longer than most outages. It is certainly longer than the ones that matter, because the outages that matter are the ones customers notice, and customers notice in seconds.

Confirming from a second region

The false-alarm problem is real, but the fix is not a longer interval. The fix is a second opinion. When a probe in Frankfurt sees a 502, Carbon asks Dublin and Virginia to check the same endpoint immediately. If either agrees, the incident opens. If neither does, the Frankfurt result is recorded as a regional blip and nobody is paged.

The gap between the first failure and the confirming check is under ten seconds. So the interval that matters, the one between your site going down and a human being told, is about forty seconds on a bad day.

What it costs

Ten times the checks means ten times the probe traffic, and we pay for that. It is the single largest line in our infrastructure bill. We think it is the right line to have there, because everything else the product does, the screenshots, the traceroutes, the status page, the incident timeline, is worth less if it starts twelve minutes late.

A monitor's job is not to tell you that your site was down. It is to tell you before anyone else does.

When thirty seconds is too often

There are endpoints that should not be hit twice a minute from twelve places: a login page with rate limiting, a search that costs money to run, a webhook receiver. For those, Carbon offers three-minute and ten-minute intervals, and heartbeat monitors that wait for your system to check in rather than the other way round. The default is thirty seconds because the default should be the one that protects you.