Why we stopped booking staging

At my last company I was an engineering manager, and for about two years I was also the person who kept the staging calendar. Nobody gave me the job. I made a shared spreadsheet one Tuesday because two teams had deployed over each other three times that week, and after that it was mine.

The spreadsheet had a row for every half day and a column for who had staging. By the end it also had a waitlist tab and a rule that you could not hold a slot for more than a day without writing down a reason. I am not proud of the rule about reasons.

What the queue actually cost

We had 41 engineers and one staging environment. A branch that needed a product manager to look at it held staging for as long as it took that product manager to find twenty minutes, which was usually a day and sometimes three.

I went back through the calendar before I left and counted. Over the last quarter, the median pull request that needed a non-engineer's eyes waited 1.4 days for a slot. That wait did not show up on any dashboard. The pull request was open, CI was green, and the author had moved on. From the outside it looked like slow review. It was a line.

The line had second-order costs that were worse than the wait:

• Batching. People saved up three or four branches and booked one slot, so every review was bigger and every bug was harder to place.
• Blame. When staging broke, the first forty minutes went to working out whose deploy did it. We had a thread for this. It was the most active thread in the company.
• Drift. The database on staging was whatever the last person left behind. Half the bugs found there were bugs in the data, not in the branch.

We tried to manage the queue first

For a while we did what most teams do. We added a second staging environment, which doubled the hosting bill and cut the wait by less than a third, because people booked both to be safe. We wrote a bot that nagged anyone who held a slot overnight.

Both were improvements to the queue. Neither asked why there was a queue. A queue forms when many things need one thing. We kept making the one thing easier to book, and it kept being one thing.

The pull request is the unit

The idea behind Canary is not complicated, and it was not new when Rowan and I started the company in 2023. If every pull request gets its own environment, nothing is shared, so nothing is booked. The author pushes, the environment builds, and a link shows up on the pull request. Whoever needs to look opens it when they have twenty minutes. Nobody else is waiting for them to finish.

What made it hard was everything the shared box had been quietly providing: data, secrets, and a bill somebody already paid. An environment per branch has to answer all three from nothing, on every push, in a few minutes:

1. The data has to look like production without being production. We seed each preview from last night's snapshot, with names, emails and card numbers masked before the copy leaves your cloud. Lena wrote up how the masking keeps foreign keys intact.
2. The secrets have to be scoped to the branch. Test keys by default, and production credentials never reach a preview.
3. The bill has to follow attention, not existence. A preview sleeps after 30 idle minutes and costs nothing while it sleeps. Theo explains the mechanics in a preview nobody opens should cost nothing.

And then it has to go away. Canary tears each preview down when the pull request merges or closes, in a median of 2m 12s.

What happens to the calendar

The old machinery disappears quickly once there is nothing to book. Halyard had one staging box and a Slack channel for reserving it. Six weeks after they turned Canary on, they archived the channel.

Their median review got a day shorter, from 2.6 days to 1.6. That is close to the wait I measured in my old spreadsheet.

If you keep the calendar today

If you are the person with the spreadsheet, here is the measurement I wish I had taken earlier. For two weeks, write down the time between "this branch is ready for someone to look at" and "someone could look at it." Not the review time. The wait before it.

Under an hour, your staging setup is fine. Over a day, you do not have a review problem. You have a line, and no amount of calendar discipline will shorten it for long.

The median time from a push to a live link on Canary is 3m 41s. The Hobby plan covers 500 preview-hours a month at no cost, which is enough to run the experiment on one real repository. The quickstart is a single file and one pull request.