A runbook someone can use at 3 am

Guide, October 2026. Applies to data pipelines, scheduled jobs, AI systems, anything with an alert. The example is a failed nightly pipeline.

The rule

One alert, one page. The page is linked from the alert itself, so the reader is one click from the instructions. It is written for someone who did not build the system, has not seen it in a month, and would like to go back to sleep. Every step is a command or a query they can paste, and every branch ends in either "fixed, here is how to confirm" or "escalate to this person, here is what to tell them".

The template

# Alert: nightly_taxi_pipeline failed

## What this means
The daily load from the raw files into silver and gold did not complete.
Reports refreshed after 06:00 will show yesterday's data until it is fixed.
Nothing is lost: raw files are kept, and the pipeline is safe to re-run.

## Confirm it (2 minutes)
1. Open the run: <link to the job runs page>. The failed task is highlighted.
2. Copy the last 20 lines of the task log. The cause is almost always in them.

## Likely causes, most common first
### A. A source file did not arrive
Check:  SELECT max(_source_file) FROM taxi.bronze.trips WHERE _ingested_at > current_date();
If empty: the upstream drop is late. Do nothing yet; re-run after 07:00 with step D.
If late past 09:00: escalate to the upstream owner (below).

### B. A source file has a new or changed column
Log contains "schema" or "_rescued_data".
Check:  SELECT count(*) FROM taxi.bronze.trips WHERE _rescued_data IS NOT NULL AND _ingested_at > current_date();
Fix:    the pipeline is built to add new columns; re-run (step D). If it fails again, escalate with the column name.

### C. Cluster or quota problem
Log contains "cluster", "quota" or "timed out before start".
Fix:    re-run once (step D). If it fails the same way, escalate to platform.

### D. Re-run
<link to "Repair run"> and choose "repair failed tasks". Takes about 20 minutes.
Confirm: SELECT max(pickup_ts) FROM taxi.gold.trips_by_zone_hour;  -- should be yesterday

## Escalate
Upstream owner: <name, channel>   Platform: <name, channel>   System owner: <name, channel>
Tell them: the alert name, the run link, which cause above you checked, what you saw.

## After
Post one line in #data-ops: what it was, what you did. If the cause is new, add it to this page.

Why each part is there

• What this means tells the reader the blast radius and whether anything is lost, which decides whether they fix it now or at 8 am. "Safe to re-run" is the most valuable sentence on the page.
• Confirm it stops the reader acting on a false alarm and puts the log in front of them before they guess.
• Likely causes are ordered by frequency, each with the exact query that distinguishes it. Three to five causes cover most alerts; a page with twelve is a design document, not a runbook.
• Escalate names people and says what to tell them, so the escalation message is useful on the first send.
• After is how the page improves. The person who hit a new cause is the right person to add it, while it is fresh.

Testing it by breaking things

A runbook that has never been used is a guess. Before launch, and after any big change, run a drill: delete a source file and let the alert fire; have someone who did not write the page follow it with a timer; note every place they hesitated or had to ask. Fix the page, not the person. Repeat for each cause. An afternoon of drills finds the missing link, the query that needs a permission the on-call person does not have, and the step that assumes a tool nobody installed.

Where runbooks live

Next to the code, in the same repository, in plain text, with the alert's link pointing at the rendered page. A wiki that nobody updates is where runbooks go to rot. When the alert is defined in code (a scheduled query, a job notification), the runbook link is part of that definition, so a new alert without a runbook is visible in review.

When to ask for help

If the system has alerts but no runbooks, writing them is a few days of interviewing the people who have fixed things, and the drills are the part most teams skip. We include a runbook per alert and a drill in every handover, which is why the handover takes the time it does.

Related: what is a runbook in the FAQ, the new column guide for cause B in detail.