The pipeline failed because a source added a column

Guide, October 2026. Applies to Databricks with Auto Loader and Lakeflow or Delta Live Tables pipelines. Examples use the public NYC taxi files, which gained an airport fee column in 2022 and have switched the casing of column names between months.

What actually happens

Three different events look the same from the outside. A source adds a column: a new fee, a new status, a new field the upstream team shipped without telling anyone. A source renames a column, often only in casing, so Airport_fee becomes airport_fee and a case-sensitive load treats it as a second column. Or a source changes a type: an integer id becomes a string, a date becomes a timestamp.

A pipeline with a fixed schema reacts in one of two bad ways. If it validates, it fails at the first file and nothing lands until someone wakes up. If it does not validate, the new column is dropped or the renamed one is read as empty, and the tables stay green while the numbers quietly go wrong. The second is worse, because nobody is paged.

The fix, in three parts

1. Let bronze absorb the change

Bronze is the landing layer, and its one job is to keep everything that arrived. Auto Loader does this with schema inference and schema evolution turned on. Point it at a schema location so it can remember what it has seen, and tell it what to do when something new appears.

df = (spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "parquet")
      .option("cloudFiles.schemaLocation", f"{checkpoint}/schema")
      .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
      .option("cloudFiles.inferColumnTypes", "true")
      .load(raw_path)
      .withColumn("_ingested_at", F.current_timestamp())
      .withColumn("_source_file", F.col("_metadata.file_path")))

With addNewColumns, a new column is added to the target table and the stream carries on. On the very first appearance the stream stops once so the schema change is recorded; a pipeline with retries picks it straight back up, which is why the job should have at least one retry configured. Anything that does not fit the known schema, such as a value whose type changed, lands in the _rescued_data column as JSON instead of being thrown away.

2. Make silver notice, loudly

Silver is where rules live, and a schema change should show up as a failed rule, not as a surprise. Two expectations cover most cases. One watches the rescued data column: if it is ever non-empty, something arrived that the schema did not understand. The other normalises names, so a casing change becomes a mapping rather than a new column.

import dlt
from pyspark.sql import functions as F

@dlt.table(name="trips_silver")
@dlt.expect("nothing_rescued", "_rescued_data IS NULL")
@dlt.expect_or_drop("in_month", "pickup_ts >= '2025-06-01' AND pickup_ts < '2025-07-01'")
def trips_silver():
    df = dlt.read_stream("trips_bronze")
    # lower-case every column so Airport_fee and airport_fee are the same thing
    df = df.toDF(*[c.lower() for c in df.columns])
    return (df.withColumnRenamed("tpep_pickup_datetime", "pickup_ts")
              .withColumn("fare_amount", F.col("fare_amount").cast("decimal(10,2)")))

expect without _or_drop keeps the rows and records the failure count in the pipeline's event log, which is exactly what you want for a schema surprise: nothing is lost, and the count is non-zero. Pair it with an alert on that metric and the 3 am failure becomes a 9 am message that says which file and which column.

3. Decide the policy for the new column

Once the column is in bronze and silver has flagged it, a person decides what it means: carry it to gold, map it onto an existing field, or ignore it. That decision is a code change with a review, not an emergency. The point of the first two parts is to buy that time.

How to test it before it happens to you

Break it on purpose. Drop a file into the landing path with one extra column and run the pipeline: bronze should gain the column, silver's nothing_rescued count should stay at zero (because the column was added, not rescued), and the alert should not fire. Then drop a file where an integer column holds a string: the rescued count should go up and the alert should fire with the file name. If either test does not behave, the setup is incomplete, and it is better to learn that on a Tuesday afternoon.

Two mistakes to avoid

• Setting the mode to rescue for bronze. It keeps the schema frozen and shoves every new column into the rescued JSON, which is safe but makes the new field unusable until someone parses it out. addNewColumns is the right default for a landing layer; rescue suits a silver table whose shape must not move.
• Deleting the schema location to "reset" things. Auto Loader then re-infers from scratch and may pick different types. If you must reset, do it with a full reprocess and a clean checkpoint, as a planned job.

When to ask for help

If the pipeline has no bronze layer at all and sources load straight into reporting tables, adding these pieces is a few days of focused work and the lakehouse walkthrough shows the shape. If sources change weekly and nobody owns them, the fix is as much a conversation with the upstream team as it is code, and we are happy to be in that conversation.

Related: Databricks lakehouse FAQ, Auto Loader and expectations in the glossary.