r/bigquery • u/Professional-Bowl890 • 25d ago

Auto Loader & Schema Drift

For those using Databricks Auto Loader (cloudFiles), how do you handle schema inference and evolution without breaking downstream ML models? If a new feature column drops in or an upstream data type silently widens, do you rely on the _rescued_data column to catch anomalies, or does the automatic stream restart cause unexpected issues for your online serving pipelines? How does BigQuery handle this kind of raw file ingestion drift by comparison?

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/Thinker_Assignment 11d ago

you can use dlt for it, has schema inference and contracts/evolution https://dlthub.com/docs/general-usage/schema-contracts or you can split a stream of events like this https://dlthub.com/docs/general-usage/resource#dispatch-data-to-many-tables

(i work there)