When I’m debugging a Delta table with millions of commits — especially tables with heavy ingestion, lots of parquet files — I often need to trace a specific record back to:
which commit wrote it
which wrote this record (Job id, Job Run Id)
which operation triggered that write
DESCRIBE HISTORY gives you this metadata, but on large tables it can be slow, and running it repeatedly while investigating a bug quickly becomes painful.
The practical workaround is to dump the entire history once into a physical table.
From there, you can filter, join, and slice it instantly — without re-scanning the entire Delta log on every query.
One-Time Dump of Delta Table History
CREATE TABLE IF NOT EXISTS databricks_support.default.describe_history__your_table_name AS
SELECT *
FROM (
DESCRIBE HISTORY your_catalog_name.your_database_name._your_table_name
);
For deep debugging (record → parquet file → commit lineage), this table becomes a fast, queryable audit log.
In practice, this works best when run from a notebook, where long-running metadata operations are less fragile.
I also have a script that can identify which row is written in which Parquet file by which commit; drop me a comment if you need it.



I would be interested in the script for row level auditing that you wrote about at the end of the post!