• which commit wrote it

• which wrote this record (Job id, Job Run Id)

• which operation triggered that write

DESCRIBE HISTORY gives you this metadata, but on large tables it can be slow, and running it repeatedly while investigating a bug quickly becomes painful.

The practical workaround is to dump the entire history once into a physical table.
From there, you can filter, join, and slice it instantly — without re-scanning the entire Delta log on every query.

One-Time Dump of Delta Table History

CREATE TABLE IF NOT EXISTS databricks_support.default.describe_history__your_table_name AS
SELECT *
FROM (
    DESCRIBE HISTORY your_catalog_name.your_database_name._your_table_name
);

For deep debugging (record → parquet file → commit lineage), this table becomes a fast, queryable audit log.
In practice, this works best when run from a notebook, where long-running metadata operations are less fragile.

I also have a script that can identify which row is written in which Parquet file by which commit; drop me a comment if you need it.

How to calculate in-memory data size

When assessing data size for partitioning in Spark, it's important to note that the on-disk size—such as data stored in S3—does not always reflect the in-memory size. This is because data formats like Parquet or Avro are highly compressed, and the actual memory footprint can be 2 to 8 times larger than the file size on disk. Understanding the in-memory size is essential for properly tuning your shuffle partition settings.

To accurately gauge this in-memory size, you can run the following Spark commands to trigger a computation and then inspect the Spark UI (specifically under the SQL/Dataframe tab) for the 'Shuffle read size':

# Read data (example: Parquet file) df = spark.read.load("examples/src/main/resources/users.parquet") # Save as no-op (does not write data, but triggers computation) df.write.format("noop").mode("overwrite").save()

This approach helps ensure that you're basing your partitioning decisions on the actual memory requirements rather than the compressed on-disk sizes.

References

https://www.databricks.com/notebooks/gallery/SparkAdaptiveQueryExecution.html

https://www.databricks.com/discover/pages/optimize-data-workloads-guide

Keep This Post Discoverable: Your Engagement Counts!

Your engagement with this blog post is crucial! Without claps, comments, or shares, this valuable content might become lost in the vast sea of online information. Search engines like Google rely on user engagement to determine the relevance and importance of web pages. If you found this information helpful, please take a moment to clap, comment, or share. Your action not only helps others discover this content but also ensures that you’ll be able to find it again in the future when you need it. Don’t let this resource disappear from search results — show your support and help keep quality content accessible!