Data platform

The data I work with lives in 30 workspaces and more than ten lakehouses, shared by the modelling and reporting projects of several companies. When a number looked wrong, there was no single place to see where a table came from, what wrote it, or whether anything still read it. Storage had grown to 40 TB over a two-year project, and the master nightly job took about 8 hours.

A lineage platform. A Python parser reads every ETL definition in the 30 workspaces, works out how data moves between them, and indexes each node: more than 1,000 tables and 2,000 ETL objects, with their notebooks, sources, models and reports, on one canvas. Open a table to see when it was last written, what triggers it and who looks after it. Tables nobody writes any more are flagged as stale. The lineage itself lives in lakehouse tables, so it can be queried like any other data. Around it sit nightly run monitoring, a catalog, a glossary and governance checks with named owners. It started as a tool for our own projects; my team is now turning it into a governed application for enterprise data projects in other companies and countries.

A weekly cleanup. A versioned VACUUM and OPTIMIZE job that runs every week across all the workspaces.

A new shape for the nightly job. I split one monolithic master pipeline into child pipelines by domain, so each part is easier to change and its errors are easier to trace. I re-planned the schedules so it no longer gets in the way of other jobs, which also lightened the nightly load on the source systems, and reworked the load patterns.

Storage went from 40 TB to about 4 TB. The master nightly job went from about 8 hours to 2.5 hours at the same scope. The largest tables in these pipelines have more than 250 million rows.