Unity Catalog - The Forbidden Love

Scattered tables, metrics stray,
Data hiding every way.
Who will guard - yet unify?
Enter UC, the reason why.
‍
Lineage traced, permissions tight,
Access clear, no endless fight.
From lake to warehouse, all in sight,
Unity makes Data & AI/BI right.

I had Databricks Playground help perfect this poem, so I won’t take full credit 🙂. But fun aside: someone recently asked me a simple question - what’s your favorite Databricks product?

At first, I hesitated. Every product has its place, and it feels like picking a favorite nibling. But I couldn’t help myself - I mumbled that if it weren’t for Unity Catalog, I might never have become such a Databricks fan.

The person looked surprised and said, “I was expecting Lakeflow, MLflow, DBSQL, or Genie - but you say Unity Catalog?”

I paused and replied, “Everything you mentioned are vital organs… but Unity Catalog is the heart of Databricks.” (He had a healthcare background 🙂)

When Data Democracy Meant Data Chaos?

I’ve been doing “Big Data” for a while and have been a victim of Hive Metastore. Like many, I’ve been part of building more “civilized” versions of a catalog - something meant to be more than just a registry of objects, a system that could automagically keep things recorded, neat, and together while still letting diverse teams run their own experiments without duct tape. But before Unity Catalog, governance was mostly at the mercy of human behavior.

As the data grew and AI became more democratized, governance challenges became prominent. In the beginning it was an afterthought, then reactive, and later a full-blown emergency. Resources were pulled from projects, governance teams were formed, standards were written in text documents and slides, and registries were bought with the expectation that everyone would register everything they produced. Everyone worked hard, trying to follow those standards.

But the very nature of the sprawl created more things in more places, which ultimately led to losing track. Then came the cycle of building yet another set of standards, buying another catalog, or pulling people off projects to enforce governance - and still, chaos remained.

Governance Without Borders: Unity Catalog

Unity Catalog was never about creating just another catalog. It started with the realization that asset registration, organizing data and models, managing metrics, ensuring quality, tracking lineage, enforcing access control, enabling collaboration, and providing observability are not isolated problems but deeply interlinked - and the only way to solve them is under one roof, automated instead of bolted on later.

That’s why UC feels different: governance and freedom actually coexist, every asset is accounted for, lineage is built-in, permissions are unified, and workloads from ETL, SQL to AI and BI run on the same foundation. It’s fully baked and years ahead of anything else, and what I love most is that the hosted, decorated Unity Catalog with Databricks' Data Intelligence Platform is not only free of cost service but also supports non-Databricks engines. By abstracting storage and providing seamless UC & Iceberg REST APIs, it lets enterprises use platforms like EMR and Snowflake on the Lakehouse without breaking lineage, duplicating access control, or compromising governance.

With its built-in simplicity and intuitive design, Unity Catalog makes it easy for users across technical and business backgrounds to browse the catalog, discover assets, and request access effortlessly from one place. In fact, Unity Catalog delivers the best catalog experience for Snowflake¹ users, making systems that are often at odds work better together.

Most of our customers and enterprises around the world are not looking to build robots that beat every math Olympiad, but to automate thousands of micro-decisions that hundreds of people take in parallel. This allows them to focus on the next best thing for their customers and drive incremental growth without worrying about creating silos or data dumpsters. And while it is hard to create automatic, chaos-free unification, Unity Catalog does it so smoothly that we sometimes don’t even realize where the magic comes from.

If you’re already on Unity Catalog - congratulations, you’re governed and productive by default! If not, it’s free and hosted by Databricks for enterprises, and it works with any Iceberg-compatible platform through open APIs. Unity Catalog truly unifies governance across the entire Data & AI stack. That’s why I call Unity Catalog the "❤" of modern Data & AI.

‍

¹Snowflake has a Public Preview where they can write via Unity Catalog (solving the problem of not being able to update a table after using it within Databricks)