At a glance
- Data quality works best when it is designed into the platform from the start.
- Governance provides the structure needed to apply quality consistently across teams and datasets.
- Unity Catalog supports this through controlled access, metadata and lineage.
- Bad data can be quarantined and corrected without blocking the entire pipeline.
- Consistency is what turns data quality from a pipeline task into a scalable platform capability.
Dashboards are an important part of data analysis. A number looks off, a trend does not make sense, or two reports tell different stories. But that’s usually not where the problem started. They’re usually just showing what has already reached the serving layer. In most cases, the issues come from earlier in the data journey, from ingestion and transformation through to how the data is structured and governed across the platform.
Many organisations still treat data quality as a downstream activity, only validating, cleaning and fixing issues when they appear. This usually leads to problems like broken reports, manual reconciliation and growing distrust in data. Data quality is not just a technical concern. Poor data quality comes at a high cost, with Gartner estimating an average annual impact of $12.9 million per organisation.
A more effective approach is to design for quality from the beginning. This means building a platform where data is controlled, observable and consistent by default. In this context, Databricks Unity Catalog plays a key role. It is not a validation tool, but a governance layer that enables data quality to be applied consistently and at scale.
Why this matters (and who this is for)
As data platforms grow, maintaining data quality becomes more challenging. The data quality challenge is especially common and impactful in industries such as financial services, insurance, retail, healthcare and logistics, where business decisions depend on accurate, timely and governed data.
When multiple data sources, teams and pipelines are involved, inconsistencies can quickly appear. Different teams may ingest data in different ways, apply different rules or manage outputs in separate parts of the platform. Before these data quality issues show up in dashboards, technical leads and data engineers are often the first to see the warning signs through failed pipelines, duplicated rules, schema changes and reconciliation work.
If left unresolved, these issues eventually show up for reporting teams and business stakeholders as delayed reports or conflicting numbers. When governance is weak:
- Pipelines become inconsistent
- Validation logic is duplicated or missed
- Teams spend more time fixing data than using it
Left unchecked, weak governance can delay insights, reduce confidence in data and ultimately result in poor decision-making across the business.
Key ideas in this blog
This blog will explore the following themes:
- What “data quality by design” means
- Why governance matters more than isolated validation rules
- How Unity Catalog supports a consistent quality model
- A simple example of handling bad data without breaking pipelines
The objective of this blog is to help organisations that want to move data quality from reactive cleanup to a more governed platform practice. The blog will help you understand the importance of governance, how Unity Catalog supports data quality through consistent structure, access control, metadata and lineage, and how bad data can be handled in practice using a simple Lakeflow Spark Declarative Pipelines (SDP) example.
What does “data quality by design” mean?
A useful way to think about data quality is through six core dimensions: accuracy, completeness, consistency, validity, timeliness and uniqueness. According to Databricks, these dimensions provide a useful framework for assessing data quality across an organisation.
In practice, organisations often bring together data from many source systems, so most datasets will not meet all six quality dimensions when they first land in the Databricks lakehouse, usually in the raw or Bronze layer. Data quality usually improves as the data moves from raw or Bronze layers into Silver and Gold, where it is validated and enriched.
Data quality by design means that quality is not enforced by isolated rules, but by how the platform is structured.
Instead of asking:
“Did we validate this dataset?”
You start asking:
“Can we trust how this dataset was created, governed and tracked?”
At a minimum, this requires a consistent structure for data, controlled access to prevent unintended changes, visibility into how data moves and transforms, and the ability to trace issues back to their source.
This is where Unity Catalog becomes foundational. By organising data into a governed hierarchy from catalog to schema to table, it creates a single control plane across the lakehouse.
Why governance matters more than validation
Validation rules such as null checks or format checks are important, but they only work well if applied consistently.
In many environments, different teams ingest data differently, apply different rules and store outputs in inconsistent locations. Over time, this leads to fragmented quality standards.
Unity Catalog addresses this by standardising how data is organised, accessed and managed. It provides:
- A consistent structure across domains
- Fine-grained access control to prevent unintended changes
- Built-in lineage to trace how data evolves
This ensures that regardless of how data is ingested, it is governed within the same framework. As a result, data quality becomes a platform capability rather than a pipeline responsibility.
A simple example: turning bad data into a visible process
Consider a customer dataset arriving daily as a CSV file. Some records contain missing customer IDs, invalid email formats or unexpected schema changes.
The diagram below shows how this data flows through a governed lakehouse, and where quality is enforced.
Data quality flow across the lakehouse
Unity Catalog governance, metadata and lineage across every layer
Figure 1: Data Quality Flow Across Bronze, Silver, and Gold with Unity Catalog Governance
Without a strong design, pipelines may fail, silently drop records or apply inconsistent fixes. None of these outcomes build trust.
A better design separates concerns across layers like in the example above:
- Bronze stores raw data exactly as received
- Silver applies validation and splits outputs
- Gold serves trusted data for reporting
In the Silver layer:
- Valid records are written to a clean dataset
- Invalid records are written to a quarantine table with failure reasons
This layer becomes the trusted source of truth for downstream consumption, where most data quality enforcement takes place.
For example:
- Missing customer ID is flagged and quarantined
- Invalid email is flagged but retained
- Schema drift is captured using rescued data columns
With Unity Catalog in place, both clean and quarantined datasets remain governed and accessible. Teams can easily inspect failed records, and data is neither lost nor hidden in the process. This approach aligns with recommended patterns such as quarantining, where bad data is isolated without blocking the pipeline, allowing teams to review and correct issues later.
Once corrected, those records can be reprocessed through the same validation path. If they pass the quality checks, they move into the history Silver dataset. The Gold layer can then be refreshed from Silver, making the corrected data available for reporting and analytics.
This turns data quality into a visible and manageable process.
Why metadata and lineage are critical for trust
When data looks wrong, the real question is what changed?
The answer often depends on metadata such as ingestion timestamps, source identifiers, file details and change history. Without this context, debugging becomes guesswork.
Unity Catalog strengthens this by centralising metadata and providing built-in lineage. Teams can trace data from source to report, understand how it was transformed and identify where issues originated. This is what makes data quality explainable, not just enforceable.
Consistency is what makes quality scalable
One of the biggest risks in data platforms is inconsistency. When each dataset is handled differently, quality becomes difficult to enforce and maintain.
A more scalable approach is to standardise processing through configuration-driven rules, consistent layering and governed schemas. For example, teams can define rules for required fields, data types, deduplication keys and quarantine conditions in configuration rather than hardcoding them separately for every dataset. These configurations can then be passed into ingestion and transformation logic as data moves through the Bronze and Silver layers.
Unity Catalog provides the governed structure: catalogs, schemas, tables, permissions, metadata and lineage. Configuration defines how each dataset should be processed within that structure, including validation rules, deduplication logic and quarantine handling.
Together, they create a system where quality is repeatable, transparent and maintainable.
Key takeaways
- Data quality is a platform design outcome, not just validation logic
- Unity Catalog provides the governance foundation for consistent quality
- Flexible ingestion should still lead to a standardised processing model
- Separating valid and invalid data makes quality visible and manageable
- Metadata and lineage are essential for debugging and trust
Final thoughts
Improving data quality is not about adding more checks. It is about designing platforms where quality is built in from the start.
Unity Catalog does not enforce business rules on its own, but it enables something more important. It provides the structure, governance and visibility needed to make data quality consistent and sustainable.
If you are seeing recurring data quality issues, keep in mind that it is often not just a pipeline problem. It is a platform design problem. Addressing it requires re-evaluating how data is structured, governed and managed across the lakehouse.
Adaptiv works with organisations to design and implement governed lakehouse platforms using Databricks. This includes setting up Unity Catalog, defining consistent data models and embedding data quality practices into ingestion and transformation pipelines from day one. The goal is to move teams away from reactive data fixes and towards a more reliable and scalable data platform.
|
If you are facing similar challenges, our team can help you design a governed lakehouse approach that improves data quality and builds trust in your platform. |
Let’s chat |
