At a glance
- A foundation layer creates trust, not just data storage.
- Governance should be built into everyday delivery.
- Trust, speed and cost define platform success.
- Shared guardrails enable teams to scale safely.
- Databricks works best with strong operating discipline.
In the first article, I made the case that AI success is not just a model problem but a foundations problem. More specifically, it is a readiness and operating model problem.
This article is the practical follow-on.
If you accept that foundations matter, the next question is not which AI feature to buy. It should be around what a foundation layer actually needs to do, if you want analytics and AI to work in production, not just in demos.
That question matters because many organisations are still evaluating modern data platforms using assumptions formed in an earlier era. That is not a criticism. In fact, a lot of senior leaders asking the hardest questions have exactly the right instincts. They care about control, reliability, risk, modelling discipline, change management, and production stability.
Those instincts are still valuable.
What has changed is the platform model.
From data platform to foundation layer
In traditional on-prem environments, the “data platform” was often a database, an ETL tool, a reporting layer, and a lot of hand-built code held together by a capable team. Governance often sat outside the platform in process, approvals, and separate controls. Changes moved through slower release cycles. Scale usually meant more infrastructure, more tuning, and more specialised people.
In a modern cloud-native environment, the foundation layer has to do more than store and process data. It must create trust, support faster change, enforce policy consistently, and keep cost visible as adoption grows across analytics and AI. If you evaluate it as just another database or just another reporting stack, you will miss both where the value sits and where the risk shows up.
That is why I use the phrase foundation layer, not just platform.
The foundation layer is not storage. It is the system that creates trust at scale.
And this is exactly where Databricks can be misunderstood.
Databricks will not fix poor ownership, weak decision rights, or a culture where every team invents its own definitions. No platform will. The difference is whether your platform makes good discipline easier or leaves every team to solve governance and delivery in their own way.
The reason I focus on Databricks in this context is not because other platforms cannot govern data or support AI. They can. The issue is how this works in practice when lean teams need one foundation to support engineering, analytics, machine learning, and AI safely, with less stitching and less duplication.
Some platforms are a very strong fit when the primary need is SQL-centric warehousing and reporting. Some fit naturally in organisations that are deeply standardised on a single ecosystem and want a reporting-first operating model. Databricks tends to stand out when the same governed foundation needs to serve data engineering, analytics, ML, and GenAI workflows without rebuilding the same assets in parallel.
That distinction matters more than feature lists.
The three tests of a modern foundation layer
For senior leaders, there are three practical tests for whether a foundation layer is fit for purpose: trust, time-to-change, and cost-to-serve. These are the same three foundations from the previous article, but here they become platform requirements.
Trust
Trust comes first because trust is what turns outputs into decisions. When trust is missing, leaders slow down, teams add manual checks, and “just to be safe” workarounds appear. Adoption stalls, delivery costs rise, and risk increases because the organisation cannot confidently explain or defend what it is acting on.
A foundation layer has to create confidence in the data products and model outputs people are expected to act on. That means more than the pipeline running successfully. It means people can answer where a number came from, what data fed it, what rules were applied, who can access it, and whether the process is reproducible.
Trust is not a feeling. It is evidence, available on demand.
This is one of the reasons Unity Catalog is such a differentiator in practice. For me, it changes the conversation because governance, access control, lineage, and auditability are embedded in the platform experience rather than treated as a separate afterthought. Other platforms have governance capabilities too, but the day-to-day operating model often feels more fragmented or requires more stitching together. In small teams, that difference is not academic. It affects consistency, speed, and trust.
The strategic point is not just that governance features exist. It is that governance sits in the path of work, where teams actually build and ship. That is what makes it useful.
Time-to-change
Time-to-change is the second test. Most organisations can build a first version of something. The real test is how safely and quickly they can change it once the business learns something new, whether that is a new data source, policy, metric, control, or even an entirely new use case. When every change depends on multiple handoffs across a fragmented toolchain, delivery starts to slow down and confidence gradually drops.
This is where “best-of-breed” can become expensive. Each tool may be good on its own, but every seam between tools becomes a handoff, a dependency, a point of failure, and usually a separate approval path. That integration tax is real. It slows delivery, fragments accountability, and makes it harder to standardise how work is built and released.
A good foundation layer reduces that tax. It gives teams reusable patterns for ingestion, transformation, quality checks, orchestration, and release. It supports environment separation and disciplined promotion into production. It lets teams ship safe changes without rebuilding the delivery process for every use case.
This is also where I think some leaders still evaluate modern platforms through an older lens. The job is no longer to hand-code everything from scratch. The job is to know where code adds value and where the platform should carry the heavy lifting. Senior leaders do not get leverage from teams writing more code. They get leverage from teams shipping reliable outcomes faster through repeatable patterns.
Cost-to-serve
Cost-to-serve is the third test, and it is where a lot of AI and analytics programmes quietly come unstuck. Early pilots are often cheap enough to approve. Then adoption grows, workloads multiply, teams duplicate logic, and costs become unpredictable. No one can explain what is driving spend and no one wants to be the person slowing experimentation down.
A foundation layer must make cost visible and manageable. Not by shutting innovation down, but by creating the controls and operating habits that stop cost from drifting into chaos. Workload isolation, scheduling discipline, observability and standardisation all matter. If every team builds a bespoke path for every workload, you are increasing technical complexity and financial volatility at the same time.
Databricks can support this well when FinOps thinking is part of the platform design from the start, not something bolted on months later. If your platform team helps domains understand run cost, isolate workloads, and use safer defaults, cost-to-serve becomes a managed lever instead of a surprise.
In my 20 years working across data engineering and enterprise data architecture, the most consistent scaling failure I have seen is the “one central team owns all data” model. It turns the platform into a queue, and teams either wait or route around it.
Choosing the right operating model
This is not about one central team owning all data and becoming a bottleneck. That model does not scale, and it is especially painful in lean NZ and AU environments. The better pattern is shared platform capabilities, domain-owned data products, and central guardrails. The platform team makes safe delivery easier. The domains remain accountable for outputs and value in their part of the business.
That is a much more practical way to think about Databricks as a foundation layer. Not as a technical destination, but as a delivery system.
|
→ |
|
→ |
|
Common foundation pitfalls
And this is where good teams still get caught out. They migrate data into a modern platform but keep the same fragmented ownership, inconsistent definitions, and informal release practices they had before. Then they wonder why the platform did not magically fix the problem.
It was never going to.
There are a few anti-patterns worth calling out plainly because they are common and expensive.
- Lifting and shifting a messy warehouse into a lakehouse without fixing ownership, definitions, and delivery discipline does not create a foundation layer. It creates a modern-looking version of the same problem.
- Treating governance as optional until adoption grows is a mistake. By the time adoption grows, inconsistency has already spread and fixing it is harder.
- Using notebooks as production delivery without testing, deployment patterns, and change control can move quickly at first, but it does not scale well where reliability matters.
- Allowing every team to define customer, product, or revenue differently will destroy trust faster than any platform limitation.
- Running GenAI pilots without data classification, privacy review, and retention guardrails is not speed. It is unmanaged risk.
None of those are Databricks problems. They are foundation and operating model problems. Databricks is most valuable when it is used to reduce those risks, not when it is expected to compensate for them.
It is also worth being honest about trade-offs. Databricks is powerful, but it assumes a level of engineering discipline. If your immediate need is a simpler, reporting-first experience with minimal platform engineering, another path may fit better. The point is not to force-fit Databricks into every scenario. The point is to choose a foundation layer that matches the operating model you need to build.
A practical reality check
So what should a senior leader do next if you are evaluating Databricks or already using it and want it to function as a real foundation layer?
Start by defining the requirements of the foundation layer before you compare features.
- What must it do for trust?
- What must it do for time-to-change?
- What must it do for cost-to-serve?
- What standards must be enforced in-platform versus by manual process?
- What will be owned centrally and what will be owned by domains?
Then run a short reality check on your current state.
- How many critical data outputs can you trace end-to-end today?
- How long does it take to safely ship a change to a production data product?
- How many duplicate pipelines or duplicate definitions exist across teams?
- Can you enforce access, retention, and sensitive data policy consistently?
- Can you explain the operating cost of a production analytics or AI use case without guesswork?
Track a small set of operating metrics from day one. At minimum, I would track time to safely release a production data change, the percentage of critical data products with a named owner and clear definition, and the time it takes to trace a critical metric back to source. If you want a fourth, track cost variance of production workloads versus forecast. Those numbers will tell you whether the foundation is improving or just getting bigger.
Building your first 90 days
If you need a practical 90-day path, keep it simple.
|
|
|
|||
|
|
|
That is how Databricks starts paying off as a foundation layer rather than becoming just another platform story.
The bottom line
The decision is not just whether to adopt Databricks. The decision is whether you are willing to build a foundation layer that creates trust, speeds up safe change, and keeps AI cost and risk visible as adoption grows.
That is what senior leaders need from a modern data platform.
And that is where Databricks is strongest when it is implemented with discipline.
Get the foundation layer right, and analytics and AI stop being isolated projects.
They become a repeatable capability.
