Most data warehouse projects fail. Here’s how not to.
By Roeland Krom & Daan Boersma • published December 13, 2021 •
In today’s data-driven reality, data warehouses are becoming increasingly crucial for companies in all sectors. Yet despite the hours and cash poured into them, some 80% of data warehouse projects ultimately fail to achieve their aims. There’s no shame in this at all; in fact, it happens far more often than you think. When the giants of the business world get it wrong, it might hit the headlines. But for most enterprises, the high project failure rate is simply kept quiet.
Realizing you need a solid and reliable data warehouse is just the tip of the iceberg. Investing serious time and funds into creating, organizing, and securing your data management comes next — and you want your project to be one of the well-planned few that make it. Below, we’ll share our insights on why so many data warehousing projects do fail, and disaster-proof strategies to help yours succeed.
Set realistic project and ROI expectations — don’t aim too high, too soon
The benefits of a data warehouse are clear: Greater efficiency and accountability, quicker deliveries, enhanced forecasting, and the ability to better adapt to changing markets and act on the opportunities they present.
Yet it’s notoriously tricky to demonstrate ROI on a data warehouse. Big promises, big goals, and big investments won’t necessarily lead to obvious short-terms wins. 6 months to a year down the line, the project may still feel like a leap of faith that’s not delivering tangible results.
Due to this fear factor, project requirements for data warehouses are often designed to meticulously track the complexity and progress of the project itself. “As long as we can prove it’s moving forward, we’ll be able to keep the funding coming in and eventually make it work,” some might say. However, data warehouse projects that only look inward, and that fail to actually address the organization’s business objectives, are doomed to fail.
The fix: Involve business users from the word go, and start small
Scoping out your data warehouse project requirements should start with the people who know your business objectives best — not the people who know your databases inside out.
By starting with a clear business goal in mind, you can reverse engineer to pinpoint the specific reporting requirements you’ll need to achieve it. Then you can design your data warehouse to facilitate these. Next, build it up bit by bit with a clearly structured approach.
For example, start by connecting up key data sources for your finance stream, so you can report on core financial KPIs. Over time, you can iterate and scale so your data warehouse includes and interlinks HR, Operations, Sales, Quality, and so on, until you’ve added all the data sources you need.
At Cohelion, we like to start by creating a data blueprint for each client we work with. What data sources do they have, what’s the granularity of the data in them, and what are their main KPIs — both now and for the future? This sets us up to construct a data warehouse that rapidly addresses each client’s core reporting needs, but that’s also designed to evolve as their business does. Next, we create a skeleton data warehouse for the client in just a few weeks. We then connect new data sources, new offices, new regions, and so on, rolling out added complexity at a pace that suits the client.
There’s another key element to bringing in business users to your data warehouse project: Cultivating true belief in the power of a data-driven company culture — and the insights that come with it. The key figures in your company need to live, breathe, and share the idea that data is what will power your organization ahead. This is instrumental in convincing the whole company to contribute however they can as you build your data warehouse, while also becoming inspired and enthusiastic about your data-driven future.
Find a project partner that’s truly invested in your business objectives
Often, organizations that aren’t tech-savvy themselves hit a wall. They know they need a data warehouse, but haven’t the faintest idea how to start — and the sheer number of off-the-shelf and customized data solutions on the market is overwhelming, rather than helpful.
This leads many companies to choose the specific expert route. You search and search to find an expert to get your data warehouse up and running (who’ll likely be on the expensive side, given they’re so in demand). Finding yours might be a relief — but be ready to be tied to their expertise. Naturally, every specialist has a personal preference, and they’ll build your warehouse based on theirs: Open-source, Microsoft, Oracle, or a best of breed solution, you name it.
There’s another red flag here. An individual data expert is also unlikely to have expertise that aligns with your sector-specific business objectives. Suppose you’re managing a trucking company, and know you need to get your data warehouse going. You might decide to recruit a data architect to initiate the project, but what are your selection criteria for picking one? And when you’ve found one, that data architect will be the best at what they do, yes. But they probably won’t have a clue about your industry or the KPIs your data warehouse needs to serve.
The fix: Join forces with a data partner that knows your sector
It’s wise to work with a data partner who understands your line of business. This insider knowledge will help them create a data warehouse around your key KPIs, ensuring you can report on them rapidly and reliably.
With a data partner who knows your business area, you’ll move forward faster. They’ll guide you on how you can iterate, scale, and enhance your data warehouse in a way that tailors it to your industry’s evolutions, opportunities, and compliance requirements.
Make sure you feed your warehouse quality data — garbage in means garbage out
Picture this: You’ve spent a huge amount of hours and funds on a data warehouse project that, if you’re honest, no one was convinced was going to succeed in the first place. Now, it’s time for it to produce its first reports. When reviewed, there are serious concerns raised about the accuracy of the figures. And that’s the nail in the project’s coffin.
Yet this happens all the time. It’s no surprise, given there are usually differences of opinion on a company’s data even before a data warehouse project is mentioned. For example, HQ might produce figures that staff in regional offices feel are way off the mark. Internal politics, lack of attention to detail, or even a desire to stay strictly compliant can also create data silos that prevent companies from communicating openly and transparently about their data. This is all fairly normal in the business world, but you’re setting yourself up to fail if these discrepancies enter your data warehouse.
The fix: Optimize accuracy within your data landscape
To succeed, your data warehouse project needs quality primary data to work with. We support our clients to adopt a structured data workflow, creating a system of checks, balances, and approvals to promote data accuracy.
Building in these sign-offs allows those in the know to approve relevant data sets, or not. It also facilitates tracing inaccurate or modified data as needed, helping to create a culture of responsibility and accountability when it comes to ensuring your warehoused data is reliable.
Building in these sign-offs allows those in the know to approve relevant data sets, or not. It also facilitates tracing inaccurate or modified data as needed, helping to create a culture of responsibility and accountability when it comes to ensuring your warehoused data is reliable.
Start your data warehouse project with confidence
Data-driven is the buzzword these days, and we see time and again that our clients need a solid, reliable, and scalable data warehouse to get them there.
From accurate and rapid KPI reporting, to being audit-ready, to meeting data governance, data protection, and sustainability reporting requirements, a data warehouse is an asset that organizations simply can’t do without.