Having recently wrapped up a cloud data warehouse project for a Blue Cross Blue Shield organization, I thought it would be great to share some insight into tips to improve your chances for successfully moving to a modern cloud data warehouse. Whether the project is a new cloud data warehouse or a migration from an on-premise data warehouse to the cloud, you will find these tips helpful.
Here Are 3 Things To Consider Before You Get Started
1. Defining Success Criteria
No two ways about it, a cloud data warehouse project is an investment. It is important before you get started to both understand the size of the investment and how will you measure success. Every organization has different reasons for undertaking a cloud data warehouse project. What is yours? Are you migrating to decrease costs, increase throughput, scale faster, mitigate risk? The “why” behind the initiative might be a good place to start defining your success criteria.
Some success measures other companies have used include:
- Potential Cost Savings
- Increased Access To Data
- Increased Data Quality
- Ability To Cost Effectively Scale
- Increased Governance Of Your Data
The key is to define success criteria early on and measure to show results. Nobody wants to invest in a cloud data warehouse solution only to have the success of the project questioned after it is complete.
2. Potential Migration Paths
There are several paths an organization may take to build out a new cloud data warehouse, or to migrate from a legacy platform to the cloud.
The term “lift and shift” is often used when describing an approach to the project, but it is not quite as easy as it sounds. What I have found is that
- Most organizations cannot afford to completely replace their solution overnight, or
- They do not have all of the elements in place and may need to take a phased approach to their project, making them re-think the “lift and shift” approach.
There are other ways to migrate your data besides using “lift” and “shift” and can be done incrementally in an agile way. The key is to identify the data assets and the most efficient and impactful order of how to migrate the data, or the “lifting” part of the equation. In many cases there is a way to leverage your existing ETL/ELT data pipelines by refactoring them in the short-term to integrate with your new platform and build into the plan to eventually replace the ETL/ELT solutions over time, aka “the shift”.
Also, the use of the right technology can both automate the process and determine more efficient ways to migrate your data. Whether that is data loading tools, automated ETL/ELT tools or leveraging an automated data quality management tool to improve trust in your data with data consumers.
What is the best migration path for you?
Factors to keep in mind when deciding:
- When was my legacy system built?
- What shape is my data in today?
- Do I need to bring in new data sources?
- Do I have the budget to perform a true “lift” and “shift” or do I need to take a phased approach?
- Can I leverage technology to help?
- Do I have the expertise on my team to do it the right way?
3. How Do We Get Started?
Your cloud data warehouse project can be highly successful. The key is to “Start With Strategy” and have a well thought out plan to ensure success. Starting with strategy on this project was key to its success. Without it, the Blue Cross Blue Shield project would have had a very different outcome.
Areas to think about as you are developing your strategy:
People and Processes
Who are the people? What is their expertise? Are they in the right role? Do you have critical gaps in expertise or capability? How does the current ETL/ELT process perform today? Is there room for improvement?
Data Models, Structure & Catalog
What shape is my data in today? How is it structured? When I move to the new platform will the current structure hinder our ability to scale? Is there a more efficient way to extract, load and transform my data?
Data Architecture & Platforms
Will my data architecture allow me to effectively and efficiently scale in the future. Will my data be accessible to the people that will need it? Should I build a data lake? A data vault? What technology should I be leveraging to improve efficiencies and drive more automation?
Data Quality Management
How do I ensure I improve the quality of my data? Can I increase transparency for my data consumers? How do I improve trust in my data? What should I be testing? Do I need to automate my data testing processes.
Data Governance
Do I have a data governance program? Does it need to be updated? Who else from the organization do I need to involve in data governance design decisions? What regulatory risks do I need to consider? How do I design data governance elements into every aspect of my data warehouse migration solution?
There are many variables to consider when planning for and executing a data warehouse migration solution. You don’t have to do it alone. Blue Cross Blue Shield didn’t. They called on me and my company, Resultant, to help them with the planning and execution of migrating their legacy platform to the cloud.
Learn more about our data warehouse services
