Home

What’s the most challenging part of business intelligence (BI) or data warehouse implementation?

The most challenging part is that every user wants everything immediately. Every technology in today's world has to be data-driven, so the biggest challenge we’ve had to solve is real-time data availability. When we were thinking about how to make this possible, we did not want to invest because real-time data involves a lot of information exchange between the source and the target system. If something fails, then you have to put in a lot of manual effort to correct that information. Then whatever productive work you had planned for that day goes out the window, because business as usual (BAU) becomes more important. So defining the architecture and the kind of KPIs that you’ll need at the starting point is essential. If those aspects are not taken care of well, we’ll have to do a lot of reworking. We cannot go to the source over and over again, because our production systems are built for transactions, not for analysis and reporting; the data warehousing systems are built for that. If we have not done the base architecturing work correctly, it will require a lot of maintenance. To solve this issue, we have invested in change data capture (CDC) tools that help us move the data on a real-time basis. We have also automated many things, which helps reduce the manual effort needed.

69 views
3 comments
1 upvotes
Related Tags
Anonymous Author
The most challenging part is that every user wants everything immediately. Every technology in today's world has to be data-driven, so the biggest challenge we’ve had to solve is real-time data availability. When we were thinking about how to make this possible, we did not want to invest because real-time data involves a lot of information exchange between the source and the target system. If something fails, then you have to put in a lot of manual effort to correct that information. Then whatever productive work you had planned for that day goes out the window, because business as usual (BAU) becomes more important. So defining the architecture and the kind of KPIs that you’ll need at the starting point is essential. If those aspects are not taken care of well, we’ll have to do a lot of reworking. We cannot go to the source over and over again, because our production systems are built for transactions, not for analysis and reporting; the data warehousing systems are built for that. If we have not done the base architecturing work correctly, it will require a lot of maintenance. To solve this issue, we have invested in change data capture (CDC) tools that help us move the data on a real-time basis. We have also automated many things, which helps reduce the manual effort needed.
1 upvotes
Anonymous Author
This is an AND challenge, not an OR challenge, provided I parsed your question correctly as you didn't actually list a challenging part In order to provide "best of breed" BI solutions, one needs to first have a comprehensive Dake Lake/Swamp/LakeHouse/TermDuJour platform in place.
1 upvotes
Anonymous Author
BI and data warehouse are different items. You can do BI on an excel spreadsheet or relational database, so it does not have to be a data warehouse. For BI, in my experience, the most crucial part is understanding the data you have, i.e. having explicit knowledge of the database tables, how they are connected and what information they store, i.e. the meaning of the data in the practical world. The BI tool will visualize the data, but you need to know what that data means to visualize it in a meaningful way that makes sense to the customer. Then the challenge is understanding what each customer wants. You need to anticipate some needs, so you build some ready-made visualization apps so when you do a demo to a customer, it makes sense what you are showing. Also, understanding their current challenge that BI can solve for them and creating the virtualization for them is. The Bussines analyst work here is more challenging/important than the technical part of coding/visually connecting the data. Data warehouse implementation varies from situation to situation. In my case, the main challenge was data growth and scalling.
0 upvotes