Home

Are increasing data integrations costs becoming an issue for your business? Any work around you would recommend?

I like Mike Kail's and Bill Philbin's comments. Work-arounds always create a bigger problem. They are obviously NOT part of a strategy - hence work-around. As you change and evolve your platforms over time - the work-arounds always have to be dealt with separately - added complexity kills agility and scale.  I would first focus on the end result. Determine your best possible outcome separately from considering solutions.  In our case (Hammerspace), we have fundamentally changed Data Management to focus on THE DATA, not the storage or repository it is siloed in. We talk to the data natively, so that we can manage the DATA across all storage and cloud solutions.  So, in my mind, it is completely worth the work to conceive of the RIGHT answer, not just the closest answer. Go for Innovation! 

27915 views
18 comments
6 upvotes
Related Tags
Anonymous Author
I like Mike Kail's and Bill Philbin's comments. Work-arounds always create a bigger problem. They are obviously NOT part of a strategy - hence work-around. As you change and evolve your platforms over time - the work-arounds always have to be dealt with separately - added complexity kills agility and scale.  I would first focus on the end result. Determine your best possible outcome separately from considering solutions.  In our case (Hammerspace), we have fundamentally changed Data Management to focus on THE DATA, not the storage or repository it is siloed in. We talk to the data natively, so that we can manage the DATA across all storage and cloud solutions.  So, in my mind, it is completely worth the work to conceive of the RIGHT answer, not just the closest answer. Go for Innovation! 
4 upvotes
Anonymous Author
If you are increasing costs as a result of acquiring new data sources that are valuable - providing raw material into data science/etc that drive business value - then I'm not sure the increasing costs alone are reason for concern.  ROI on those costs is a better way to look at it. But tools like Denodo can eliminate movement of data and provide analytics in-place, building upon the "virtual" data warehouse concept.  Performance trade-offs need to be evaluated when these architectures are implemented in self-service analytics.
2 upvotes
Anonymous Author
For sure data integration is a constant challenge from a cost and a risk perspective.  There's no one good answer I'm afraid. I think it's all about leveraging automated tools and standardization where you can, ensuring the appropriate skill level of your data integration staff, and educating IT Governance about the criticality of data integrity as the constant requests for new integrations come in!
2 upvotes
Anonymous Author
Anonymous User: Could you elaborate on the original question? What aspects of data integrations are you most concerned about? In my mind, this issue can be broken down into integration costs (maintenance/development), resource costs, security/privacy costs, classification costs, storage costs, etc. Which fall into areas of concern for you? (one, some, all above)? Asking because I think it will lead to more focused discussion on this thread.
2 upvotes
Anonymous Author
It is not a new problem, but the increased need to tie disparate business silos and processes and the increased use of SAAS and cloud platforms has raised the bar on complexity. Two critical components to making progress... Understanding the data sources, especially system of record for data, and establishing a capable integration platform can help once you overcome the initial learning curve.  There is an important longer term benefit to avoid the expediency allure of point to point connections between major systems... Spend the effort to think through an Enterprise integration architecture.
2 upvotes
Anonymous Author
There are many integration as a service platforms for both on-premise and cloud based data sets. It really depends on level of integration and abstraction required for a specific business outcome. It is also important to understand the data lineage and future purpose before deciding on the integration strategy. 
1 upvotes
Anonymous Author
Why is this question asked by an Anonymous  User ? Sorry... Don't talk to anonymous people....
1 upvotes
Anonymous Author
I think it mostly depends on business type. If the business is primary type and directly communicates with the clients, then the cost is not high enough compared to the profit return from the utilisation of that data...
1 upvotes
Anonymous Author
I would recommend a longer-term architectural approach like leveraging an integration framework. I have always put in place some flavor of ESB (enterprise service bus) like IBM Boomi, Mulesoft ESB, Apache Camel, etc. whenever I go into any company because it strategically provides several things: 1) architecture for integrations that will scale (these frameworks treat integrations like applications and you can cluster/scale them like you would any other application type) 2) any-to-any endpoint support, 3) decreases cost of integrations (because over time, you end of re-developing when needed typically the outbound part of the integration, for example when replacing out one system with another, 4) self-documenting (IBM Boomi for example will document the integrations and data pipelines within the tool for no extra effort). There are more advantages but that's a good start. For decreasing cost of data processing, storage, I would recommend looking at what data types you use for storage (most of the cloud vendors have options for longer term storage types that are cheaper than real-time application disk/memory usage). Cursory, but the above will give hopefully provide some other areas to consider. 
1 upvotes
Anonymous Author
In general, these can be overcome by centralizing management and job scheduling of IT background processes so that administrators can easily automate and control their systems. However, it is important to understand the challenges that many companies face with Data Integration (DI), so that the appropriate tool can be used to address them. The Big problems can be due to different types of data (structured, unstructured), Accuracy of data, timeliness of data, completeness of data etc. Poor DI practices can have an effect on all of these characteristics. Also, the cost in software licensing alone can make DI significantly  more expensive than it needs to be. If the company is facing the majority of these problems, then it may be worth changing the data integration paradigm. An operational shift is also likely to necessitate new technology investment to deliver the functionality needed to improve visibility and increase efficiency. DI tools come in all shapes and sizes, so it is helpful to go into the planning phase with a baseline set of features to look for.
1 upvotes
Anonymous Author
It all depends on the maturity of your current systems and quantity and size of your current databases. And of course the vision of your CIO, The best work around is to foresee as far as possible the needs and then design the IT services accordingly. Small systems with powerfull DNA. If your company is old, and you have a really big messy combination of databases, systems, platforms, etc. The work around is the same. Start small, with one platform, preferrably don't buy services, and hire people that knows, so you don't depend on outsiders with untamed bills. If you have the right people, you'll spend more on them and less in services. IMHO.
1 upvotes
Anonymous Author
Data Integration is a reality of todays digital business. Anything else will be more expensive and is unlikely to bring in real transformation. Further in a true digital collaborative world where innovation counts, time to market is the key to win newer markets and customers what else will matter more. Hence the integration is a desirable pain in the interest of business growth and staying relevant to customers.
1 upvotes
Anonymous Author
Our challenge is more around the value accrued and integrity and velocity of the data with constantly evolving business model. From our perspective, approaching it with re-useable and standardizing integrations, focussed on the key areas of sales & finance or whatever is important to run the company (customer, prices, discount, commissions, revenue, margin etc.) is the path forward. If we simplify and make it low friction, then costs can be a less of a concern relatively speaking
0 upvotes
Anonymous Author
I'm not a fan of "work arounds", so I'll go with a solution that I've used with great success in the past, that being SnapLogic's iPaaS solution. (https://www.snaplogic.com/glossary/data-integration-platform) This allows one to continually evolve as data sources ebb and flow.
0 upvotes
Anonymous Author
2 years ago I came into an area where almost all integrations were made with Database links. There were 100 applications each with their own DB instance and all bound by a spiderweb on interdependency. To get through I started looking at various data virtualization technologies. This would allow our systems to share data without the strong coupling of the links. Two standout technologies I found were Snaplogic and Denodo. Both worked as if they were instant api creators without changing the underlying tech of the source and destination systems. Both have their merits. Feel free to contact me if you want some details.
0 upvotes
Anonymous Author
There are a number of data integration platforms out there depending on your data sources including platforms like Amazon Redshift. I would start with the end state of what you're looking to do with the data and work backwards. Several of the ISV's in the virtualization space are building data integration including governance into their toolset. I wouldn't be afraid to also build some tools internally  - if you have the capability - as to understand the value of the end state data and the often changing business requirements mean you may want to have a look around the corner a bit before a commitment.
0 upvotes
Anonymous Author
I never use work around, because in the end they will fail. Do it right the first time. I am also concerned about data integrity if using cloud service providers as opposed to on-prem platforms. If you decide to use a third party for data integration be sure your contract is written to allow you to recover that data if your relationship with them is severed. The cost can be an issue but the better question is what is the cost to the business if it is not done correctly?
0 upvotes
Anonymous Author
When you consider where, what to leverage automated tools and standardize, consider RPA as a potential how.
0 upvotes