Quick Answer: How Does Data Lineage Accelerate Data Management Capabilities?

More Detail

• Data moves along the data integration pipeline from operational/transactional systems toward the analytical ecosystem. Examples of this data include large payloads (the part of transmitted data that is the actual intended message) historically batched extraction, transformation and loading (ETL), change data capture (CDC) and message-based integration and virtualization.
• Data is transmitted among services in a distributed-service architecture. Examples of small data payloads include the data required to process one payment, onboard one new employee or acknowledge receipt of one shipment.

Figure 1: Sample Representation of Business and Data Lineage in a Marketing and Sales Analytics Scenario

Five Ways Data Lineage Impacts Data Management

• Data engineering and DataOps — Data lineage enables multiple data management personas (including data engineers, data architects and database administrators) to assess impact by tracing data dependencies across enterprise systems. It also improves the troubleshooting and monitoring capabilities of data flows.
• Cloud-based data migration strategies — Data lineage facilitates seamless cloud migrations by leveraging cloud-native tools, ensuring data integrity and continuity during transitions and reducing the risks associated with moving data to new storage locations.
• Enhanced decision making with AI Integration — Data lineage, integrated with AI-driven analytics, automates the extraction of insights from data, enabling faster and more accurate decision-making processes, which are crucial for optimizing business operations and enhancing products and services.
• Data and analytics governance — Data lineage provides granular visibility over the data life cycle, which helps businesses manage risks, comply with industry regulations and perform audits.
  Utilization of real-time data streams: Real-time data lineage tracking optimizes the use of both new and old datasets, ensuring businesses can promptly respond to market changes and evolving data collection techniques and technologies.

• Building trust in data used across self-service BI and analytics tools
• Identifying data that is covered by regulatory and legal obligations, including the European Union’s General Data Protection Regulation (GDPR), the Basel Committee on Banking Supervision (BCBS) 239 standard and the U.S. Health Insurance Portability and Accountability Act (HIPAA)
• Providing metadata that supports data and analytics governance effort processes with objectives like diagnosing incorrect data on an analytical dashboard
• Providing metadata that can be used during the design phase (or at the gathering requirements stage) for impact analysis

Common Data Lineage Best Practices

• Implement metadata management software. Data and its lineage are dynamic. You need to move beyond manually capturing data lineage in spreadsheets and automate the process. Most established metadata management solutions include data lineage as a key capability (see Tool: Vendor Identification for Data and Analytics Governance Platforms). Some vendors specifically focus on data lineage, such as Orion Governance and Solidatus.
• Include metadata sources. Database management systems (DBMs), big data tools, ETL software, Master Data Management software and other custom applications create their own data about the data they process. Include this metadata in your lineage, as it helps to understand data flow and modifications.
• Verify metadata sources. Encourage the owners of applications and tools to verify their respective metadata sources, as these data stewards are the people who clearly understand the accuracy and relevance of metadata.
• Extract metadata and lineage using forward and/or backward data flows through your system. This simplifies the process of mapping connections, relationships and dependencies among systems and within data. For example, you can start from the CRM system and work forwards, or start from the analytics dashboards and work backward.
• Validate end-to-end data lineage. Validate lineage progressively by starting from high-level connections between systems. Next, delve into connected datasets, followed by data elements before validating transformation documentation.

Recommendations When Deploying Data Lineage

• Invest in a standardized data integration platform streamlines pipeline data movement, offering a unified interface for tracking. Additionally, data virtualization and fabric naturally enhance data lineage.
• Ensure that your data lineage solution accounts for input parameters, runtime information and default values, and parses all of these elements to automate end-to-end data lineage delivery.
• Choose other technologies that may offer the right solution when understanding programming logic is impossible because of inaccessible or unavailable code (pattern-based lineage), or if you are considering using a transformation engine to control data movement (lineage by data tagging).
• Introduce active metadata management, which allows you to add automation capabilities that bring a proactive approach in data lineage delivery (see State of Metadata Management: Aggressively Pursue Metadata to Enable AI and Generative AI). For example, proactive notifications can inform you about overly complex parts of your data pipelines where refactoring or redesign would help to reduce the risk of failure.
• Use manual lineage when code is unavailable or inaccessible. Manual lineage requires interviewing people to document data flow and focusing on the data life cycle. Use spreadsheets for simple mapping techniques. Reviewing code involves manually checking tables and comparing columns, which can be time-consuming and tedious. Increasing code volume and complexity further complicate manual lineage.

How Data Lineage Supports Data Observability