ASG Perspectives

Blog > March 2021 > Seeing is Believing: How Data Lineage Improves Data Quality, Trust & Compliance

Seeing is Believing: How Data Lineage Improves Data Quality, Trust & Compliance

Organizations have long struggled to improve data quality and trust. With the explosion of data sources and escalating needs to use it for customer insights, this challenge has continued to intensify.

To meet the complex challenges of managing their sprawling data landscapes, data leaders are rediscovering data lineage. Originally, a tool used by IT professionals for data integration activities and root cause analysis, data lineage has been extended to support a wide array of business use cases that need quality, well-understood and protected data.

Data Lineage Defined

Before you can confidently use data for any purpose, you’ll likely need to know the following:

  • What data is available?
  • Where did it come from?
  • How is it being used and for what purpose?
  • Can I trust it?
  • Is it ok if I use it?

Data lineage can answer these questions.  Simply defined, data lineage is a map of your data. It provides an end-to-end view of your data’s journey from the data source (where it was created) to where it is being consumed in reports, analytics, as trigger events or other uses. During its journey, data it is transformed as it moves inside and between diverse processes and systems.  Data lineage records these transformations, providing understanding and confidence in how the data was derived.

This technical data lineage also helps to map the relationships between the data and the applications, providing deeper context into the data and how it is being used.

16blog.png

Data lineage example: Personal Identifying (PI) data can be easily traced to all the data assets consuming it.

Automating the Data Map

Traditionally, data lineage documentation was a long and arduous process that could take weeks or even months, depending upon the data sources being traced.  The complexity of how data flows from the various sources to the business intelligence environments demands manual review, inspection, and documentation. This task is often conducted with spreadsheets and interviews with Subject Matter Experts (SMEs).  With employee attrition, this tribal knowledge is often lost. Also, since data environments are constantly changing, by the time the manual process is completed, the report is often obsolete.

As a result of these challenges, manual approaches can’t scale across the enterprise. Organizations are conducting these inventories on an ad hoc basis as projects or new mandates require.

There’s a better way!  ASG Data Intelligence (ASG DI) automates end-to-end lineage capture, tracking, and modeling.  It does this by capturing metadata from many data sources. Technology-specific models are used to store captured metadata, which is then used as the source for a technology-agnostic consolidated lineage model. What’s more, it’s automatically updated, so your data inventory never goes stale.

The captured metadata traces the movement of metadata (“where lineage”). Information from analyzed code supplements the “where” lineage to expose how data changes as it moves (“how lineage”).  The combination of “where” and “how” lineage is unique to ASG DI.

Artificial intelligence and machine learning can also be used to supplement any gaps that may occur due to custom code or application of manual processes. ASG DI also offers a governed “stitching” process for authorized users to “train” the lineage and fill in any gaps.

Enriching the Lineage with Business Meaning

ASG DI’s robust metadata management platform includes a Business Glossary with RACI-based data governance workflows enabling data stewards and owners to collaborate to create, maintain and share a common language of business descriptions, data privacy classifications, business rules, policies, processes, data ownership and other context.  This “business metadata” augments the technical data lineage for a 360° view of your Critical Data Elements (CDEs).

ASG DI’s “single pane of glass” view of your data is why we call it “data intelligence.”  It’s unique to ASG DI and the industry has taken notice, including the A-Team Group’s Reg Tech Awards celebrating leading technologies. Late last year, they awarded ASG DI as the “Best Data Lineage Solution.”  In addition, Gartner positioned ASG Data Intelligence (ASG DI) as a Leader in the 2020 Magic Quadrant for Metadata Management Solutions for the third year in a row. Other recognitions, including recent awards from KMWorld and Enterprise Management 360 can be found here.

Once you’ve created this foundation of data intelligence, you will quickly become the “data hero” of your organization. Business stakeholders will be beating a path to your door with new use cases that you can quickly address!  For instance, one global financial organization originally purchased ASG DI for their Treasury Department to comply with financial regulations including CCAR and BCBS-239.  When data challenges of the upcoming LIBOR retirement presented themselves, the CDO was able to quickly address this new need with data lineage.

The use cases are many and boundless:

  • Spot and rationalize data bloat
  • Conduct impact analysis
  • Streamline and “risk-proof” legacy application modernization and cloud migration
  • Power analytics with trusted data
  • Comply with diverse geographical data privacy laws,
  • Comply with financial regulations, including FRTB, CCAR, BCBS-239, HIPAA and more
  • Find and repipe LIBOR rate references

 

These initiatives can all be managed more efficiently with data lineage.

If you want your company to be data-driven – and who doesn’t? – automated data lineage is a crucial capability for your arsenal!