Blog > June 2019 > Applying Intelligence to the Data Flow Analysis

Applying Intelligence to the Data Flow Analysis

As companies try to scale data to the masses through self-service, the backend technology cannot rely on tribal knowledge. Even partially automated solutions, such as business glossaries or project-based metadata tools, are falling short of the high demand for rapidly produced datasets and business intelligence (BI) insights. By combining AI with automated scanning capabilities, organizations can scale and speed up the process as well as find new insights, relationships and patterns from the data. ASG is bringing more value and agility to businesses by infusing artificial intelligence (AI) into our data intelligence (DI) lineage capabilities—and our customers are finding increased efficiencies while reaching their lineage goals.
In a recent AI/DI webcast with Trestle Group and ASG, we reviewed five emerging scenarios for AI and DI:
  1. Automating lineage gaps
  2. Aligning the business assets to the data
  3. Data quality analysis and assessment with AI and lineage
  4. Detecting fraud
  5. The intelligent data flow
Traditional lineage projects can be notoriously difficult depending on the complexity of the data environment and how unruly the data manipulation has become. In this scenario, we always tell our customers to not expect 100 percent automation in the first iteration. Instead, most customers are happy to uncover exactly what’s happening to their information when they take the initial “snapshot” of the data. However, one of the most time-consuming tasks in these projects is to validate the lineage and work with SMEs to identify the gaps. Applying intelligence to the data flow addresses this challenge by reducing the amount of time required by SMEs, increasing how far and wide your data traces disseminate and even bridging some of the unknown data gaps you are faced with.
Applications for AI and DI
AI and DI make it simpler to enable and streamline “vertical lineage,” which involves associating and embedding business information into the data lineage to the point of becoming an Information Supply Chain (ISC). Examples of vertical lineage include knowing where the forecast ratios originate or that your business policies and rules align with these ratios. It starts with linking business definitions and assets to the data elements and subsequent lineage. Typically, vertical lineage is a manual task carried out by data stewards. AI finds the similarities between business and data, learns the patterns and makes business-to-technical linking suggestions with great accuracy. In this way, AI lessens the burden and cost it takes for data stewards to put the puzzle together and sustain it going forward.
AI also helps fuel innovation within the enterprise. Data-driven initiatives need more higher-quality data, yet how tedious is it today for data scientists to check the data quality on a new dataset? How do you know that the dataset is complete and compliant? In most cases, the data scientists rely on SMEs to find good data for the business requirement. With the right tools, however, AI and DI can automate the Q&A process describing where the data came from and understanding what type of statistics are relevant for this dataset, and then pinpoint the outliers. Going forward, the analyst can set up alerts for when the ISC has changed, thereby increasing the quality and trust in the entire dataset.
In general, whether you are studying multiple or individual ISCs, using AI will help you uncover data complexity issues, inefficiencies in the code, manual interfaces, third-party tools and third-party data redundancies to name a few. It brings a new, ongoing automated method of detecting the inefficient flows of the data, the inaccuracies of the code and the misaligned business objectives of the data itself. At ASG, we like to call this the creation of an intelligent data flow!
To learn how you can construct intelligent data flows with ASG, visit this product page and explore our Data Intelligence capabilities. For more information on combining AI and DI for data management, watch our webcast here.