Monday, September 29, 2008

The Data Intelligence Gap: Part Two

In part one, I wrote about the evolution of a corporation and how rapid growth leads to a data intelligence gap. It makes sense that a combination of people, process and technology combine to close the gap, but just what kind of technology can be used to help you cross the divide and connect the needs of business with the data available in the corporation?

Of course, the technology needed depends on the company’s needs and how mature they are about managing their data. Many technologies exist to help close the gap, improve information quality and meet the business needs of the organization. Let’s look at them:



CATEGORY

TECHNOLOGIES

HOW IT CLOSES THE GAP

Preventative

Type-Ahead Technology

This technology watches the user type helps completes the data entry in real time. For example, products like Harte-Hanks Global Address help call center staff and others who enter address data into your system by speeding up the process and ensuring the data is correct.

Data Quality Dashboard

Dashboards allow business users and IT users to keep an eye on data anomalies by constantly checking if the data meets business specifications. Products like TS Insight even give you some attractive charts and graphs on the status of data compliance and the trend of its conformity. Dashboards are also a great way to communicate the importance of closing the data intelligence gap. When your people get smarter about it, they will help you achieve cleaner, more useful information.

Diagnostic and Health

Data Profiling

Not sure about the health and suitability of any given data set? Profile it with products like TS Discovery, and you’ll begin to understand how much data is missing, outlier values in the data, and many other anomalies. Only then will you be able to understand the scope of your data quality project.

Batch Data Quality

Once the anomalies are discovered. A batch cleansing process can solve many problems with name and address data, supply chain data and more. Some solutions are batch-centric, while others can do both batch cleansing and scalable enterprise-class data quality (see below).

Infrastructure

Master Data Management (MDM)

Products from the mega-vendors like SAP and Oracle or products from smaller specialists like Siperian and Tibco provide master data management technology. It features, for example, data connectivity between applications, the ability to create a “gold” customer or supply chain record that can be shared between applications in a publish and subscribe model.

Enterprise-Class Data Quality

Products like the Trillium Software System provide real time data quality to any application in the enterprise, including the MDM solution. Beyond the desktop data quality system, the enterprise-class system should be fast enough and scalable enough to provide an instant check of information quality in almost any application with any number of users.

Data Monitoring

You can often use the same technology to monitor data as you do for profiling data. These tools keep track of the quality of the data. Unlike data quality dashboards, the IT staff can really dig into the nitty-gritty if necessary.

Enrichment

Services and Data Sources

Companies like Harte-Hanks offer data sources that can help fill the gaps when mission-critical data is missing. You can buy data and services to segment your database, check customer lists for change of address, look for customers on the do-not-call list, reverse phone number look ups, and more.


These are just some of the technologies involved in closing the data intelligence gap. In my next installment of this series, I’ll look at people and process. Stay tuned.

2 comments:

Anonymous said...

Hey Steve. What about metadata management and ETL? Certainly has a role in managing data, no?

Steve Sarsfield said...

Great point. Metadata management is important to data governance, as long as we keep it to ourselves. Don’t invite the business users to your meetings when you talk about data models and schemas; their eyes will glaze over. Also, we go into any initial meeting about metadata assuming that we can’t trust it. We can’t be sure that FIRSTNAME really contains a person’s first name, for example, unless we’ve profiled the data.

Disclaimer: The opinions expressed here are my own and don't necessarily reflect the opinion of my employer. The material written here is copyright (c) 2010 by Steve Sarsfield. To request permission to reuse, please e-mail me.