E-Commerce

Is your business intelligence suffering from the Information Paradox?

Is your enterprise finding it difficult to get valuable information from your data? If so, you might be feeling the ills of the Information Paradox. Find out what some of the warning signs are before your organization invests in a new enterprise system.

Despite the huge investment businesses make on new enterprise systems, which promise better information flow and flashy analytical reports, the fact remains that it is still quite common—and maddeningly frustrating—that many of these same businesses struggle to obtain critical high-value information. These enterprises are victims of the “Information Paradox.”

If you're thinking of investing in a new enterprise system, ask whether the investment will improve your capability in two fundamental areas:
  • Capturing and recording your business’ transactions and analysis
  • Providing accurate reporting and performance measurement

If your new system doesn't pass this test, chances are it will only create an "Information Paradox": the chronic inability to access information despite spending huge sums of money on "information" systems. In this article, I’ll present an Information Paradox scenario and three reasons why enterprises find themselves faced with this problem.
For examples of how organizations can use business intelligence, read “The intelligent organization: An introduction to BI.” In future articles about business intelligence, we’ll look at solutions for an organization’s business intelligence problems.
An illustration of the Information Paradox
Recently, a friend of mine—the chief operating officer of an industrial service company—told me of his ongoing frustration with his inability to obtain daily labor-cost information across 20 business units. Control of this cost was essential to making or breaking his financials.

“You know how it is,” he said. “You just can’t get good numbers. The crazy thing is how much we’ve spent on information systems trying to fix it.”

Heard often in the halls of many (if not most) organizations are similar statements that reveal the ubiquity of the Paradox:
  • “We have mountains of data, but we can’t get access to it.”
  • “We spend a lot of time at our operations meetings arguing about who has the right numbers.”
  • “Everyone knows some of the data we have isn’t very good.”
  • “We don’t even make requests for special reports; we’ll never get them. They’re always wrong or late. We end up extracting our own data and running our own reports."

Organizations suffering from the Information Paradox likely fell victim due to one or more of these three common issues:
  1. Data is fragmented across a number of databases.
  2. Data is fragmented across a single database.
  3. Translating business questions into database queries requires special expertise.

Problem one: Data is fragmented across a number of databases
While reviewing Florida’s recent problems during the presidential elections, several interesting facts came to light. Newsweek reported that because of database irregularities in Miami-Dade County, approximately 6,000 people listed as legitimate voters in 1998 were either convicted felons or dead.

In June 2000, following a quick, fix-it scramble to blend data from several databases, 12,000 registered voters from the same area were mistakenly labeled as felons and told they couldn’t vote.

Data problems like these can happen because traditional information systems tend to fragment data by decomposing the “whole" entity into separate databases. The view from a single database is therefore incomplete, making decisions based on such a database more speculative than factual.

Why the fragmentation? Organizations are traditionally structured around functional areas like finance, manufacturing, marketing, and purchasing. Over time, computer applications are designed and deployed to automate these functional areas one by one.

Applications deployed in this manner are often called “legacy applications.” What results is a hodgepodge of data files and databases, each serving the data requirements of a single application.

Accordingly, no single database contains the complete picture—about a customer, a market, a product, or an employee. In addition, individual spreadsheet files and small databases may exist by the hundreds, created by power-users to “fill the cracks” between the legacy applications. Put all this together, and it’s likely you have a picture close to depicting the state of your own enterprise information.

Problem two: Data is fragmented within a single database
Most information systems are designed to record business information quickly and efficiently. A hotel system, for example, records room reservations in a few milliseconds. This efficiency is important because often the system must satisfy the requirements of many users simultaneously.

Data for these systems is frequently stored in a normalized relational database, characterized by numerous tables related by “keys.” For example, a customer’s name, address, contact, and other information is stored only once in the entire database—in a “Customer” table.

This individual customer record, perhaps thousands of bytes in length, will from now on be referenced by a unique primary key only a few bytes in length, which works as a kind of shorthand. In the same way, the travel agent’s information is stored in a “Travel Agent” table represented by a unique key, and so forth.

Now, when your reservation is entered, it can be done simply and quickly by writing a single record in a “Reservations” table, a record containing only “keys” and nonredundant data. This system of tables and keys now makes it possible to record complex, lengthy information very quickly and efficiently using only a few bytes.

Suppose, however, we wish to create a report summarizing reservations made last year between March 1, 2000, and March 15, 2000, then sort this report by travel agent, city, and customer’s last name. We’ll discover that reconstituting this information requires writing complex queries containing lots of SQL “join” statements in order to put the information contained in multiple tables into a coherent whole.

These queries are difficult to write and, when “run,” can adversely affect the reservation system's performance.

Problem three: Translating business questions into database queries requires special expertise
Generating a special report such as the one above may be problematic for two reasons. First, complex SQL statements are required to query the database. As you’ve seen, that’s not easy when databases are normalized because the data is spread across multiple tables and multiple databases. Thus, getting answers to simple business questions can get quite complicated and involve temporary databases and multipage queries.

Second, database technologists writing the query must understand the business question. If they don’t, you may receive an incorrect answer. Often, the fallout from this lack of effective communication means you’ll wait days or weeks to get the correct report.
Do you face some of the same problems illustrated here? Start a discussion below or send us an e-mail.
0 comments