If the Microsoft antitrust trial did nothing else, it finally convinced IT managers (and their bosses) of the need to purge old e-mails and other electronic documents. This was no small achievement. It took the sight of various executives trying to reconcile their memories with e-mail threads to overcome the natural “packrat” instincts of many IT professionals.

More and more organizations started developing electronic record retention policies and drawing up schedules for periodic purging of old files. Record management became a hot topic among knowledge management (KM) vendors and consultants.

Unfortunately, not every organization can afford a full-blown KM system. In fact, many IT departments are being asked to deal with records retention and purging as another set of projects, a feature built into new applications and retrofitted into old apps. In this column, we’ll look at three factors that complicate record management.

The complicated question of ownership
When trying to apply a record retention policy to a particular application, you have to ask, “Who decides how long you keep the data?” The obvious answer is usually, “Whoever in the organization owns the data.”

The problem is that determining ownership of electronic records, whether complete applications, specific files, or even individual rows in a database, is more complicated than it looks.

Consider an individual record from a transaction database, one that tracks purchases in a retail store. Who owns that record? Well, depending on whom you ask, quite a few people could claim ownership. They include:

  • The customer who bought the item.
  • The sales associate who ran the bar code through the register and processed the customer’s credit card.
  • The network engineer who monitors the satellite network that connects the company’s retail outlets to the company’s warehouses and main office and who sent that transaction as part of the daily activity.
  • The data center manager who oversees the company’s storage area network (SAN), which stores the transaction.
  • The accounts payable clerk, who must reconcile the transaction with payments from the customer’s credit card company.
  • The CFO, who produces monthly revenue reports based on the contents of the transaction database.
  • The customer service rep, who needs to be able to view the transaction record in order to answer customer billing questions.
  • The payroll administrator, who uses the transaction when calculating the sales associate’s commission for the period.
  • The corporate counsel, who provides guidance on legal requirements for record retention.
  • Other “controlling legal authorities.”

As complicated as that sounds, it can get worse. For example, many organizations have to comply with external companies or government agencies that establish requirements for electronic record retention and purging. In our retail example, the company would have to comply with the major credit card company guidelines for resolving charge disputes, which specify how long retail transactions are available.

In addition, the retailer has to keep transactions for ready access by the public accountants who perform the annual audit so that they select random records for inspection.

Companies that do businesses with government agencies will find themselves bound by local, state, or federal vendor record retention regulations.

I cut my teeth in the telecommunications industry. Depending on where they operate, long distance companies have to keep archival quality copies of customer invoices. This is not only for handling customer requests, but also to support inquiries from the state Public Service Commission and law enforcement agencies (who may subpoena an individual’s past bills).

How many copies? What format?
As the above example shows, one of the difficulties in designing and implementing an electronic records retention policy is that an organization might keep many copies of the same transaction in many different applications and databases. While a good data warehouse structure can minimize the duplication, it’s unlikely to eliminate it.

Adding another wrinkle to the problem is the fact that data takes many forms. Many retail stores accept a digitized signature for credit card transactions. Many public utilities can’t simply store the underlying transactions of their customers. They are required to store the invoice in its original form.

What do you do?
If you’re tasked with designing and implementing an electronic record retention policy for a specific application or process, do the following:

  1. Flowchart the process and try to identify the internal stakeholders.
  2. After speaking with the internal stakeholders, establish any requirements by external agencies or companies.
  3. Identify multiple copies of the same data, and label the various formats in which they are stored.
  4. Determine where the electronic records are stored, and forecast how your retention and purge policy could impact SAN capacity.
  5. Consult frequently with both Accounting and Legal: they will probably have the most elaborate and detailed requirements.

That’s just a start, of course. After all, there is a reason why KM products are so expensive! However, these guidelines should provide a framework as you think about your data project.