Storage

Tracking changes in an automated world

One of the age-old questions to answer as an IT professional is, "What changed?" IT pro Rick Vanover shares some thoughts on change control, automation, and ways to report on changes.

In every IT circle, there are questions of what has changed and who did it. But what do we do about changes that are done automatically? In today’s infrastructure technology landscape, we are dealing with technologies that can (if configured) automatically do things that constitute “changes” in the environment. The best example I’ll put up as an automated “change” would be the task of migrating a virtual machine from one host to another. Another popular example is automated lab management software that will dynamically provision server systems to developers to use for code testing. This can be done with an administrator or user action, or automatically via management software.

When it comes to determining what has changed, do we need to consider also the automated tools in place? I say, absolutely. It is a two-edged sword in a way. On one hand you want to deliver the maximum amount of return for your investment in management and automation software. The other hand wants to ensure that all “change” activity is done in accordance to all policies and procedures. Most importantly, an automated “change” should not make a material change to the critical measures of the infrastructure. These are anything that affects availability, security, content management databases, and other key definitions of the infrastructure landscape for an organization.

If automated software is in place, what does the administrator do to determine what has changed recently? In some situations, the natural response may be to traverse through logs that are scattered through a number of places. But, there are options now to manage the automated changes that have occurred for objects through their lifecycle. For virtualized infrastructures, Veeam Reporter 4.1 has just been released to help administrators track this very issue for changes made either by automated mechanism, users, or administrators. To help the administrator answer the question of “what changed?” a 24-hour instant report is retained in the free product to answer this question for the day-to-day operational support role. The free version of Veeam Reporter 4.1 offers a number of features, including:

  1. A customizable dashboard and a publishable dashboard to an external tool such as SharePoint.
  2. 24-hour interval of instant reports of changes to the environment documenting what happened for each object.
  3. Agentless collection.
  4. Excel reporting on topics such as VM changes, permissions, storage management, and inventory.
  5. Visio reporting on storage and virtual infrastructure configuration.
  6. Storage capacity reporting.

A sample view of some of the change management reports of Veeam Reporter 4.1 are shown in

Figure A:

Figure A

Figure A

Any insight the administrator can have into what has changed is important. But lining up the right tool is critical. Veeam Reporter is one tool that has just come out, and I’ll be digging into it more in the very near future. You can download a free copy of Veeam Reporter 4.1 Free Edition from the Veeam web site. Another good resource is this datasheet which outlines what is available with the Free edition.

Underneath it all, there is a fundamental question that needs to be answered. Are automated “changes” okay for your environment? I say yes, partly because I want to get my money’s worth from the software I have spent and secondly because it is unavoidable. This is already the case with automated storage tiering. There is no way that the storage administrator will be able to effectively determine which areas of disk are busy and are candidates to move to a faster tier of disk at the sub-LUN level. I’d rather have my expensive storage array do that task 24 hours a day anyways.

Are compensating controls such as tools that provide us the log of what has changed enough? Or do you demand more? Share your comments below.

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

5 comments
ryumaou@hotmail.com
ryumaou@hotmail.com

Veeam Reporter looks like a great tool for *virtual* machines, but is there anything like it for actual, physical machines? Frankly, virtual machines are easier to roll-back if you're taking regular snapshots of them, which, realistically, I think should standard practice.

MyopicOne
MyopicOne

This is a useful discussion. In certain regulated industries where Quality Assurance is tasked with upholding company policy in support of regulatory compliance, change and configuration control and management is really needed. Many of the rules for managing those systems have grown out of manufacturing where traditionally a system was installed and all changes were disallowed - OS, middleware, applications - which can only work today if segregated from the rest of the corporate world. Maintenance of the validated state is critical to maintaining regulatory compliance; more important than improving overall system stability and security. In those environments, automated changes are a very hard sell because it is difficult to adequately test and document those activities to their satisfaction. Even under a risk-based approach it can be challenging - even GAMP has adopted a risk-based approach to address that unreality. Given the lack of support ($$) for automated testing in many organizations, the only options are to 1) not patch ever or 2) patch and pray. Guess which one happens? If the physical changes can be automatically detected then there is at least a known starting point for backout and service restoration. Being able to separate the important changes from the rest of the changes would be highly desirable - but currently it often requires analysis to identify those important items. Improved vendor documentation would greatly help such as a simple of list of registry locations (for Windows, of course), configuration files and database tables that are important to basic system operation. This would help answer the question "what *important* item changed" as opposed to "what changed" - those are very different questions. Simply identifying those items is a start, but identifying what specifically changed requires that their content as installed and configured be preserved. ITIL configuration management is conceptually sound but many of the tools supporting it are good from the infrastructure perspective but IMO weak from the application perspective. Some of the tools supportive of PCI standards are very interesting in this regard; over time the tools will get better. At this time the compensating controls seem to represent the most cost effective approach.

b4real
b4real

I don't know but, would imagine System Center has some visibility -> But I can't think of situations where changes would happen automatically. I.E. I wouldn't configure my MSCS cluster to automatically fail over.

b4real
b4real

Thats a good point that you raised of what *important* thing has changed. To be fair, I really don't care if my virtualization engine gives VM more CPU if he needs it. If it takes all that it can get, well of course that is another problem. The smarts behind how these are deciphered is important to deliver relevant data/reports, etc. I'll soon have a full review on this particular tool, so I'll get a feel for what can be examined.

MyopicOne
MyopicOne

...do offer automatic overwrite of changed files - XIntegrity for one. During configuration the admin specifies what can be automatically overwritten if a change is detected (i.e. to a known good configuration). For stable websites/systems that handle financial transactions I can see a definite benefit with this approach.

Editor's Picks