PCs

What constitutes a change in today's IT environments?

Today's technology mix is, in most situations, generations ahead of the typical change control process. IT pro Rick Vanover raises the question of what really constitutes a change nowadays.

In a recent conversation, a topic came up that has manifested itself a number of different ways to me in my IT career. The main issue was that one organization required a change order to migrate a virtual machine from one host to another. This would be leveraging a feature such as VMware’s vMotion technology that will migrate the active state of a running virtual machine from one ESX(i) host to another, leveraging shared storage with no downtime. In my professional experience, I’ve never had a situation where a feature such as vMotion cannot be run on a production virtual machine.

The issue at the root of that discussion is what constitutes a “change” in the IT environment, however. There are changes in the virtual machine during an event like that, such as it (presumably) now has more CPU or memory resources to work with on the new ESX(i) host. There may be an ARP table update as the virtual machine would now be on a new switch port. Finally, that same virtual machine would be inventoried on a new ESXi host; within the same cluster. These “changes” are academic at best, but in the scheme of traditional IT practices; the core running configuration of the virtual machine has not changed. Other unchanged attributes of the virtual machine include operating system configuration, TCP/IP address and other network attributes in the guest virtual machine, DNS name, configured memory and processor resources, and finally the application inventory.

The virtual machine migration example is repeated en masse in today’s IT landscape. Other events can include:

  • putting a VMware ESX(i) host in maintenance mode
  • adding a DNS server to a production server’s IP configuration for additional resolution
  • resetting a tape drive out of production timeframes
  • replacing a failed hard drive on a RAID array for a rebuild operation
  • deleting a stale user profile on a server
  • performing a dynamic expand of a virtual machine disk of a running virtual machine to address free space issues.

There are scores and scores of situations like this that aren’t really changes to a production workload’s running integrity, but are in fact functions of the technologies we have today. What do we do about this gap?

In my experience, I approached these challenges with a mechanism I referred to as a reserved administrative function. Basically, these are operational practices that are regular, predictable and within the capabilities of the technologies we approve for use. If these operational practices were identified to be within the realm of a reserved administrative function, they did not need prior approval through a traditional change management board. This doesn’t necessarily mean they didn’t need approval; it could be a peer review or management decision. Further, tracking must be met. This can be in the form of a system log, support case, or a physical log.

How do you approach changes in the new era of IT technologies? Am I off base with the concept of reserved administrative functions? Share your comments below.

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

22 comments
slickfiche
slickfiche

What seems to be missing in a lot of these posts is that one important thing we are trying to maintain is service availability and therefore value of services to our customers through Change mgmt, by wrapping process and control around operational (in this case) activity. In the past techs rebooted at will. As a result the perception of the service was low, and so therefore was the value of the service. Never mind that in those cases if a recurring problem existed it could not be identified or analysed as it wasnt tracked. Wind the clock forward and IT now has to demonstrate its ROI to the business, ITIL offers suggestions to justify the value that IT does actually offer the business, indeed the framework offers a way to show that increased value is possible. But it is only a framework so we adapt it to suit our needs, the important thing is we continually monitor our processes to see where improvement can be made, and therefore increase the value we offer.

cmrpm
cmrpm

A change is a change and it should therefore at the very least be recorded somewhere. I agree too that not all changes need to be submitted for approval unless the environment is extremely risk-averse. The problem I see is in the implementation of change controls. The more effective ones are flexible and originate from the inside where it???s acknowledged a change control process can help. Only then will you find harmony and buy-in between those that do the work and those that approve the work. Regardless of the implementation, good or bad, each must have detailed historical change auditing in place that spans as much of the infrastructure as possible. Only a detailed record of changes made to an environment will provide the opportunity to see where the change control process needs adjusting and how changes may have impacted the organization. Chris Rich Product Manager NetWrix is #1 for Change Auditing: Simple, Lightweight, Affordable

dan.sandel
dan.sandel

Documentation is key. If you take the original build document then follow all subsequent change controls that should equal your system in its current state.

deirdre.donovan
deirdre.donovan

Reboot a server without a change review and notice to the customers (internal and external) and systems administrators affected by the reboot? What about when the server reboot interrupts service to customers or developers? The whole point of "standard changes" is that they don't affect people's ability to work or to access resources.

Charles Bundy
Charles Bundy

Spot on wrt to predefined changes like server reboots, but I always got nervous when the change board started adding maintenance windows to the list. I wasn't always certain that critical patch management fell under the assessed category! (unless you had a configuration lab certifying that the update in question was tested within a viable simulation of the production environment...)

Juan Ferzara
Juan Ferzara

You could say that the changes that you mention can be viewed as maintainance. As such, there are to be expected and could not need approval. However, it depends of the shop (or IT department ). Logically, such maintainance requieres a log and a audit trail. Something that involves more than 50% of modification of a certain asset, however, can be viewed as "change". Upgrade one program in a image is a maintainance, but upgrading a image to the last version of a OS is a change, under this idea.

Bladex64
Bladex64

I'm a bit rusty on this side of things but I like the ITIL idea of "Standard Changes" i.e. predefined changes where the impact has been assessed and are in effect pre-approved. These can be templated so minimal admin is involved but they maintain the audit trail and visibility/planning.

Charles Bundy
Charles Bundy

I often shake my head at the implementation of change management, esp if the approval boards have no clue over what they are approving. Love it when you see a member stamp "approved" on a documented CM that already took place as part of an emergency response! :) Change is documented as part of configuration not events. You are exactly right that routine events such as a server reboot do not require CM approval over and over. In those cases "Once is enough."

chris.parker-jones
chris.parker-jones

Three types of changes in the Operational Environment (Real changes, Events and day to day changes) 1. Real changes which require downtime or effects large portions of the production environment and need a communication process because they are planned need the change process the CAB - upgrade the app, new server, change IP etc 2. A failed device and its replacement is not a change it is an event, hard disk failure, reboot of a server because of a memory leak, use the change log to document, no CAB needed 3. Operational changes which require manual intervention should be documented in a change log so you know what has happened but these changes do not require a CAB, change to a system parameter, DNS name change, adding or removing a printer - This means the change can be reversed

b4real
b4real

And it is difficult to balance everyone's needs in an organization. How do you go about addressing this topic?

miroc
miroc

I completely agree Dan... the design should reflect what constitutes a change. I would consider the entire infrastructure a single entity, and the infrastructure was likely designed to be so. If I were to move a virtual machine from that infrastructure to a completely different implementation, that might be a change. Moving a workload within that infrastructure I wouldn't think is a change. I think one of the problems in IT is the lack of understanding who the audience is for a particular piece of documentation. If you are documenting a change, aside from giving stakeholders a chance to understand the change prior to its execution who are you expecting to read the documentation? As an example, I would probably read change documentation to help with troubleshooting... vMotion information is likely not going to help me.

Charles Bundy
Charles Bundy

In an ideal world... But we have things like Altiris or OCS to deal with the reality of operations :)

Charles Bundy
Charles Bundy

If A CR is approved that server A will be offline On Tuesdays from 21:00 to 21:15 and this is communicated to stakeholders do we really need to repeat ourselves? Hence "once is enough" is the watchword...

Bladex64
Bladex64

I wasn't specifically talking about server re-boots, though I guess that would be part of your impact assessement - if you have redundancy these could still be a standard change.

Charles Bundy
Charles Bundy

Is to make everyone's life easier by streamlining the process when it shouldn't effect the end user. I'm curious about the "systems administrators affected by the reboot". Why not include the network analysts who will go crazy when the nic goes offline? It would seem they would be more in need of notification than the folks pushing the shiny red button :)

b4real
b4real

If we MUST patch and update, do we need to have approval to do the thou shalt operations?

b4real
b4real

I.E. -> Define to organization that these are new maintenance procedures..

b4real
b4real

Every change, even operational ones that don't need prior approval should still must have to ability to be tracked.

b4real
b4real

One old-school example I had is user account provisioning. If we hire new users all the time, we need to add them on demand. But Active Directory is a production system, right? So, is every nitpicky change (such as user add/delete) to require a change order -> I hope not!

b4real
b4real

Have you ever had to advocate for a new feature/technology that is NOT part of the change?

b4real
b4real

Good point.. I frequently extend the boundary of the entity from a single server to a cluster.

slickfiche
slickfiche

So an agreed maintenance window? Sure, good idea. However once is not enough. What happens if there is no change work scheduled for one particular Tuesday? And you have a requirement for high availability of a service? At best, if you only publicise once (and btw once...ever? or once every so often?) and if everyone understands that the service is unavailable and on one occasion no work is scheduled then you have taken out the service for that time for no reason, its costs to have downtime so make the best of it. At worst customers forget, so if you publicise once they WILL forget 2 months from now and then they have that urgent report they need to have in Wednesday morning and require network or remote access for instance and its down? First thing they will do is call the SD, irately. Its bad communication and bad service, therefore low value and that is what we are trying to maintain, (increasing) value to our customers.

Editor's Picks