Software

Ask something
Discussion Locked

Change Control in a tight times

0 helpful
11 replies

by david.planchon 2 views

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Change Control management for Windows Servers environment.

I'd like to hear from my peers out there: how do you do it?

Let me draw a picture of my environment: 3 Sys Admins for approx 130 production Windows Servers. Needless to say our weeks can be very interesting at times.

Recently we've been asked to introduce a Change Control process into our workflow. I don't have a problem with that. However, our procedure requests that any (I do mean ANY) change to the Operating System, Hardware, or non Business Applications residing on the server be subjected to a Change Control Request.

We've discussed this in our team meetings and we're trying to define when we should put in a Change Control Request and when we don't need to.

I'm still working on this 'definition', so far I've got the following:

when deciding if one needs to create a Change Control Request form prior to implementing a change on a server:

1) the server is live and in production
2) the server has been identified as a DR server
3) the possible affect on the business if the implementation fails and the downtime overflows past the maintenance window.
4) has this change been successfully applied before, if not has it been tested on a non production server
5) the change to be implemented is a new piece of soft/hardware, or a full new revision level than the current one
6) the change affects more than ?xyz? people

Some other things to think about before making a change to a server are:
1) it?s past reliability with changes
2) it?s age
3) the number of people using this server might vary during the month
4) look at the big picture, does this server feed information to a more critical server
5) what are your back out options

11 replies

by david.planchon

Sure ITIL helps with best IT business procedures and processes.

What I'm wondering is what or how other people are fitting Change Control into their Windows Servers maintenance and management schema.

Thanks for the post though.

by tony hopkinson

You'd be sorry if you didn't.
Take account of differences in you os/hardware mix between boxes. Seen people forget that, test it out in dev , all goes smoothly, put it in production total mayhem.

by david.planchon

I've seen that happen a few times on OS and hardware upgrades, but many many times with software.

I'm not getting much feed back on this posting. I was hoping to generate a debate here... ;-(

I haven't seen much conversation on/by Techrepublic members about Change Management and Change Control for System Administrators.

by dr dij

that bulk of admins are in shops that have no or rudimentary change control, tending to be smaller companies?

by tony hopkinson

It's a difficult one. One of the things you do need is some sort of impact threat level to be assessed and then tailor your change management against that.
I worked at one place where they were completely anal about it and the procedure for an OS upgrade was that same as for slotting in a new CD ROM.
That sort of thing just makes your system look stupid and people tend to be naturally averse to wasting their time with dumbass requirements.

I've also worked in 24/7 operations, one of the big things there is that once you do some change, you have an extremely limited window in terms of backing out. That imposes restrictions on how and when you do certain types of change.

OS / Hardware upgrades were something I was less worried about as we were in a dual environment so I could prove on the standby and then swap it in.

In the unlikely event of me missing something critical, just swap it back out. So the change control requirements were dependant on the environment.

Now an upgrade to the inhouse data collection and distribution software, was a major nail biter, as it was next to impossible to test for all potential scenarios. This meant generally if it didn't fallover with the first five minutes any difficulties would be patched, because losing the collected data with a restore ceased to be an option in a very short space of time.

You've already mentioned the considerations and seem to have covered most of the bases, but the options you have in impelmenting a change impacts chnage control itself. In the above if I'd only one server or the servers were different, I would have been in nowhere near as good a position.

by kfellows

Every time I have implemented change control procedures the first pass ends up with an overly complex process and series of sign-offs. Largely, this is because the people requiring Change Control don't trust the changes being made. Eventually they will learn what a real change is versus manipulating configuration parameters. Both types need to be traceable and audited but they do not need the same level of scrutiny during the approval process.

To speed through the learning curve I break down the changes into categories:

1. Operating System and Program Code Changes
2. In-production service parameter changes to improve performance, reliability, or security
3. Changes that adversely affect users
4. Emergencies

For Category 1 the scrutiny and approval needs to come from all of IT. Application Developers, Systems Admins and Engineers, and Security all need to agree and be prepared for these changes. If the change is going to involve down-time then the down-time only needs to be approved by the business (partial application of category 2).

For Category 2 the scrutiny and acceptance has to come from the user community for anything that will negatively affect them which includes down-time, Application upgrades (they may need feature training, the data may be at risk during the upgrade, etc.) but this does not need scrutiny and approval from everyone in IS, just those that support the application.

3. The business and user community need to understand that there are changes that can happen that don't need their scrutiny and approval, or even all of IS, if they are not negatively affected by the change. These changes are things like adding an additional route to a router for higher availability, or creating an administrative OU in Active Directory. These changes need to be scrutinized and approved by the system owner and documented but they can be done outside a formal change control meeting.

4. The final category, Emergencies, is tricky because it often gets used as a catch-all. It should almost never be used. An organization that plans and deploys in an orderly fashion will seldom have emergency changes. In fact, once your change control practice is in place for a while you'll see a drop in the number of emergency requests. Emergency scrutiny and approval should come from a very small group of IT and Business decision makers who are available 24x7 to basically take the responsibility for the change. Since it is an emergency it means no testing has been done and little if any due diligence so someone needs to take the responsibility for saying it is truly an emergency and the risks for delaying the change outweigh the risks of performing the change.

Change Control has to actually be manageable in order for it to truly work but keeping change to a minimum and documented does keep systems running smoother.

by vawns

Hi David,

I think your best bet is still having an ITIL Change Mgt process. You can get further info here:

http://www.itsmwatch.com/itil/article.php/3527471

It doesn't have to be complicated but if you have a change controll system then all the change information is stored and approved in the same place and there is a paper trail should any issues / questions occur. It's also much easier to deal with failed changes with this process in place. Here are some of the things to look at when making a request for change (RFC)

Change Detail Explanation
Title Does it make sense?
Description Does it match the title and explain clearly what the RFC sets out to do?
Reason for change Why is this work been carried out? How important is it? Is it business or IT driven?
Impacted CI What will be impacted by the change? I.e. infrastructure, users, buildings etc
Impact of change not being implemented Would there be any adverse business impact if the change was delayed or backed out?
Pre implementation testing Have UAT and SIT testing been carried out? If not, why?
Implementation Plan An overview of the work being carried out (can be attached if necessary) and documentation of roles and responsibilities so that CM can understand what is involved in the change
Post implementation verification Has the change been applied successfully?
Risk Are there any major risks associated with this change? If so can they be mitigated against?
Risk severity Is this a high / medium / low risk change?
Back out plan Can this change be backed out if something goes wrong? If yes, how long will it take? If no, is there a workaround?
Will change impact other environments? I.e. will the change just impact the production environment or will it impact other environments (C&R, Development, Training) too?
Has this change been mirrored automatically to the C&R environment? If not is work scheduled to take place? Or if not, why not?
Locations impacted by RFC Single site, multiple sites etc?
Impact analysis Will this change cause any of the following:
Financial loss
Reputational damage
Compliance failure
Risk to health and safety?
Authorisation Has the change been authorised by all appropriate signatories?
Communication Has the change been communicated to all key stakeholders? Is the Service Desk aware so it can send out a change communication e-mail to users if necessary?
Change closure Has the change been completed successfully?

Hope this helps,

Vawns

This discussion is closed to new replies.