Discussion Locked

What/how would you document to guard against 1-person failure point in IT?

0 helpful
35 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I am a single-person I.T. Department for a non-profit child welfare agency, managing 150 users, 100 desktops, 6 servers, and all aspects of technology. We have our email, database, app, file, and print servers, phone system (within a couple of weeks moving to VoIP), accounting and case management systems all in-house. The only thing I don't host or manage is the website, due to lack of time and flukey local utilities.

I have not been what I would call "diligent" when it comes to keeping change-logs when servers are set up and apps installed- I tend to keep the most relevant stuff in my head. This scenario, however, is a disaster waiting to happen. If something were to happen to me, 1) nobody in this organization would be able to fill my shoes even temporarily (social workers...) and 2) I would think a new person would have a hard time getting acclimated to the network, due to the lack of *settings* documentation. I did create a "LAN Book" that contains things like server names and hardware configs, LAN physical layout (which is quite convoluted), services and apps running on particular servers, and DHCP/DNS settings.

My problem is this: I built this network from nothing, and this has been my only work experience in the IT field, so I have no others to look to. I have been tasked with creating a manual that would allow someone to run the network in case I die in a car wreck on the way home.

What would you include, and where in the world would you start? To me, this seems like an insurmountable volume of data... I know that my CFO wants a domain admin account password kept in a safe deposit box, but where is the line drawn- do I include that the fiber between buildings is 6-strand 6.25 micron direct plant with SC ends, just so that person knows?

What say those in the trenches?
~Thanks,
~Dave

35 replies

by mr.wiz

I'd document everything down to the smallest detail. If you want someone to be able to step in in an emergency, they'll need to know everything you know.

by wadloper

Documentation is a somewhat necessary evil, which techies normally abhor. It is wise, however, to set time aside for documentation to make your own life and that of others easier. Depending on how complex the network is, and how many well-qualified staff are around you can decide that it is inefficient to dedicate resource to documentation. Contrary to that I have always maintained that when we are all in a bus on a day out and the bus falls into a ravine and we're all killed, a new team should be able to restore normal support within a week. This calls for documenting what skill sets are needed for the replacement team, and gives you an insight to what your own skill set is. It will highlight your areas for development and opportunities to stay one step ahead. If higher management ask for information, you have that information at your fingertips, and if there are problems which needs an outside company to help resolving the problem, your own fault analysis can save an enormous amount of expensive consultancy time.
As an example: a company comparable to ours spends about ?100K on consultancy per annum, because they have not been documenting their network, nor do they have procedures in place to log/track network changes. This company has a relatively high turnover on staff, and is destined to outsource all services to an outside support company, costing even more. It is very detrimental to customer service, that's why we are documenting and staying employed.

by faradhi

Everything i know does not help. First, you should explain that a non-technical person will not be able to do your job. There is just not a way to anticipate every problem you encounter on a day to day basis.

So what I did when I was in your shoes is

first I documented restore procedures to get you where you are today. This includes admin passwords, service accounts and their passwords, where to find replacement software, images if you have them, and backup tapes. Also, a list of vendors, licenses, support contracts with phone numbers. Disaster recovery will cover a lot of documentation you will need to do.

Second, Document how-to's when it comes to common day to day operations. How to create a user account, how to restore a desktop from image, how to restore a file from backup, how to reset passwords, etc. (this is to help the non techie survive until the calvary comes)

Next, document any specialized scripts and processes that you created. Assume a more technical person will be reading this. Your employer could hire a contractor if needed.

I am sure others will give you more. I hope this helps get you started.

Good Luck.

by sstever

Documenting how-tos seems like reinventing the wheel to me. I've had bosses who've asked me to do that and I've refused. If something happens to you, a wise manager would bring in a professional IT person...not someone who has to be taught how to create an account, etc. You can't be expected to write a "how to be an IT professional" book. If it was that simple, everyone would buy the book and do what we do...lol

by spring_chicken

In my opinion this is clearly not a realistic option, in order to find that perfect ideal 'professional' IT person is not an easy task at all for HR. It could take weeks to find someone who could fill in. We as IT professionals are not the best writers or the best at documenting but we must give our best efforts. The IT professionals that I work with who don't do documentation do it for a purpose. That purpose is never looked on as a positive from management's perspective because it shows that IT professional doesn't want anyone to know what he does everyday. This is essentially 'job security' so he doesn't have to worry about getting fired. If management doesn't know what he does everyday how could they find a replacement to fill in perfectly?

It is in the best interest that you compile the documentation at your best efforts. It will give you good experience and prepare your company for the worst which isn't a bad thing.

by mllwyd

We have to document how to create accounts and other such things that any "professional IT person" could do because we do it in special ways. For instance, our programmers use specific fields in AD to pull data, therefore it's very important that you put the correctly formatted info in those fields. We also have to add users to another half dozen systems. No, you don't have to document how to add a user to Windows, but you have to document how to add a user to your system.

by jaqui

It's a huge task overall, break it into logical sections:

1) Software, what apps are used [ including versions ], admin passwords, configuration settings. where original cds are, product keys, manuals...

2) Hardware, list each piece of hardware, and serial numbers.

3) layout. detailed description of the network layout.

4) Backups Ect.. Detail each policy for system management, and why it's in place, how often are backups done, where they are stored, why is this timing used..

I listed those in the order of importance, since they need to have the admin access to do anything, and having documentation on the software and confiiguration is going to be the biggest help from the start.

as the Previous posted mentioned, everything needs to be documented, but if you prioritise the task into smaller sections it reduces the impact of the size of the task to document it all.

by tony hopkinson

for another employee.
I don't care how good you could get the documentation and keep it up to date, during your time off from curing world poverty, solving cold fusion and inventing an FTL drive. Handover will still be horrible.

by chipshopman

If your CFO expects you to do this and your day job, it's never going to get done and it'll be like painting the SF bridge - never ending due to the changing technical environment. Not only that, but if you do it, you'll miss stuff - you're too close to the coal-face and you'll write procedures and documents that inherently take things you know about the place for granted and they'll fail should they have to be used.

Your CFO is obviously concerned about this single point of failure so I suggest you go back to him and persuade him to spend money on a technical author. A Tech author will speed things up and actually deliver it. They'll be expert at sucking information out of you and writing it down in the most useable format. It'll be money well spent and it'll let you do the job you're paid for and get the documentation done quickly.

Regardless, you should prioritise your documentation in accordance with the most likely scenario first and the least likely last, i.e. it's most likely that you'll be absent for a protracted length of time or leave the company. It's least likely that someone will blow the place up. So, concentrate initially on things that someone would need to know on Day 1 if you left the company - how would they maintain business as usual? Then concentrate on the other stuff.

Another option: Get close to one of your suppliers and see if they're interested in providing ad-hoc cover for you. If there's a bit of cash in it for them, they might be interested and they could put someone on site to learn about what goes on...

by jdmercha

Check the local Universities. Perhaps they have an IT or Engineering student in need of an unpaid Internship. If you can find the right one they may be able to write this documentation for you. Or at least take over some other duty that frees you up to write the documentation.

But if you get a bad one you may end up wasting time training them.

by rob mekel

This makes a lot of difference whether it would be just anyone or an IT-skilled person.

The breaking up in to smaller parts, as suggested by Jaqui is a good thing to do. If it is just anyone then add a manual on how to operate the simple things, user enrollment, adding authorization, etc. Or, but do check with your CFO if your company can effort it, setup a minimum of requirements as what this new person has to have on IT-skills.

Also the suggestion of n0tl0b to hire some to help you is a good one, so you can document the total system, as it will be time consuming.

As you can't document 100%, there will always be a certain minimum of what a replacement person will have to have on IT-skills.

Rob

by bschaettle

OK, you've been given a mandate by the CFO for system documentation. It's become a priority for the organization, so they're clearly willing to pay for it in terms of increased project times. Add about 20% to the time estimate for every project for writing the documentation. Then be sure to actually write the documentation.

Realize, of course, that once you develop all this great documentation the CFO can easily outsource your job.

by mac1ne

yes, you want the organization to be able to function if you should you get hit by a bus, but keep in mind that bus driver could be your boss cutting costs.

by crikey

Thats right. If you document everything it will take just short of forever. Your boss could decide to replace you but dont fear: IF they have less skills no matter how much you document they couldnt do it. Document a Network Topo, Admin Passwords to all their components and your equipment modle numbers and if need be components inside. If someone is going to stepin they need to be qualified.

by bork blatt

If you want to keep your job, be valuable. Don't try to find artificial ways to hang on to your position. Usually the people fighting to protect their turf the hardest are the people who least deserve to be there in the first place.

1. Make sure you are an asset to your company. Don't just think about "your patch", but offer advice and work to move your company forward. A manager who fires this type of employee to cut costs is insane.

2. Make life easy for a replacement. Not only is this good if you want to leave, but it enables you to be promoted. Irreplaceable people are not promoted for that very reason - they cannot be replaced. If you want the same job title for the rest of your life, simply don't document, don't learn any new skills, and don't plan for a new position.

3. Stop worrying about stupid managers. Yes they exist. Yes they can make life painful, but if they are truly incompetent, they won't be around forever. Do the best you can within your boundaries, and know that you are doing your best. If nobody higher up in the organisation is capable of spotting a problem manager, you might want to find another company.

To summarise - be valuable. Be indispensable, not irreplaceable, and people higher up will take notice. If they don't, you probably need to find a better company to work for, so stick it out until you can move. If you do move, don't leave your current company in trouble, this will just damage your reputation.

by chipshopman

Sorry, but I completely disagree. Not documenting the technical environment is a very, very bad mistake. Any technician worth their salt should be working to standard, written procedure. To not do so is putting the whole company at risk and in my opinion is more likely to find the spotlight on you come budget review.

Documentation does not result in the neutering of a technician's worth to the business nor influence things during a budget process, particularly in a one-man-band IT department. In fact the CFO should recognise the work and appreciate it for reducing financial risk.

We are IT professionals, let's act like professionals and do a whole job, not just think about our own skins because that's real short-termism. IMHO you're more likely to find an outsourcer walking in through the company doors if there's no documentation as the CFO takes actions to reduce company risk which he gave you an opportunity to put right.

by stuck_in

In a similar situation with 16 servers, as AS400, remote sites, etc. I have been told my documentation is excellent by the vendors that come in to work on systems. For me it's a selfish reason; I don't have time or the inclination to be a human tape recorder so I provide with enough information to get the job done.

I also have binders that tell you everything from how to add a user to fix the database. I only have a part-time person that I train and they leave which causes a hugh productivity hit. By having manuals I jsut direct them to the binder they need and tell them to read it. Am I worried that I could be replaced? Sometimes I feel 'do me a favor' but the truth is I know how it all inter-relates and which binder to go to for help.

My brain is my job security; my ability to learn!

by emanon km

Alternatively, you want to work yourself out of a job - and into a better one - "Gee Bob, we'd like to give you that promotion, but we can't find a replacement for you, since no one knows what you do..."

If your job isn't documented, how can your performance be rated? How does your boss know what a terrific job you do?

People who refuse to document are usually trying to hide something. Maybe they're not as good as they think they are, or as good as others think they are. Maybe there's even some fraud/theft going on that they're trying not to show.

Docuement all policies, procedures, and what you do. Share that information. Don't assume that you'll be cut because of documentation.

by greg o

You make an assumption that is usually not true in a "one-man IT shop" in a small business, like I am. In most of these scenarios, and mine, the IT person may work straight for the CEO/Pres/whatever, and there IS no promotion potential. I work in a civil engineering firm, and unless I'm a civil engineer, there's no vertical opportunity at all. I could document almost all aspects of my IT job and the CEO could easily outsource it to a local on-demand service that could do it quite a bit cheaper than my salary, provided I gave them a good outline of my unique systems. I enjoy my job, so its not in my best interest to make it easily replaceable with no other avenue to stay in the company. My boss knows if I'm doing my job when all the users are happy, system is always up, and we're making good money using the technology at our disposal. I have good documented and tested disaster recovery plans, and a few sheets of key info if I'm out sick, but that's about as far as I take it. The only thing I'm "hiding" is the knowledge that my boss could save more money by contracting out my job. If it meant you leaving your job/company, would you tell your boss how to eliminate your position?

by robertb

As opposed to some of the other replies on this subject, I agree completely with the idea that you be careful not to document yourself out of a job. Having worked as an IT consultant, I saw multiple instances of exactly this scenario in smaller IT shops. The Boss says "Our network runs smoothly, and everything seems perfectly documented, so what do we need an IT guy for?" at which point the IT person is downsized. I usually got called in a couple months later as things started falling apart (due to lack of regular maintenance, etc. which the old IT person was doing faithfully, but, even when documented, everyone else was just "too busy" to do after the IT person was let go). It was at this point that I made sure the boss understood that the network ran well and smoothly BECAUSE they had an IT person on staff. If you document the nitty-gritty detail of everything you do, the temptation exists (in smaller IT shops) to try running without you.

I would suggest documenting Logical LAN layout, Server names and services on those servers, Admin and service account passwords, all hardware (Servers, Network equipment, Printers, etc.) and Software. Then I suggest drawing up a list of necessary skills required for running and maintaining the network so that if you are ever taken out of the picture (tragic accident, illness, or other reason) the company management knows exactly what type of person they need to hire to take over.

Lastly, and do not underestimate the importance of this, make sure that the managment understands that with the documentation you created and if they hire a person with the qualifications you outlined they will definately be able to take over where you left off.

by understaffed

The complexity of the environment, and the level of conciousness of the users would require somebody to be onsite, so outsourcing wouldn't be feasible. And to bring in somebody else, the pay is so low, and we are so far away from any real city centers, that any perspective employee would be certifiably insane to take what they would offer... (think low $30,000's)

by Member 25186758

Take every opportunity to document your self out of a job.

You don't know what is around the couner, Screw the bus, what about a promotomion or what if you want to leave.

What if your company is taken over by bigger company with more IT staff. If your job is documented you can easiy move into the the new IT department and take on more responablitliy and do different things. If you are seen as been required to remain where you are you limit your growth protenital.

If you want to leave will you be happy to walk away with the currnet state of play. If you do wish to leave you don't want to spend the last few weeks doing the documentation.

The more replaceable you make yourself the more valuable you become to the company in other ways.

As for the "document everything posts", we all know how to get into the AD , we all know how to add a new user. What you need to document your username / password poloicies and anything unquie to your company that goes beyond the basics. IE our faxing soultion required the last 4 digits of the persons fax number in the fax # field.

Shaun

Shaun

by manager_it

Dave,

Make at first two drawins off the network, first the standaard overview (servers, router's,) then make a second drawing off the building whith de logical placing of equipment.

then if you have time, make a drawing off your server room. (wireing over view, equipment over view, Take you time whith it (Rome is not build in one day)

Then make assamble a list of main equipment, you can read it out off your drawings.
Gife every item a number between 1-5 (1 not imported standaard equipment, 5 very very imported cost a lots off work to reconfig it again)

Then starting whith all 5's to document this item. (hardware, configuration, reseller, warrenty, adhoc telefone numbers to get support ect) do this for your all 5's

Then starting whith your 4's ect ect.

I think you must document it because if you walking under a train then nobody now's nothing. If you have documenting it the have something to work with.

I had also read to take a student for a practice, a very good idee because he can help you, and can also do a lot off work for you. But he can also read your writeing and then ask if he can understand it.

You also write you are starting whith VOIP, take the time to document at first all the changes you are going to make to make this VOIP project a succes. Then you have your first document all ready, and keep it up to date.

Put your documentations in a map and place it on your desk, if you want to make change's take also a pencil en make the change in your prited copy. Put a yellow paper on that page and make every week are 14 days the time to make the changes in de orginal digital version.

I come at a lot off places (tempory IT manager) where no document's are and al ways starting on this way. After a few months you have your documentations under controll and up to date. And every time you have some spare time you take a next level off documentation

If you have done this, than you can also easy build a recovery/disaster plan for you company. I think its a must for a company like size you say.

Regards

John

by ryoung2

I have read alot of comments on articles and this is right up there with the best advice I have seen. I am going to start this process now. This is why techrepublic earns it's keep with me day in and day out.

by ha wizard

Don't include what someone else already knows - just include who that person (or company) is and how and when to contact them.
The idea to planning for single points of failure is to eliminate the single point of failure - in this case, yourself.
If you contract your dedicated lines from ??ISP, include the contact information for ??ISP (and contract # or whatever else they need to know to escalate). Additionally, if your servers include 3-yr. service agreements, include that in the documentation and the contact information (plus contract #, etc.) to obtain the service.
Your run-book should only include instructions for what you do exactly and only the necessary information for those tasks (Server Specs., Apps., Network Topology, Domain Names, etc., etc., etc.) grouped by criticality to the business.
You want someone to know where to start and what is most important to run the business on a day to day basis - spending too much time on non-critical systems wastes precious up-time on more critical systems.
Too much unnecessary information will make the run-book overwhelming and non-cohesive.
You should do a proper Business Continuity plan and include a Risk Analysis (this will identify your single points of failure) and a Business Impact Analysis (this will identify your critical services).

by todder

While most people would have the fear of getting down-sized when asked to document their knowledge, in IT, it is a necessity. Providing proper documentation is a skill that will benefit you throughout your career. With Sarbanes Oxley and other auditing requirements nowdays, learning how to do this will only help you. With that said, I would create a very secure folder where whomever has access depends on your environment. In your case I would assume yourself, CFO and President. In that folder I would put the following:

- Complete vendor/contact list
- Admin/Service Account Password list
- Network Diagrams
- Application manuals and any special setup configurations for them
- Data Backup scope/schedule/process
- Disater Recovery steps
- Any policies you created (i.e. network usage, security etc..)
- Run BGINFO on servers and have the output sent to this folder
- Phone system details
- Building alarm codes/setup

That is very good you were able to build this network from the ground up. After completely documenting it, you will visually be able to portray what you have accomplished to your current employer as well as show prospective employers down the road what you did (sans the private information in the documentation.)

I agree with the posts that don't recommend 'how-tos' unless there is some very specific in your environment I hope this helps.

by tkoenings

I agree with Todder. The only change I would make is to document from a disaster recovery prespective. What would an IT person (you or someone else) need to know if x, y, z or all the above were to fail? What would it take to get it back to where it is today? If this needs to be step-by-step then write it as such. Otherwise, just include notations. Remember, to update the documentation as you make server/network changes. Finally, remember someone else will problably be doing the wiring and similar tasks, under a complete recovery so only document the basic information. That's my 2 cents.

by sparker

It seems like your boss wants everything for nothing! I agree with others that you do not need to document simple admin type procedures (adding a user, resetting a password, etc) unless you do them in a strange way.
I would, though, make lists of vendors, customer #'s, passwords, admin passwords, etc...
You should have a wiring map of your network with cable types, etc noted.
I have had to come into situations as a contractor where the "one man band" decided to quit without notice and it was a mess. Documentation is a hard thing to keep up with, but spend a few days walking around with a note pad (or PDA) and write down the things you do. Ask yourself "would someone else know how to do this?"
Lastly, on your next review demand a raise AND an assistant, you obviously deserve BOTH!!!

by sls2655

I have always worked in small shops and deal with this issue frequently. In writing our documentation, we assume that anyone coming into the department would have an IT background. You shouldn't have to explain the steps to add a user to active directory. I used the IT Operations Guide by Prosoft Systems International, LLC to determine the documentation I should have. The guide addresses several additional topics, however the chapter on documentation provided a great outline of what we should have.

by warpindy

My old IT Director aways told us told us to document every change and build on the network. As he put "Just in case you win the Lotto what do we need to know."In fact when I was hired I had to create new network LAN/WAN and DR documents. One thing we did was do a dry run on each document, say a email server build, to see if the documentation need to be changed and if was too complex. Our thing was to have document so see to rad that if the entire IT staff got rich a person from marketing could rebuild or keep the network running.

by sysadmin/babysitter

The fact that the CFO is still the point-of-contact to management says volumes.
As the '1 man' IT department supporting that many people, I doubt that you even have the time to do a detailed network manual.
However, I'm sure that the tip-of-the-iceberg request is ACTUALLY the start of creating and implementing a full disaster recovery plan.
If a disaster recovery plan is existing, it should have EVERYTHING that is needed to restore the system to a position where:
1-Prior Data can be accessed.
2-Workers can perform their duties.

Once a TRUE disaster recovery plan is implemented, you will be well on your way to the documentation level that is needed.
If you are unable to create a disaster recovery plan, ASK FOR HELP!!!!!
If your CFO cannot vouch for EVERY PENNY when financial reporting time comes, (REGARDLESS OF ANY DISASTER) You WILL be replaced!

by processmanager

I would not only document everthing, as other have stated, I would also include a list of possible " substitute" that is firms or consultants that could be called in should you not be available. You're not only creating the documentation you're interviewing the possible outsource supplier as well. Not that you're hiring them but this is part of the disaster plan.

by understaffed

Each year, we look at our positions and the current body of employees and identify those that are in a position to fill in temporarily, could be educated to replace, or could replace at the present time. One of the things we do right...

by bigbigboss

You have to face the fact that it is impossible.

Your organization, like most non-profit social agencies, is understaffed, period. You can't do all that and still document. However, your boss will not be able to understand, and if he/she understands, there is nothing he/she can do anything about.

So, face it. What do you wanna do ? Help those people well in the mean time, and document what you can when you have time, or, let them suffer and spend you next few years documenting ?

I guess it is time to prioritize.

You should at least document the stuff you personally would need to remember, and putting away passwords in safes. That's work ethics.

Your next part would be documentation, augmented with product manuals, targeting people with technical training and experience to take over your function. I will call you a superman if you can accomplish this with your workload.

Documenting till a non-technical person can do your job is impossible. But your boss won't believe it. But you shouldn't worry about it, as when this is required, it won't be your problem anymore.

by mckinnej

Yeah, I know a lot of folks have had bad experiences with these, but think of it as a wiki. In fact, wikis make great knowledge bases. What you're after is a log/journal that you can search and find those "lost episodes" by keywords, topic, etc. This is a contract requirement for my job, but I even do this at home with my server. It has saved me untold hours of hair pulling because the system is nicely documented. It doesn't take long either. I fill in my wiki at the same time I do my timecard. The whole thing takes less than 5 minutes. If you're doing something really detailed or complex, you might want to keep it going all day to catch all those details that will slip by at the end of the day.

This discussion is closed to new replies.