By Ruby Nahal, Senior Engineer
As Network Admins, we are all guilty of making a change without taking the time to document that has come back and hurt us. Even more frustrating, is when you get stuck working on an issue due to someone else’s lack of documentation. I, myself, have worked on several issues where undoing undocumented changes took several hours to revert.
Hence, Change Control occupies a very special place in my heart. It saves you a ton of heartache, forces you to pre-think the changes you are making, the consequences, possible roll back plans (very important) and the entities and processes that may be impacted by the change.
So What is Change Control?
Change Control is a systematic approach to managing all changes made to system(s). The primary purpose is to ensure that no unnecessary changes are made, that all changes are documented, services that will be disrupted are documented, downtime is scheduled and a roll-back plan is in place.
Following are some of the things I have learned about implementing a proper Change Control process in an IT environment:
1. Include all steps to be taken and all services to be impacted. – Unexpected things can happen and that is to be expected, but make your best effort here to pre-think the possibilities. This will also help in communicating and scheduling a proper downtime with the client.
2. Have a plan B – Having a well-formed Plan B may not always be possible so sometimes your Plan B may just mean you need to roll back or restore from a backup. But considering this before the actual Change Control process begins forces you to have a backup to go back to. For example, this may be as simple as making a backup of a firewall configuration before you modify it. Rolling back to a backup configuration will be way easier than manually undoing changes made.
3. Testing plan – Having a testing plan ready allows you to confirm that changes made are in fact working and you can set a monitoring time before considering the process complete.
4. Put a peer review process in place – Have a peer or a senior review and approve big changes – a second pair of eyes on the changes helps catch anything you may have missed.
5. Adapt to changes along the way – Document anything that changes in the original process.
6. Formalize the plan – Take time to formalize the process and disseminate throughout in your organization so it is taken seriously and given the place it deserves in daily operations. Having a casual change control can do more harm than good because expectations may be raised or lowered because of it and that may result in inconsistent processes.
7. Don’t go crazy – Let us not create a Change Control for every small change. My rule of thumb is, have a Change Control in place for anything big, critical or anything that affects more than 10 people. Having change control for resetting passwords or updating a workstation is a bit of an overkill. Change control for firewall changes, server upgrades, ISP cutovers is just right.
Why Change Control?
Chaos is inevitable and Change Control can help you get a handle on the situation. We can save ourselves and others hours of wasted time, frustration and downtime with a proper change control management process.