In the past two weeks I have witnessed a couple of contrasting situations involving configuration changes in IT. In one environment the client had a strict adherence to the practice of using Change Management in all their IT operations. In the other operation, the client had not implemented Change Management. When it came time for one of those inevitable problems that occasionally hit the Information Infrastructure, the outcomes for the two firms was very different.
What is change management?
Here is the definition from Wikipedia based upon the industry standard Information Technology Infrastructure Library (ITIL).
Change management is an IT service management discipline. The objective of change management in this context is to ensure that standardized methods and procedures are used for efficient and prompt handling of all changes to control IT infrastructure, in order to minimize the number and impact of any related incidents upon service. Changes in the IT infrastructure may arise reactively in response to problems or externally imposed requirements, e.g. legislative changes, or proactively from seeking improved efficiency and effectiveness or to enable or reflect business initiatives, or from programs, projects or service improvement initiatives. Change Management can ensure standardized methods, processes and procedures which are used for all changes, facilitate efficient and prompt handling of all changes, and maintain the proper balance between the need for change and the potential detrimental impact of changes.
A change is an event that is:
- approved by management
- implemented with a minimized and accepted risk to existing IT infrastructure
- results in a new status of one or more configuration items (CIs)
- provides increased value to the business (Increased Revenue, Avoided Cost, or Improved Service) from the use of the new or enhanced IT systems.
In case #1, the client㇐a company based in Orange County, California㇐was switching to an MPLS circuit from a layer-two wireless bridge to communicate between two offices on their campus. Over time, the bridge had proven to be unreliable and the client’s IT staff decided to go to an MPLS circuit to resolve the issue. Alvaka and the client’s management team were not consulted or brought in to plan for the change. Instead, the IT team and the carrier decided to cut-over to the MPLS circuit. All sorts of networking issues ensued that brought down both the computer network and the client’s phone system. That is when they called Alvaka in a panic…and all sorts of emergency gyrations followed in order to get them back online as soon as possible. The temporary solution was to revert back to the old setup until everyone could get their arms around what the best long-term solution was to this problem. At the end of the day, all the players that should have been involved were now involved. The client incurred a lot of expensive downtime and IT lost a lot of credibility. Now, a new approach will be planned with all the key stakeholders involved, so a better and more successful solution can be implemented without all the disruption to the business. For this company, this event became a catalyst to implement and embrace IT Change Management.
In case #2, the client㇐a manufacturer in Riverside, California㇐had been a disciplined follower and advocate of Change Management process since the beginning of 2014. This client was successful with every Change Management endeavor, meaning that their changes caused little to no disruption to the business and everyone knew the changes were coming. Everyone involved and/or impacted by a change was involved in the process. 10 days ago, there was a plan to move a new CRM deployment from the development environment to production. A Change Management Plan was created and sent out to the client’s management team and all the stakeholders for approval, including a new CRM vendor. The approval was secured and a migration was planned for a Sunday when the impact on operations would be minimal. As it turns out, the CRM vendor was fairly casual in their recommendations with regards to the requirements for a successful migration. They also did not exercise discipline in reviewing the Change Management Plan. What ensued was a failed migration to this important new CRM system. I received an email from the COO, who was a bit miffed over the failed migration. He had a right to be upset, as they spent a lot of time and money planning this change, and they followed the directives of the CRM consultant. I soothed the nerves of the COO a bit by acknowledging the frustrations and disappointment of the failed migration, but I told him the Change Management Plan fulfilled its duty. Sure, the migration was not successful, but I always insist on a strong roll-back plan in case things don’t go properly. A roll-back plan allows you to get back to how things were before the changes, in case things don’t go right. In this case, the roll-back plan worked. I told the COO, “Yes, you are disappointed, but you did not incur any downtime and you did not lose any data.” This was the first time they had to run a Change Management Plan through its full course and it worked as it was supposed to work. While the client did not get the outcome they hoped for originally, the CM Plan worked. Now, the CRM consultant can be counseled about doing some more serious migration planning, and everyone can recalibrate as to what needs to be done to successfully execute the migration next time. I am sure it will be migrated with full success.
Change Management is your friend. It brings a disciplined and controlled process to making changes, so that problems in IT are limited and the opportunity for success is maximized. Read more from our “Why Are Patch Management and Change Management Important?” blog.
Smoke testing is a term used to describe the testing process for servers after patches are applied. 
Estimating an average time for patching servers and PCs can be a bit tricky. It can vary from one month to the next, depending upon the number and complexity of the patches released by your software vendors. You must consider all versions of operating systems and have a complete inventory of all your application software to do this job correctly. Our experience has shown that manual patching of systems takes on average of about 1.5 hours each.
This is a basic cost calculator for you to compute your typical monthly cost for patching your servers, PCs, laptops, tablets and associated application software. It also forms the basis for you to begin calculating your Return on Investment for software patching, or for comparison with alternatives to the manual process of patching operating systems and application software—such as Patch Management as a Service, also known as Vulnerability Management as a Service.