Written by Dan Tynan (Writer, Editor, and Consultant). Originally posted on CIO.com on June 25, 2018. Alvaka Networks CEO, Oli Thordarson, and CTO, Unnar Gardarsson, contribute to this discussion on what projects IT professionals dread the most.
The projects that most IT professionals dread are the thankless yet essential tasks that garner little acknowledgment or respect.
Patch management isn’t glamorous, but fail to keep up and your organization could suffer a massive security breach. Cloud migration projects aren’t often sexy, but running email servers on premises in 2018 is a bit like using rotary phones. Enterprise resource planning (ERP) implementations and upgrades have been the poster child for IT project failure for more than a decade, but most large organizations can’t survive without some kind of ERP system.
Technology is almost never the root cause of hellish IT projects. More often tech initiatives fall into disarray thanks to unrealistic expectations, a failure to adequately scope, cultural clashes, procrastination, or a lack of adequate support from on high.
The solutions to hellish IT projects are usually straightforward, but the devil is in the details.
1. Enormous patches at the last minute
Patching is one of those essential chores nearly every IT pro hates to do. But put it off too long, or fail to ensure that all your machines are up to date, and your organization could end up like Equifax.
“We are constantly being thrown big projects where patching has not taken place for months or years,” says Oli Thordarson, CEO of IT services provider Alvaka Networks. “Sometimes it can be hundreds of production servers, along with dev and test servers. Knowing that if something goes wrong we can blow up a big internet presence for a brand name is stressful. That’s why internal staff sometimes procrastinates on patching, until all of a sudden management looks at the situation and realizes it is overwhelming.”
A few years ago Alvaka was doing a big Windows patch job for a city police department, says Thordarson. Everything went fine for the rank and file’s desktops. But the PCs for the chief of police, captains, and lieutenants were all down for the count — and they were not happy about it.
“Top brass at police departments are not shy people,” he says. “They’re calling us and they’re PO’d because they can’t log in and do what they needed to do.”
Turns out they were using slightly different machines with third-party apps that were causing a conflict. Thordarson’s team was able to roll back the patches, isolate the problem, and fix them in about 30 minutes. But it was a tense and stressful half hour.
The worst patch jobs are the ones where systems have been neglected for years, adds Alvaka CTO Unnar Gardarsson (who says he is one of the rare IT pros who actually enjoys patching).
“The ‘oh sh*t moment’ is when I’ve patched something and the servers take abnormally long to come back, or don’t come back at all,” he says. “In some cases I can go into the virtual machine recovery console and look at the server. In other environments all I can do is patch and pray.”
Lessons learned: To avoid worst case scenarios, take extra care to make sure you have a thoroughly tested backup plan that accounts for every single system, says Gardarsson.
“For every patch job I do I have a project plan that takes into account everything from normal backups and additional snapshots to notifying people and stopping services,” he says. “Some systems are really finicky and there’s a very specific process to taking them offline. Being prepared and thinking through every possible scenario is key.”
Smoke testing is a term used to describe the testing process for servers after patches are applied. 
Estimating an average time for patching servers and PCs can be a bit tricky. It can vary from one month to the next, depending upon the number and complexity of the patches released by your software vendors. You must consider all versions of operating systems and have a complete inventory of all your application software to do this job correctly. Our experience has shown that manual patching of systems takes on average of about 1.5 hours each.
This is a basic cost calculator for you to compute your typical monthly cost for patching your servers, PCs, laptops, tablets and associated application software. It also forms the basis for you to begin calculating your Return on Investment for software patching, or for comparison with alternatives to the manual process of patching operating systems and application software—such as Patch Management as a Service, also known as Vulnerability Management as a Service.