Why Is Software Patching So Important?…Failure to Patch Can Be Very Costly
Failure to patch vulnerabilities in computer code can lead to losses of information that can cost more than your company can afford to pay.
Here are some real-life examples. On October 20, 2023, Twilio had a cyber security breach incident. It could have been prevented by patching the vulnerability before the breach rather than after. The attack exploited a vulnerability in the Okta integration used by Twilio, potentially granting unauthorized access to customer data like phone numbers and SMS logs. While the full extent of the data accessed remains unclear, Twilio acted swiftly to mitigate the damage. They immediately disabled the vulnerable Okta integration and advised customers to update to version 2023.10.26 or later. This update addresses the exploited vulnerability, significantly reducing the risk of similar attacks. Remember, prompt patching is crucial to prevent unauthorized access and potential data breaches. Regularly reviewing and updating third-party integrations is essential for maintaining strong security. A layered security approach combining patching, access controls, and user awareness helps minimize overall risk. By taking these steps, organizations can significantly reduce the risk of falling victim to similar attacks that exploit vulnerabilities in third-party integration.
Another recent example of a software patching failure was the Log4Shell vulnerability in December 2021. This vulnerability came to light within the Log4j logging library, widely utilized by various software applications. Exploiting this vulnerability permits attackers to execute unauthorized code on susceptible systems, potentially granting them full control over the compromised systems. Despite the rapid release of patches for the Log4Shell vulnerability, numerous organizations lagged in applying these fixes promptly. Consequently, this led to a surge of attacks that took advantage of the vulnerability, including the Colonial Pipeline ransomware incident, which disrupted fuel supplies in the eastern US in May 2021. Another incident that resulted from software patching failure is the Hafnium attack on government agencies and global businesses. In March 2021, ProxyLogon vulnerabilities for the Microsoft Exchange Server granted attackers access to a multitude of servers. Though Microsoft promptly released patches, many organizations failed to patch their servers in time. These are just two of many incidents of recent software patching failures.
There are many other benefits to applying software patches, including adding features, and fixing bugs that make the software run slow or not work right. All software needs to be patched. Whether the software sits on a disk and runs on a server, resides on a chip within a firewall, or is an app that is in your tablet devices, it all needs to periodically be updated and patched in order to be secure.
The following list of 18 software patching best practices is what we follow at Alvaka Networks when delivering on our Patchworx Patch Management service. It is important to note that all these steps are important; however, they are not always all utilized, or they can be utilized in different ways depending upon the needs of the client. Like us, you will need to decide what your patch management plan needs to look like to best suit your needs.
18 recommended best practices for patching your software:
1. First, identify all the software you are using. Today’s IT systems present a challenge because most systems run dozens of different software titles. You can’t know what you need to patch until you know what you have. You have operating systems, server applications and desktop applications.
2. You need to patch servers, PCs and mobile devices.
3. Remember, you need to have a strategy to patch many of your hardware based appliances, like firewalls, routers, SANs and more.
4. Set a regularly scheduled routine every month to patch your systems. You can do it most efficiently all in one big event over a weekend, where all systems are patched. Or, you can elect to do 20% of them at a time over the course of the month, to mitigate impacts from unexpected patching problems. There are many other approaches you can take. You need to decide what is best for your business.
5. When dealing with multiple servers, you need to identify if you have dependencies that require a certain server reboot in order for everything to work right on restart. For example, it is best practice to bring down a multi-tier system by starting with the presentation tier (web server), then the application tier, and then lastly, the database tier. Your systems should be brought up in reverse order.
6. Read the release notes or ReadMe files to learn more about the implications of deploying a set of patches. You should also review software user forums to see if anyone else is reporting problems with the new patches.
7. It is good to apply patches in a timely manner, but unless there is an imminent threat, don’t rush to deploy the patches until there is an opportunity to see what effect it is having elsewhere in similar software user communities. A good rule of thumb is to apply patches 30 days from their release.
8. Before applying patches to your production system, you should test the patches out on a test environment. This can be difficult and expensive for most companies, since it requires buying a lot of extra hardware and software to build the test environment. If you don’t have a budget to do this, there are alternatives. There are companies that provide patching services. Testing those patches before deployment to production systems is one of the tasks they perform.
9. During the testing process in the test environment, determine if the computers require a reboot or if they do so automatically. If so, then you need to plan for a maintenance window in which to apply the patches to the production system, so you don’t have unexpected system reboots that hurt your business operations or do damage to databases, etc. You can expect 90+ percent of your patch deployments to require reboots.
10. After you have applied patches, utilize a smoke testing procedure to make sure all applications and services are back online and running properly when servers and PCs restart.
11. Change Management is important, but often overlooked. You need to involve other stakeholders in the organization before making the changes. Often they will let you know of system or organizational demands that will have an effect on your patch deployment task. Read our blog, “What is Change Management and Why is it Important?”
12. Notify your end-user community of your planned time frame for patch deployment, so they know what to expect. When patching workstations, remind the users just prior to patching to save all documents, close all applications, and logout of their workstation, but DO NOT SHUT THE PC DOWN. Let them know what they should do if they encounter a problem after the patch deployment.
13. Have a good roll-back plan. A roll-back plan allows you to quickly reverse the patches and go back to the pre-patched system if there is a significant problem with the deployment. Good patching tools and procedures will allow for a roll-back of patches.
14. Have a good backup of all your systems and, if possible, take an image snapshot of your servers right before your patch deployment.
15. Are there any auto-scheduled maintenance jobs running to do maintenance, such as for a SQL database? If yes, be sure to put them on hold, as they can really mess things up if left running.
16. Use a service or automated tools whenever possible. Don’t use tools like Auto-Update, as you cannot control when patches are applied, you cannot test applications before patches are applied, there is no smoke testing procedure post-patching to determine everything is running fine, and there is no patch deployment reporting that is required to show yourself, management, auditors and regulators that you are running a securely patched operation.
17. Review the patching report after deployment and look for patches that failed to deploy. Investigate why they failed to deploy, develop your remediation plan, and then redeploy.
18. Make sure you accommodate your exceptions. Sometimes certain servers or applications cannot be upgraded or patched in order to maintain compatibility with a critical application that is in use. When this happens, you need to make sure you have an alternative strategy for securing that system from the vulnerability left exposed by the inability to patch the software.
The Center for Strategic and International Studies (CSIS) study made a strong case for patch management. The study, conducted over a three-month period, found that simply applying the most recent patches to six software packages on Windows machines could prevent 99.8 percent of malware infections!
Click HERE to see the HHS Office of Civil Rights partial list of companies paying heavy fines; some are in the millions of dollars, for breaches of their IT systems. These companies were already victimized once by the breach. I bet they felt victimized a second time when the fines were levied. Don’t let that happen to you. The moral of this blog is that you have to patch eventually one way or another, so make sure you do it in a timely and professional manner to avoid terrible impact on your business. It will bring you much peace of mind.
Smoke testing is a term used to describe the testing process for servers after patches are applied. 
Estimating an average time for patching servers and PCs can be a bit tricky. It can vary from one month to the next, depending upon the number and complexity of the patches released by your software vendors. You must consider all versions of operating systems and have a complete inventory of all your application software to do this job correctly. Our experience has shown that manual patching of systems takes on average of about 1.5 hours each.
This is a basic cost calculator for you to compute your typical monthly cost for patching your servers, PCs, laptops, tablets and associated application software. It also forms the basis for you to begin calculating your Return on Investment for software patching, or for comparison with alternatives to the manual process of patching operating systems and application software—such as Patch Management as a Service, also known as Vulnerability Management as a Service.