If a cyber security professional would tell you not to implement a critical security update on your servers or endpoints you might look at them quizzically. Yet in a new research paper released by Tanium, 81% respondents (CIO/CISO) said that they have abstained from placing an important security update or patch in their infrastructure, due to concerns about the impact it might have on business operations. In fact, over 52% said they had done so more than once.

So what’s behind this reluctance to place critical patches on their infrastructure ? It seems one of the main reasons may be organizational leaders lacking the understanding of security hygiene and the ability for organizations to keep their production systems running.

Almost 32% of the respondents said that many of their business units operate in silos and they sometimes lack visibility and control over IT operations. Since there was a lack of visibility in those business units, it can have a direct affect on the business, with the majority (80%) of CIOs and CISOs having found out that a critical update or patch they thought had been deployed had not actually updated all devices, leaving the business exposed as a result.

Over 94% of respondents said that they have to make compromises in how well they are able to protect their organizations from disruptions to technology, including cyber threats and outages. The key reasons for making these compromises is the pressure to keep the systems up and running and 31% stated that a focus on implementing new systems takes precedence over protecting existing business architecture.

Some other interesting facts are that approximately 26% said that they were being restricted from doing more on the security side due to legacy IT commitments and 23% stressed that internal politics was the key driver.

I’ve Seen It First Hand

Those who think that placing critical patches on systems immediately is the best solution may want to remember this story that I was a part of over 10 years ago.

I was working for a large organization with thousands of employees in various countries including North America. One day, the Incident Response team received a frantic call that while mostly everyone in the region was able to log on to the network, they were unable to run any applications. The organization was in affect going through a nontraditional “Denial of Service”  since no one could work and there seemed to be no hardware or applications errors.

After about an hour, it was discovered that the latest signature patch for the Anti-Virus program had been pushed out to all machines as was the procedure but instead of blocking viruses, it was not allowing any applications to execute.   The AV company released a new patch within the hour and the problem cleared up but the damage was done, staff members had lost valuable time and possibly income for the organization due to this bad patch.

Lessons Learned

While it seemed like the right process and procedure to push all new AV signature files to all machines whenever they came out from the vendor, a critical process was omitted. That process is testing and validating that the patches not only don’t crash the system but that they work as advertised.

Vendors such as Microsoft have pulled back patches for a few reasons such as incompatibility with certain setups and that security patches didn’t mitigate the threat.

So what should you do ?

Even though you should have a documented change control process, if you are in charge of keeping your systems up and secure here are some easy steps you can follow:

1. Make sure your company has a test environment which is not connected to the production environment
2. When a new security patch comes out, assess the risk to YOUR company. Just because the vendor rates it critical and most times it will be so for your organization, there may be times that you have a configuration that is unaffected by the vulnerability
3. Test the patch on ALL you systems unless they are all configured exactly the same way. Even though this may be tedious, it will save you from production crashes
4. Develop a rollback plan for each patch you install on your important production systems
5. After testing and rollback plans have been signed off on, start your migration of the new patch(es) on the weekend. This will give you time to rollback any production systems before Monday morning