Downtime versus Security

Software vulnerabilities are a fact of life for both developers and IT admins. Barely a week goes by without some kind of security update for a high profile application or operating system. Patching is a constant cycle that is frequently automated at an enterprise level. Microsoft release security fixes for Windows and Office monthly, as do Chrome and Firefox. Rolling out those patches quickly and speedily can be challenging when security concerns have to be balanced against compatibility testing and the risk of downtime. All too often, those patching schedules are dictated by the severity of the threat and the risk of leaving a system unpatched for an extended period of time.

Security experts always advise users to apply software updates as soon as possible after release. IT standards such as ISO 27001 require this before organisations can be certified. Yet, this doesn't always happen, particularly in smaller shops which don't the resources to manage complex applications properly. Management in these same companies rarely appreciates the risks associated with delaying security updates longer than strictly necessary.

Day Zero

In early 2021, a high profile zero-day vulnerability affecting Microsoft Exchange Server put those risks into the spotlight. Zero-day vulnerabilities are always dangerous. The term refers to security threats that are actively being exploited by hackers at the time the patch is released. That only applies to a small proportion of the many updates released by developers, but it's always big news whenever one is discovered in a widely used application. It then becomes a race against time to apply the relevant update before you're hacked.

Not all zero-days are actually critical for every software end user. Some zero-days are difficult to exploit, some only have a very limited impact, while others only apply to certain configurations. That was emphatically not the case for the Microsoft Exchange vulnerability that the Hafnium hacking collective exploited over the past few months. This threat was unusually severe because the flaw in Exchange was easy to exploit and could lead to the entire server being compromised. It has already been used to install ransomware and steal confidential data.

The worst part of the Hafnium threat, though, was that multiple hacking groups were widely exploiting it prior to the eventual patch being released. It has been suggested that a Microsoft partner leaked a Microsoft developed proof of concept exploit kit intended prior to public disclosure of the vulnerability. This implies a level of coordination between the security community and hackers that is disturbing, although one possible in a nation state led hacking campaign. As a result, Hafnium became headline news due to the number of companies being breached compared to past Exchange vulnerabilities.

Known Consequences

Given the circumstances, it was entirely foreseeable that any delay in patching the Hafnium vulnerability would lead to Exchange installs being hacked. Yet, thousands of IT admins did precisely that. Many servers were left unpatched for days with only one result. It is estimated that 250,000 servers fell victim to the attack. All to avoid a few minutes of downtime for the corporate email server.

It is far easier to prevent a hack by installing a security update than it is to clean up a breach once discovered. Hafnium installed backdoors on the infected servers, which other groups are actively exploiting to install ransomware. Removing these backdoors can be complicated, often involving input from specialist security experts. For this reason, the best practice is to wipe the infected server and start again. That's not always possible with something as mission critical as an email server. Due to the scale of this particular problem, an automated clean-up tool for Hafnium has been released through the Microsoft Defender antivirus tool built in to Windows. Too late for the compromised Exchange users.

Known Alternatives

Perhaps the easiest way to avoid being breached by Hafnium is to follow another common security recommendation. Don't run your own email server unless you're large enough to have a dedicated email security team to manage it. Microsoft Exchange is a complicated piece of software that many IT managers don't really understand. That was true when I managed Exchange 2003 over a decade ago, but the problem has become worse since then. Email is a major threat vector for phishing, viruses and ransomware, requiring a suite of specialist security tools and hands-on monitoring.

As an internet facing service that has to be available 24/7, email servers are constantly being targeted by hackers. As such, ensuring they're configured correctly is essential. Yet too many SMB Exchange admins don't even know how to configure the application properly, or don't have the resources to build in redundancy and failover servers to avoid downtime. There is a reason, why many organisations in high security industries such as Government or Finance now adopt a cloud-first approach. They know that the major cloud players are far more secure than their self-hosted servers will ever be. This month, their peers learned the same lesson.