Cybersecurity professionals and IT vendors spent the weekend scrambling to shore up systems before hackers exploited a zero-day vulnerability in the popular Apache Log4j open source logging tool.

Many were too late. Shortly after Apache disclosed the remote code execution vulnerability (CVE-2021-44228) on Dec. 9 and released a patch, threat researchers and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) sounded the alarm that attackers were already exploiting the security flaw, which received a perfect 10 out of 10 severity score.

Microsoft warned that attackers were mass scanning the internet for vulnerable systems and using Log4j to install coin miners, Cobalt Strike to enable credential theft and lateral movement, and steal data.

According to Palo Alto Networks’ Unit 42, the vulnerability is “incredibly easy” to exploit.

“By submitting a specially crafted request to a vulnerable system, depending on how the system is configured, an attacker is able to instruct that system to download and subsequently execute a malicious payload,” the threat hunting team wrote in a blog post about Log4j. “Due to the discovery of this exploit being so recent, there are still many servers, both on-premises and within cloud environments, that have yet to be patched.”

Over the weekend, CISA Director Jen Easterly said Log4j “presents an urgent challenge to network defenders given its broad use,” and urged organizations to “urgently patch or remediate this vulnerability.”

Who and What Uses Log4j?

However, this broad use is what makes the vulnerability so dangerous — and, according to many security researchers, ensures that it will likely be months before the full scope of the damage comes into view.

Log4j is a logging library that’s used in a huge number of Java-based applications and software. The vulnerability affects Log4j versions 2.0-beta9 to 2.14.1.

While the security flaw was initially detected in Microsoft’s popular Minecraft game, Log4j is also used in many enterprise apps and cloud platforms from vendors including Apple iCloud, Amazon, Cloudflare, Microsoft, Cisco, VMware, Qualys, Fortinet, Twitter, and RedHat.

Plus, it’s widely used in open source software including Apache Struts, Apache Solr, Apache Flink, ElasticSearch, Logstash, Kafka, and others.

So in addition to determining if their developers used vulnerable code from this library to write internal apps, it’s very likely that companies use third-party platforms running compromised code in their IT environments.

In other words, the attack surface is huge.

First Come Miners and Botnets. Next Up: Targeted Attacks

“Every organization out there is going to have some exposure to this most likely,” Matt Olney, director of Talos threat intelligence said during a Cisco livestream on Monday. “Very few will escape.”

Forrester says the vulnerability affects as many as 3-billion-plus devices currently running Java.

“This vulnerability is so dangerous because of its massive scale,” Forrester Analyst Allie Mellen said. “Attackers are starting to use this vulnerability to target victims with cryptominers and botnet attacks, but expect more devastating attacks (like ransomware) leveraging this vulnerability in the future. This vulnerability will be used for months if not years to attack enterprises, which is why security teams must strike while the iron is hot.”

Cisco Talos, which said it observed attackers exploiting the flaw as early as Dec. 2, echoed Mellen and said coin miners and botnets were the early adopters. However, the threat researchers warned that more targeted attacks were likely to follow.

“If the pattern holds, and we expect it to, many actors with different objectives ranging from financial to espionage will rapidly adopt this exploit in the coming days to secure access either for immediate use, for resale, or for long-term footholds,” according to a Talos blog.

What Should Organizations Do?

In addition to upgrading to Log4j version 2.15.0 and applying the mitigations recommended by their software vendors, CISA also recommended organizations take three other immediate steps to prevent and detect attacks.

First, enumerate any external facing devices with Lof4j installed. Second, security operations team should take action on every alert on any of these devices, CISA says. And finally, update web application firewall (WAF) rules to automatically block any exploit attempts so that the security team can concentrate on fewer alerts.

Some WAF providers including Cloudflare also pushed updated WAF rules that automatically block exploit attempts.

And VMware’s threat research team recommends applying outbound microsegmentation rules to prohibit new connections from being established out from your workloads.

In fact, much of the detection and response activities rely on basic security hygiene, Cisco Talos’ Olney said on the livestream.

“The fundamentals of security practices are still in play,” he said. “There’s nothing technological about this development that changes the game in that way. So all of your internal process reviews, all of your lateral movement, monitoring all of your threat hunting, all of your exfiltration protections, all of your post-exploitation mitigation, all of your segmentation— all that is still valuable and in play.”

Several vendors and threat researchers have also developed Log4j-specific detection tools and posted resource blogs to help organizations remediate any exploit.

For example, Huntress Labs created a tool to test applications for the vulnerability, and Cybereason developed a vaccine to protect against exploits. Additionally, Zscaler launched a free tool for businesses to run an internet attack surface analysis to check for any external attack surface using Apache.