3CX Attack Shows The Dangers Of ‘Alert Fatigue’ For Cybersecurity

The widely felt supply chain compromise of VoIP vendor 3CX was not caught as quickly as it might have been, as both the vendor and users initially assumed the alerts were false positives, according to cybersecurity experts.

The far-reaching supply chain attack on communications app maker 3CX might have been caught sooner if security professionals weren’t so desensitized to the stream of alerts and frequent false positives coming from their threat detection tools, cybersecurity experts told CRN.

This phenomenon, known as “alert fatigue,” is widely recognized but difficult to fix. The 3CX compromise shows the real-world consequences of the issue for cybersecurity, experts said.

[Related: 3CX Supply Chain Attack: Big Questions Remain]

Alert fatigue played a “crucial role” in the delayed detection of the 3CX attack, Exabeam CISO Tyler Farrar told CRN.

While users did report receiving warnings from SentinelOne about 3CX as early as March 22, both the users and 3CX support team seemingly assumed that the detection was a false positive—possibly due to experiencing “numerous false alarms in the past,” Farrar said.

CrowdStrike threat hunters were the first to determine that the detection of malicious activity coming from the 3CX app was not a false positive, and the company publicly disclosed details about the attack in a post March 29.

Most likely, the initial lack of urgency around the issue was influenced by desensitization to alerts, Farrar said.

The bottom line is that “there’s way too many alerts. We can’t catch them all,” said Christina Richmond, chief strategy and growth officer at security services and solution provider Inspira Enterprise, and a former program vice president at research firm IDC.

In the future, it’s likely that automation and AI will do more to help with the issue, Richmond said.

“But for now, there’s a lot of alert fatigue that leads us to then not pay attention to some of the most critical alerts,” she said.

CRN has requested comment from 3CX and an interview with Nick Galea, founder and CEO of 3CX.

Galea told The Register that after learning about the SentinelOne alert, the company checked its app with antivirus aggregation website VirusTotal. After getting an “all clear” from VirusTotal, “we considered the SentinelOne alert a false positive,” Galea told the site.

The 3CX CEO told CyberScoop it’s probable that hundreds of thousands of customers did actually download the malicious version of the vendor’s VoIP phone system software.

A Harsh Reality

Many 3CX users had seen their endpoint protection software incorrectly flag legitimate software as malicious in the past, said Greg Notch, CISO at cybersecurity vendor Expel.

Since 3CX’s software was expected in their environment, they assumed it was the endpoint security software that was incorrect, rather than suspecting the 3CX software had been the victim of a supply chain attack, Notch said.

For overtaxed security operations teams, these biases “are just the reality” of their roles today, he said.

According to 3CX, its customer base totals more than 600,000 organizations. Major customers listed by 3CX include American Express, McDonald’s, Coca-Cola, NHS, Toyota, BMW and Honda.

CrowdStrike has attributed the attack to a North Korea-affiliated group that it calls Labyrinth Chollima. 3CX has hired Mandiant, a foremost incident response provider that’s owned by Google Cloud, to perform an investigation into the attack.

“We’ll continue working closely with our Mandiant advisers to investigate how this incident occurred and put in place measures to prevent any recurrence,” Galea wrote in a post Saturday.

‘More Wrong Than Right’

Ultimately, when it comes to dealing with alerts, defenders need to be able to quickly decide if a certain alert deserves more attention, said Johannes Ullrich, dean of research for the SANS Technology Institute.

Experience can sometimes help, he said. However, in many cases, experience actually hurts because it tells defenders that “these tools are more often wrong than right,” Ullrich told CRN.

Triggering a full incident response based on an alert like this is “risky” as well since it often causes major business disruption to follow up on an alert, he said.

Ultimately, technical personnel reviewing alerts are too often missing the context necessary to make a quick decision distinguishing valid alerts from false positives, Ullrich said.

Unfortunately, security software often creates “many more false positives than it triggers true alerts,” he said.