‘This Can’t Be Happening’: One MSP’s Harrowing Ransomware Story

Cybersecurity guru Brian Krebs tells CRN that MSPs and cloud providers are being targeted because the bad actors have learned to count on them for weak, unpatched networks. ‘Who is vetting these providers? Who is asking if they’re doing things right? On a lot of occasions, they’re not.’

A California-based MSP was on a vacation, driving up the Pacific Northwest coast with his girlfriend when a customer called him with a problem that would end up consuming all his time off, and plunge him into the murky underworld of ransomware negotiation.

“He described what was going on and I said, ‘That sounds like you got hit by ransomware,’ ” said the MSP who asked to remain anonymous. “Then I had another customer call me … I was with my gal. I said ‘This can’t be happening.’ I’m on the phone driving to Oregon calling all of my customers and telling them to turn off their computers.”

The MSP is one of dozens nationwide that’s part of an expanding roster of solution providers whose networks and customers have fallen prey this year to ransomware. Whether via a phishing campaign against random sites, or using powerful ITSM tools in a targeted strike, hackers have picked up the pace of ransomware attacks in 2019, which has seen a five-fold increase in hits to government systems alone, according to the National Association of State Chief Information Officers.

[RELATED: Ransomware Expert Fabian Wosar Analyzes Five Recommended Defenses For MSPs -- And Adds Two More]

Renowned London-based “ransomware killer” Fabian Wosar -- who has been featured on the BBC and in The Guardian newspaper, among other media outlets, for his work in undermining the bad actors carrying out ransomware attacks -- told CRN that these attacks were born from lax security and are not going away anytime soon.

“Hacking an MSP and then encrypting all their clients is hugely profitable. There is such a huge return on investment … its low hanging fruit,” he said. “MSPs never had to deal with it, so in a way they got away with a lot of shady practices, and bad cyber-hygiene. Either they were lazy or they didn’t know any better, you had a lot of them who are vulnerable to this type of attack.”

Cybersecurity guru Brian Krebs, who runs the krebsonsecurity.com blog, told CRN that MSPs and cloud providers are being targeted because the bad actors have learned to count on them for weak, unpatched networks.

“Who is vetting these providers? Who is asking if they’re doing things right?” he said. “On a lot of occasions, they’re not. Or they haven’t updated their security procedures and requirements for years.”

Additionally, Krebs said many MSPs focus on backup and disaster recovery to a fault, rather than monitoring their systems for unusual behavior, which could turn up an intruder before the attack happens.

“It’s not like the bad guys get in and flip a switch and they got everything ransomed. And it’s not typical that a ransom is just going to spread by itself through the network,” he said. “In most cases the bad guys get in -- might be an opportunistic compromise, might be a mass phishing email, or it might be targeted -- the point is it can be weeks or months before the bad guys launch the ransomware. So, there’s the opportunity for all potential victims to avoid that occurrence of having to suffer a ransomware attack, if they’re set up to assume that bad guys are going to get in and they’re set up to look for compromises inside their own environment.”

Recently, 22 towns in Texas were hit with a ransomware virus that appears to have been spread using MSP tools. In the wake of those attacks, state officials put out a list of best practices for MSPs to avoid being the next victim. Among the steps were to only allow authentication to remote access software from inside the provider's network, using endpoint monitoring to keep an eye on Powershell, and enabling multi-factor authentication.

“I’m a big proponent of 2FA now,” the MSP who was hit said with a chuckle.

Prior to the attack, he said he had outsourced the IT work on the victimized customers to a white-box, third party IT provider who he believes left login credentials on a compromised machine.

Three of his customers – a moving company, a doctor’s office, and a corporate recruiter -- and around 50 endpoints had been locked up with ransomware that had infected their systems. What he learned, he said, is that you have to develop next steps immediately, and part of that, for him, involved calling an intermediary to negotiate with the criminals.

“You send them a file and they tell you what variant of ransomware you have,” he said. “Then in the end, they supposedly work with the hackers and the FBI. In a way, they only make profits off of people who get ransomware, so that makes their business questionable to some of my customers. But they were very helpful.”

The company he called handled all of the upfront work and took over communications with the hackers.

“They said ‘don’t talk to them directly. Let us do the whole thing. There’s a dance there and they get the price down,” he said. “They wanted $3,500 bucks to get me back up and running and then $2,500 and then a grand. However, they’re in the same position as the ransomware people. They only make money off a company that has the money to pay it.”

The moments following an attack are critical, he said. Many times, files are unknowingly locked behind two or three layers of ransomware.

“In the first 24 hours of a ransomware attack, you should have multiple remedies going at the same time, depending on how important the data is,” he said. “You want to start your recovery process from your backups. You may want to start something offline and even go after decrypting the ransomware. That way when you are going to your C-Level people and saying ‘Hey, we have to make a decision here,’ you have all the options already in motion. You have quotes for them so they can make a decision.”

Though, in this case, the MSP decided not to pay the ransom, the intermediary provided him with all the information about the virus and a little about the attacker.

“They said ‘We know this guy.’ They gave me a little back story,” he said. “They looked at the files and said it’s been encrypted three times. Two are with this ransomware, one is with this. So when they talk to the hacker, they make sure they get all three decryption keys.”

For Wosar and New Zealand-based Emsisoft, they don’t talk to the hackers, but rather try to hack the ransomware itself.

“Essentially what we do, when we see a new ransomware family out there, what we do is we look for problems in their cryptography and other things that allow us to come up with a de-encryption tool that allows the victims of that ransomware family to decrypt their files completely for free and we offer those tools for free to everyone as well,” he said.

Wosar estimates that he and his company have helped more than a million end users recover from ransomware attacks, many without paying.

While the MSP who CRN spoke with didn’t pay a ransom, it still cost him thousands of dollars to recover. As part of the clean-up, he spent weeks rebuilding his customer’s systems – new servers and PCs for three customers -- at no cost to them. One client quit.

He said the rebuilding process is not only costly, labor intensive, and a massive loss of time, but he said it “tested all of our documentation, backups and where we store the CDs.”

“Imagine having to reinstall all the software on all the computers,” he said. “Well, ‘where do you get the download from?’ And ‘where is the installation key?’ and ‘you have to activate it online, but it says it’s already activated.’ So you’re dealing with the granular nature of getting all these PCs and servers rebuilt.”

He said because of the blast message to the rest of his clients asking them to shut everything down, he now feels he’s lost their confidence.

“No one trusts me anymore,” he said. “You go from IT adviser to being questioned on everything.”