When a business that is well known within their industry, and with high-profile clientele, was hit with a potentially devastating ransomware, they enlisted the help of Krome Technologies to help them recover.

When this business was hit with a ransomware attack, during a bank holiday weekend, they enlisted the help of Krome to mitigate the damage, save their data and restore their operations as swiftly as possible, avoiding having to pay the half a million-pound ransom demand made by the attackers.

Initial Signs of Compromise

Over the Easter Bank Holiday weekend, when most employees were away from the office, some users had reported that they were unable to send and receive emails, at this point in time it was assumed by the internal IT team that they had an issue with their mail servers, however upon investigation, it became apparent that this had been compromised and had been encrypted. The IT team had also noticed that their endpoints were disappearing in their antivirus management console. At this point they knew that they were under attack and needed urgent assistance, Krome was contacted in the morning on Bank Holiday Monday to assist with recovery.

Immediate Response

Krome’s team of security professionals immediately evaluated the attack, quickly ascertaining that the attack was still in progress, and instructed the client to shut down all of their systems to ensure that no further files could be encrypted. The customer’s server hosts appeared to be unaffected, however, it was advised that they shut down their systems, until it could be confirmed that they were not infected.

“In an attack scenario, it is safer to assume that everything on the network is compromised, the best course of action is to shut it down as quickly as you can to stop further encryption, and concentrate on what can be rebuilt and restored,” stated Ben Randall, Krome’s Technical Director.

Working with the clients’ incumbent IT partner, Krome provided a full analysis of the systems and the attack, along with a report of possible compromise sources and our immediate remediation recommendations. When the incumbent provider was unsuccessful in recovering the data, Krome was requested to give a second opinion; “Within 48 hours of being instructed to assist we had fully evaluated how much data was accessible, what had been encrypted and we had found files that we could successfully restore from,” said Ben.

The Extent of the Attack

Due to the nature of the client’s business activities, they had a vast amount of large files stored; it was because of the sheer volume of this data that they were actually protected to some extent. In a ransomware attack, to encrypt the data, each file has to be individually encrypted, therefore the larger the data footprint, the longer it takes to complete the process.

“When we were initially asked to evaluate the extent of the attack, we investigated all of the systems and folders, once we had got to the lower level folders, we realised that it hadn’t got that far and that it was in fact, still working through encrypting the files, it was basically taking a long time due to the amount of data. This gave us the chance to stop the encryption and drill into the lower level folders to recover files before it reached them,” explains Ben.

The ransomware had added an extension to the end of each of the encrypted files, which was specific to the machine that was completing the encryption; it appeared that one single machine had executed 90% of the infection, whilst there were other extensions coming from different machines, the majority had been initiated from the one extension. Out of the clients’ 1.3 million archive files alone, 460,000 had been encrypted over the Bank Holiday weekend.

A Series of Unfortunate Errors

Whilst Krome cannot be certain of the exact way in which the attackers gained access to the network, unfortunately, there were a series of factors that could well have facilitated the attack and others that amplified its consequences.

The client had recently started to run internal phishing testing with their users, a very large proportion of their staff had failed the tests, with some users admitting that they had clicked on genuine phishing emails previously, without reporting it at the time. Many of the users had also been granted administrative rights to systems to lower the administrative burden, however, this meant that a compromise of any of those users’ accounts could easily cause a widespread breach. Additionally, there was a remote access solution that was no longer in use, but had been left available publicly.

The client’s data storage platform was also full, with just 5% free space remaining, so they had reduced the snapshot schedule down to 8 hours to avoid maintenance on the SAN; this however meant the most recent snapshot that they had taken, had been encrypted.  They were archiving data to another SAN, but again this had been limited in terms of the numbers of copies kept to avoid maintenance.

Finally, the client was backing up using Veeam to a data store in their office, however, the attackers had gained access to this as part of the attack and specifically deleted the backups, then encrypted the storage.

Recovery & Rebuilding

“We initially shut everything down, so that no further files could be infected,  we drilled down into all of the files and low-level folders to see what data, if any, could potentially be retrieved.  In many cases, recovery can successfully be achieved from a backup copy of the data, however with the client having no off-site backup, and no immutable copy of their backup, unfortunately, this was not an option,” Ben explains.

Whilst the latest SAN snapshot had been infected, the Krome team managed to locate additional copies of data which fortunately had not yet been encrypted. “Once we had located this data, we were confident that we could successfully rebuild and recover the majority of their systems and get them operational again,” Ben confirmed.

With that copy of the data, Krome built a new virtual host, on its own separate virtual network (VLAN), to ensure that it was isolated from everything else, and used that to attach to the datastores, in order to understand what data was there and whether it had been compromised at all. The systems permissions were also checked, to ensure that the account that was used in the attack did not previously have access to the recovery system.

Krome loaned the client a temporary storage array and copied the data from the recovery, creating new volumes on the SAN, once a clean copy of the data was available users could be provisioned on the new SAN, and the encrypted files could be removed from their original datastores.

“We retained a copy of the data from the old snapshot, and we purged all of the infected data off the primary site, we then moved all of the volumes back from the loan SAN onto their own storage array,” states Ben.

Krome consultants also went to the site to clean and rebuild all of their client devices, rebuilding everything from scratch to ensure that there was no risk of the ransomware being brought over again from any of the client devices.

Within 10 days of contacting Krome, their systems were operational and approximately 80% of their data was successfully recovered, unfortunately, the remaining 20% was irretrievable.

Future Safeguarding

Following on from the incident Krome has now been enlisted to work with the client to support their IT environment and help ensure that their systems are safeguarded against further attacks.

If you would like to speak to the Krome team about how to protect your organisation from a ransomware attack, or would like to discuss this ransomware case study further, please do not hesitate to contact us on 01932 232345 or use the form below.