Executive Editor’s Note: This article and the related article on page 1 of this issue were submitted as one article, but to highlight the need that was caused by this attack, that portion of the article has been separated into a second article.
A Day Like No Other
January 27, 2022 was like any other Thursday, with one exception — it was Homecoming weekend, and many were gathering on campus for the festivities. I was in my office doing some last-minute work on my computer. It was a few minutes after 3 p.m. — I know, because what happened next is forever etched in my mind. My desktop computer shut down in the middle of a document I was typing. We rarely have an interruption in power, and the lights on my computer were still on. It was then that I realized this could be really, really bad.
The Takeover
Not only did my computer shut down, but I quickly learned that all 230 desktop computers on campus had rebooted and this message was on the screens: “You now have a new Admin.” CBC Information Technology (IT) personnel went to work immediately, unplugging all desktop computers and anything else that was on the network, and our IT director began the process of powering down the servers. It was grimly obvious what had occurred, but a phone conversation would soon confirm one of my greatest fears — we were undergoing a cyber-attack and a ransomware attack as well. What we didn’t know initially was how deeply they had infiltrated our system, nor did we know when and how it occurred. Had they only taken over the desktop computers, or were they in all 35 of our servers? All we knew for certain was that it was not good.
The Call
Approximately 15 minutes later, while I was standing in the foyer of the David T. Watkins Academic Building, I received a call on my private cell phone from an unknown number which sometimes I don’t take. But considering the urgency of everything going on around me, I answered the call. A man with a middle eastern accent, the one we would eventually label the Threat Actor (TA), spoke to me and identified himself as Michael. I asked him how he got my phone number. He said, “I know everything about you.”
According to the FBI agent, the TA was probably someone speaking on behalf of the person making the threats. In the agent’s words, “I think it would be safe to say we will never see the person behind this. They are most likely operating out of North Korea or Russia.”
The Threat
During the initial phone call, the TA instructed me to look on my phone for a message he was sending that would tell us how to unlock a file on one of our servers. There the “hackers” would tell us the amount of ransom they were demanding and how it must be paid. He told me we had 72 hours to decide or “bad things would happen” and, among other things, our “business would never recover.”
In one of my later calls with the TA, he claimed they had obtained a large amount of data that would do us great harm if sold on the dark web. He said I had not followed their instructions, and that if we did not comply, “Bad things will happen to us.” In that same call, he made numerous other threats: a DDOS attack (Distributed Denial of Service), unpleasant things that will bring the college down, continued ransomware attacks on CBC that would destroy us, and leaks of student and employee data, including their personal information.
Over the coming weeks, I would answer four such calls from the TA. One of those threatening calls lasted 20 minutes, as I listened to him rant about what the “hackers” were getting ready to do to CBC.
Incident Response Team
One of the first entities I called was the Federal Bureau of Investigation (FBI) Field Office in Little Rock. We were able to make quick contact with the “Cyber Security” agent for Little Rock. Providentially, IT had recently signed an agreement with CDW-G, a network solutions company, for times when we need online help not available locally. Our IT Director began to obtain what would become a cyber security Incident Response Team (IRT). The IRT members, spanning several time zones, worked virtually to get our system up and going within a short amount of time. We also contacted our insurance company after-hours, Brotherhood Mutual, to check our coverage for this event.
We had no idea how long it would last or if there would be permanent damage to our system and/or loss of data. From that day forward, the attorney for our insurance company gave advice as we met either via Zoom or phone call on a daily basis. Those calls would continue for the next six to seven weeks. Our IT Director, Jerry Dowdy, was on-site for the next five weeks, every day and night, working virtually with the IRT. Members of the IRT worked literally around the clock for what would be, not days, but weeks.
Negotiation
At the advice of the attorney, we retained the services of Coveware to manage the specialized process of cyber extortion negotiations. Its sole purpose was to communicate with the TA via a chat the TA provided in an encrypted file. The person assigned by Coveware was posing as someone from our IT Department, speaking on my behalf to negotiate the ransom demands and mainly to buy us time so we could assess the damage, put safeguards in place and restore all systems.
Ransomware experts troll the web looking for small vulnerabilities. They can enter the system and watch the various activities as long as they want, unnoticed until they are ready to make a move. Most importantly, they have the ability to exfiltrate files, ones they consider to be of most value, to hold over the heads of their victims. That was how they planned to damage CBC should the ransom demands not be satisfied. Additionally, and perhaps the most evil thing they do, is encrypt files on servers, making them useless.
Over the next several weeks, IT and the IRT would focus on getting the hackers out of our system, to determine how many files, if any, had been exfiltrated, how harmful would they be and what files were encrypted on our servers. The Coveware Negotiator (CN) requested from the TA samples of the material that they had exfiltrated. The TA eventually released some documents that proved they had something. Then it was up to IT and the IRT to figure out if the files they had were of any value for them. In the end, we learned that they did not have “everything” they claimed to have. They were lying and using threats and manipulation to hopefully, get us to pay the ransom.
The Ransom
In the initial chat conversation with the hackers, the ransom demand was $3,000,000 USD paid in Monero (XMR). Monero is a cryptocurrency similar to Bitcoin, but it boasts that it is secure, private and untraceable. Though the TA told me on the phone that we had 72 hours to decide, when the demand came through on the chat with the Coveware negotiator, the deadline given was five days, after which the ransom would increase to $6,000,000. Over the course of the daily negotiations, the ransom demand was reduced to $600,000, then to $500,000, with a final offer to settle at $400,000. Again, the goal of the negotiation was to buy time for the IRT and IT to do their work of restoration and recovery of all systems.
Held Hostage
The hackers kept their word in holding us hostage over the next five weeks. All 35 servers remained off-line as the IRT worked with IT staff to determine the best way to restore as much data as possible and get our system back up and running. You may wonder what is controlled by those 35 servers. Here is a list:
• Campus-wide Wi-Fi,
• Canvas (course management software),
• Admissions Office functions (including recruiting of new students),
• Financial Aid Office functions,
• Registrar’s Office,
• Access to Campus Anyware (student information system),
• Business Office functions,
• Access to some of our fundraising database (donors and alumni),
• HVAC controls,
• Campus-wide security camera system (116 security cameras),
• All voice-over IP phones and
• All electronic door locks.
Campus-wide Network Outage
It was a true blessing that classroom instruction continued uninterrupted via Zoom technology, with messaging to our students and employees about a “campus-wide network outage.” Since the insurance company attorney assumed oversight of messaging, she approved the 18 “network outage” e-mail updates I sent out to faculty, staff and students over the five-week period. The messages were intentionally vague, not only to avoid any negative public relations, but also because the last thing we needed at such a critical time was for people to panic. I did enough of that for them.
During the protracted outage, the campus-wide network was inaccessible to faculty, staff and students while the IRT worked with IT to not only restore all systems and preserve data, but also to strengthen and fortify the network to prevent any such future incidents.
Restoration and Recovery
We were blessed to have redundant off-site backups. Though ransomware detonation was successful on 262 endpoints — about half of our total — remediation and recovery activities were eventually successful. Those activities included: restoration from backups, reimaging servers, hardening the system and blocking all identified files and network indicators of compromise. On Feb. 24, 2022, restoration was deployed with critical systems brought up first. Over the next three weeks, the restoration continued, which extended six to seven weeks for some systems.
Evil to Good
As usual, God took what was intended for evil and brought us good from it. Here’s a list of upgrades to our network infrastructure:
• One added storage server and one new server host,
• New Firewall and software to monitor and detect threats,
• Deployment of Multi-factor Authentication (MFA) for all systems,
• Installation of Carbon Black on every server and computer on campus — 253 endpoints,
• Determined Personal Identifiable Information (PII) that was exfiltrated and informed the person(s) of the breach via US Mail, as required by law and
• Review of other recommendations to further harden our servers.
Thanks to the excellent service rendered by Brotherhood Mutual, CBC only had to pay the insurance deductible. That saved the college hundreds of thousands of dollars.
No Ransom Paid!
It’s important to note that no ransom was paid to the hackers thanks to the swift response by CBC’s administration and IT department and the tireless efforts of the FBI, IRT and Brotherhood Mutual.
Executive Editor’s Note: See related article on page 1 of this issue to understand the immediate need related to this attack on the college.
