The Hack that Stopped a Country

Jolie Hales:
I’m probably going to slaughter this pronunciation, either Snabe or Snobb or Snobbay… It reminds me of that Office episode – Sabre versus Saber.

Ernest de Leon:
Oh, man.

Jolie Hales:
Hi everyone. I’m Jolie Hales.

Ernest de Leon:
I’m Ernest de Leon.

Jolie Hales:
Welcome to the Big Compute Podcast. Here we celebrate innovation in a world of virtually unlimited compute. We do it one important story at a time. We talk about the stories behind scientists and engineers who are embracing the power of high performance computing to better the lives of all of us.

Ernest de Leon:
From the products we use every day to the technology of tomorrow, high performance computing plays a direct role in making it all happen, whether people know it or not.

Ernest de Leon:
So Jolie.

Jolie Hales:
Yes.

Ernest de Leon:
Have you heard enough about SolarWinds at this point?

Jolie Hales:
I think it’s super interesting, but it’s a bit exhausting, I’ll be honest.

Ernest de Leon:
Would you say that you’re solar winded?

Jolie Hales:
Oh my gosh. That is… Wow. Now that you have a baby, you have apparently been inducted into the Association of Dad Jokery, so my full congratulations.

Ernest de Leon:
I’m sorry. I just couldn’t pass that one up.

Jolie Hales:
Before we’re enlightened by more of this hilarity, I did just want to point out that this episode is basically part three of a three part series about cybersecurity. So if any of our listeners missed that last couple of episodes about the SolarWinds hack, we highly recommend you go back and listen to those before continuing with this one so that you have all of the background information that you need in order to understand what’s going on.

Ernest de Leon:
Yes, or you might miss some of the context around parts of this episode.

Jolie Hales:
For the rest of our audience, just a quick review on where we ended up in the last episodes. So SolarWinds’ popular IT management software called Orion ended up being hacked by what is suspected to be Russian intelligence. In fact, we just barely heard on the news this morning that we’ve got some repercussions, right?

Ernest de Leon:
Yes.

Jolie Hales:
Some sanctions, I guess, on Russia.

Thomas W. Bergeson:
President Biden signed an executive order Thursday holding the Russian Federation accountable for efforts to engage in and facilitate malicious cyber enabled activities against the United States. It was in response to the month’s long SolarWinds hack, which affected more than 18,000 customers of SolarWinds software, including most Fortune 500 companies, government agencies, hospitals, universities and even the city of Tampa.

Jolie Hales:
It’s suspected that the official hack began in early 2020 around the same time that the coronavirus pandemic was starting to spread across the globe, but it wasn’t discovered until December when the cybersecurity company FireEye caught it and then they sounded the alarm.

Ernest de Leon:
Right. The truth of the matter is that this is still an evolving situation with a lot of unknowns, including exactly what information was gathered in this hack. We do know it was extensive and being called one of the most sophisticated hacks of all time. In our last episode, we talked about worst case scenarios or what hackers from a nation state could potentially use this acquired data for and what they might want to do with it.

Jolie Hales:
Those possibilities include everything from gathering vaccine research data and national defense information or technology secrets to more nefarious possibilities like overriding the electric grid, messing with water supplies, interrupting supply chains, tinkering with the United States’ financial system and all of that, which is scary.

Ernest de Leon:
Anything to cause chaos.

Jolie Hales:
Yes. It’s interesting because as I’ve been doing some research around the SolarWinds hack, I learned that a sort of worst case scenario breach has actually already happened before. Yet, it’s crazy, Ernest, it seems like very few people have even heard of this story. I mean, it basically took out a country, and it cost billions of dollars in damages. Yet, still so many people haven’t even heard about it, which is crazy. Since this story hasn’t been widely told, I think it’s time that we shine a light on it here, because I really think it illustrates what can happen to the average person should a larger attack or hack occur. So in my typical fashion, I’d like to paint a picture for you and put you and our listeners into the scene of what happened during this crazy cybersecurity breach. Get ready, Ernest. Are you ready to use your imagination?

Ernest de Leon:
Enchant us with your prose.

Jolie Hales:
Okay, I’ll do my best. Picture this, it’s a couple years before the pandemic. Let’s just say you’re working in an office building around 20 miles away from where you live. It’s not long after lunch, and you’re wrapping up work early today because tomorrow is a big national holiday and you’ve invited some friends over to your apartment for a little celebration. The problem is, your fridge is basically down to empty and you desperately need to do some grocery shopping.

Jolie Hales:
So you wish your coworkers a good afternoon, hop in your car and you drive to the closest grocery store where you painstakingly pack your cart full of much needed groceries and then you pull it up behind a few people in the checkout line. You stand and you wait, pull out your phone, maybe start scrolling through social media, maybe some emails, jump on Reddit, whatever. Then after a few minutes, you look up from your phone and you realize that the line hasn’t moved at all, and many more people have lined up behind you.

Jolie Hales:
So of course, you look ahead to see what the deal is, and the grocery store clerk looks completely confused, pushing buttons on the machine in front of her as the customer at the counter keeps swiping her credit card again and again but to no avail. Just as you’re about to move to another checkout line, you realize that the next clerk looks just as confused and the next clerk. In fact, every register clerk is frantically pushing buttons in complete bewilderment.

Jolie Hales:
Store managers eventually appear from back offices to help troubleshoot the issue, but nothing is working and everyone is basically at a standstill. Well, this is inconvenient for you, but you suppose you can wait a bit, at least. So you go back to your phone for a while longer, but then after 10 minutes of cat videos or whatever, you look up from your phone to see that still nothing has moved and people are really starting to get impatient. Just then, a store manager raises her voice to the crowd and says-

Grocery Store Manager:
We’re very sorry, our credit card system is down right now. We’re working as fast as we can to get it back up. But right now, we can only take purchases made with cash.

Jolie Hales:
You open up your wallet, you have seven dollars. Not even close to cutting it. Knowing that there’s an ATM outside, you leave your full cart and you head toward the front doors hoping to grab some quick cash and then go back and then buy your stuff. As you leave the store, you turn around and you catch a glimpse of the register monitors. All of them are black with the same red text on them. Weird. Anyway, you get to the ATM, and another person is already there pushing buttons in frustration until they finally walk away in a huff. You step up to the ATM to insert your own card, but then you stop. Instead of a welcome message or advertisement, the ATM screen greets you with red letters over a black background.

Automated Voice:
If you see this text, then your files are no longer accessible because they have been encrypted. Perhaps you are busy looking for a way to recover your files, but don’t waste your time. Nobody can recover your files without our decryption service.

Jolie Hales:
You recognize this screen as being identical to the ones that you saw as you were leaving the grocery store, and that’s really strange. Then you spot a bank across the street, so you run over to its outdoor ATM, but you can’t believe what you see when you get there — the same black screen with a red lettered message. You run to another bank. This one’s three buildings down, and again their ATM greets you with red text over a black screen. This is really starting to creep you out. You decide to give up on the groceries for now, so you hop in your car and figure you’ll grab some gas and then head home. But as you pull up to the gas station, you notice an unusual line of cars backed up from the pump and people standing around. You roll down your window and you ask what’s going on.

Driver:
The pumps are down.

Jolie Hales:
Someone says.

Driver:
They’re not accepting payments here.

Jolie Hales:
Another driver overhears your conversation.

Another Driver:
This is the third gas station I’ve been to.

Jolie Hales:
He says.

Another Driver:
The others are having the same problem.

Jolie Hales:
You look at your gas gage. You don’t even have enough gas to make the drive home. Frustrated, you park your car into the gas station and then you run five blocks to the nearest rail station thinking maybe you can hop a train home. But when you get there, you’re greeted by more confused crowds of people and more electronic systems that aren’t working, making purchasing and scanning tickets impossible. You stand there flabbergasted. What in the world is going on? Just as you’re trying to come up with an answer, the power goes out.

Jolie Hales:
This is a thing that really happened on June 27th, 2017, suddenly around 10% of all the computers in the country of Ukraine went black with this red message with huge swaths of systems going down at the same time at places like supermarkets, banks, the mail service, airports, government agencies, energy companies, TV stations, hospitals and even the radiation monitoring system at Chernobyl.

Ernest de Leon:
I’m going to take a quick second here to delve into one of my favorite areas-

Jolie Hales:
Oh no.

Ernest de Leon:
… movies that are so bad, they’re good.

Jolie Hales:
Oh my gosh. Everything we talk about on this show reminds you of some terrible movie.

Ernest de Leon:
Yeah, it’s the truth.

Jolie Hales:
What’s this one?

Ernest de Leon:
The movie is Live Free or Die Hard.

Live Free or Die Hard – Trailer:
You have no idea who you’re dealing with. I’ll take it from here. You just killed a helicopter with a car. I was out of bullets.

Jolie Hales:
Is that the Bruce Willis movie?

Ernest de Leon:
Well, all of the Die Hards are Bruce Willis, right? It’s his franchise or whatever. So he’s completely typecasted for this. The movie introduces the term fire sale. Now, that’s not a true cybersecurity term, at least until the movie came out, but it parallels the story. So I’m not going to ruin the plot for those of our viewers who have excellent taste in film, but you should absolutely watch it-

Jolie Hales:
Excellent.

Ernest de Leon:
… if you love these types of bad movies like I do.

Jolie Hales:
That is so random. Now I guess I have another Ernest recommended movie to add to my watch list, or maybe my do not watch list, alongside, what was it, Lavalantula and VelociPastor.

Ernest de Leon:
Yes, those are two excellent movies.

VelociPastor trailer:
A troubled priest.

Priest:
How long has it been since your last confession?

Woman:
The time you turned into a dinosaur and ate someone.

Priest:
What?

VelociPastor trailer:
The VelociPastor.

Jolie Hales:
I mean, if we’re going to talk about movies, I honestly think that the Ukraine hack would make a really fascinating movie, or at least a solid documentary. I haven’t seen one made on this. Maybe there’s one that exists, I don’t know. The effects of this breach were enormous. I mean, maybe not as extensive as SolarWinds, but very big. For example, the Ukrainian Postal Service, they have a fleet of 2,500 trucks, and they have apparently around 74,000 employees. They had more than 16,000 of their 23,000 computers literally wiped out in less than a single minute. I mean, it was business as usual until one minute in the early afternoon when suddenly something like 70% of their computers just went black with this red message. Then none of their data was suddenly accessible.

Ukrainian:
The computer became uncontrollable. A chain reaction started. The first, the second, third, fourth, fifth computer.

Jolie Hales:
Ukraine’s second largest bank lost 90% of its thousands of computers in less than one minute, so really fast. This is just in Ukraine. I mean, we heard the example of the grocery stores. A lot of them lost ability to take any payments, and some were accepting cash.

Thomas W. Bergeson:
Already, the virus has spread across the globe. French country company, Saint-Gobain, German rail, Deutsche Bahn, Russian energy giant, Rosneft, and the US pharmaceutical company, Merck, are just some of the private firms to have been hit.

Jolie Hales:
For example, if you were a trucker heading to, let’s say, the Port of New Jersey in the United States to drop off your container for some ocean shipping on this date of June 27th, 2017, you would’ve encountered a line of hundreds of stranded 18-wheelers trying to get to the Maersk Terminal at the port but being unable to enter because basically the entire company’s computer system had been wiped by the malware, which not only prevented Maersk employees from being able to use the computers, phones, scan barcodes, but they couldn’t even open their electronic gates to let the trucks inside. So there were hundreds of these big trucks just lined up wondering what in the world was going on, and the road was too narrow to even turn around.

Jolie Hales:
Now, that was just the Port of New Jersey. Maersk is one of those companies that even if you don’t realize that you’ve heard of them, chances are you’ve driven by their semi-trucks a million times and would recognize their logo because they’re the world’s largest shipping company with 80,000 employees across more than 500 offices in 130 countries with terminals at all the major ports, including New Jersey and here in Los Angeles near me.

Jim Hagemann Snabe:
We transport roughly 20% of world trade in containers, so we’re a very significant part of the infrastructure of making the world actually run.

Jolie Hales:
That was the voice of Jim Hagemann Snabe at the World Economic Forum in 2017.

Jim Hagemann Snabe:
Every 15 minutes in average, a container ship will come through a port somewhere with between 10 and 20,000 containers.

Jolie Hales:
Apparently, their breach all began with one single computer being infected. Then within hours, the entire company around the world was down, and not just down, but their data across every server was basically wiped completely clean.

Ernest de Leon:
A company of this scale and resources surely had backup stored offsite.

Jolie Hales:
Okay, well that’s the thing. Apparently they had backups, but the backups were connected live to the same network as everything else in order to allow for an instant recovery if a server went down. So the backups were wiped as well.

Ernest de Leon:
Beautiful.

Jolie Hales:
Yeah. Crazy. So all of this vital information about billions of goods being transported across the globe by customers everywhere was suddenly gone instantly. US Lieutenant General Thomas W. Bergeson paraphrased from an awesome Wired article that I’ll link to in the episode notes describing how this hack immediately affected Maersk.

Thomas W. Bergeson:
As a result, ships were unable to communicate their contents to the terminals. Sprawling fields of containers were stacked eight high over square miles of shipyards with no way of knowing their contents without opening them individually.

Ernest de Leon:
This brings a whole new perspective to the recent grounding of the massive container ship Ever Given in the Suez Canal.

Jolie Hales:
Oh yeah.

Ernest de Leon:
Imagine that incident but now every container ship from a huge company is blank.

Jolie Hales:
Oh my gosh. That’s an interesting way to think about it. I’ve got to say, Maersk should be thanking their lucky stars, frankly, because they had a major miracle bestowed upon them in this situation. Apparently when the malware infected and shut down all of their systems around the globe, there just happened to be one single domain controller located in Ghana, Africa that had been knocked offline by a power outage at the time the systems were infected.

Ernest de Leon:
The randomness of the universe.

Jolie Hales:
Yeah. Nuts. Since there wasn’t the bandwidth available to transfer the data from Ghana to the Maersk IT Department in Great Britain, they ended up physically flying that one uninfected server overseas from Ghana to Nigeria to London exchanging hands, because people traveling with the server didn’t have the correct visas to travel with it. Then they were ultimately able to use the data to completely rebuild every computer in the company one-by-one over two months. In total, even though they got really lucky there, the breach ended up costing Maersk more than $300 million.

Ernest de Leon:
That seems rather risky. Imagine if that plane had crashed anywhere along the way. I would’ve just disconnected the server from the network in Ghana, imaged it and flown the image drives over.

Jolie Hales:
I agree, but maybe they didn’t have the resources to do that kind of thing or something at that Ghana location. I’m not really sure. Talk about not being able to relax at all as that airplane was flying to Great Britain. If I were them, I’d be so freaked out. Honestly, Maersk is just one of the companies hit that happened to be publicly forthcoming about this hack. In total, they say $10 billion in damage is attributed to this breach, including $400 million in damage for FedEx and $188 million in damage for the company that makes Oreos and Cadbury Chocolate, which that definitely makes this personal.

Ernest de Leon:
Absolutely. If you affect the junk food market, there will be consequences.

Jolie Hales:
Yep.

Ernest de Leon:
Do we know who was behind the hack?

Jolie Hales:
Experts allege that, this is going to be a shocker-

Ernest de Leon:
Allegedly.

Jolie Hales:
Similar to SolarWinds, this hack was done by Russian intelligence.

Ernest de Leon:
Not surprising at all. Most of the problems we see in the cybersecurity world originate from less than a handful of locations.

Jolie Hales:
One of the reasons that they’ve come to this conclusion is because typically ransomware attacks are done by individuals or small groups that were looking to make a quick buck by holding your data for ransom until you send a certain amount of Bitcoin. Then they provide a decryption key to restore all your data, as we’ve talked about in past episodes. Although, we should note that people should not cooperate with them, because it only encourages them to keep doing this.

Ernest de Leon:
Like the ransomware attack you talked about a couple of episodes ago.

Jolie Hales:
Exactly. I did not pay, if you hadn’t heard it. I did not pay. As it turned out, while this particular malware that hit Ukraine definitely looked like ransomware on its face with that same typical, “Oops, your data has been encrypted,” ransom message and all, it turned out that this message was just a façade intended to disguise what the virus really was. While ransomware usually makes it clear and easy to send the perpetrators money and they encrypt the files in a way that can be easily decrypted with a key, this particular malware did neither of those things.

Ernest de Leon:
Probably a repurposed malware.

Jolie Hales:
Yeah, I think it was in fact. I mean, for instance, the email address listed for Bitcoin payments was basically made defunct almost immediately after the virus launched. Apparently, it also encrypted the files in such a way that it would extremely difficult to ever decrypt them. So because of this, it became clear that the purpose of this Ukraine attack was not to make money for a small group.

Ernest de Leon:
It was intended to do massive damage.

Jolie Hales:
Exactly. These “encrypted files,” were actually never meant to be restored. The hacker wanted to delete data and systems and cause an incredible amount of chaos for Ukraine and anyone who did business with Ukraine. That takes it out of the ransomware category and puts it in the cyber weapon category.

Jolie Hales:
From supersonic jets to personalized medicine, industry leaders are turning to Rescale to power science and engineering breakthroughs. Rescale is a full stack automation solution for hybrid cloud that helps IT and HPC leaders deliver intelligent computing as a service and enables the enterprise transformation to digital R&D. As a proud sponsor of the Big Compute Podcast, Rescale would especially like to say thank you to all the scientists and engineers out there who are working to make a difference for all of us. Rescale, intelligent computing for digital R&D. Learn more at rescale.com/bcpodcast.

Jim Hagemann Snabe:
That meant that we were actually collateral damage of a probably state attack situation.

Jolie Hales:
That’s the Chairman of Maersk again.

Ernest de Leon:
What was the inception point and the replication pattern?

Jolie Hales:
Apparently it started when the update server of accounting software called MeDoc was hacked. The virus was placed there, and then MeDoc pushed out an update to all the systems using the software. So somewhat similar to what we heard about SolarWinds. Apparently in Ukraine, MeDoc is very widely used. It’s more widely used in Ukraine than TurboTax or QuickBooks is here in the United States. So that update went to a lot of machines. Then when it first infected a person’s computer, people didn’t realize it. Then it acted like a worm from there, and it quickly spread through known vulnerability in Windows systems that weren’t updated with the latest patch. So if you had a patch, then it didn’t spread directly to your system at that point.

Ernest de Leon:
Color me surprised.

Jolie Hales:
Then as the worm spread, it then gathered credentials from systems that allowed it to infect other computers on the networks that did have the Windows patch in a matter of minutes. Then it would start encrypting all the files on each computer. So it didn’t matter at that point if your computer was patched or not. It was able to get there through these credentials. After about an hour and a half of encrypting all the files on your computer, then the computer suddenly reboots, maybe as you’re using it. Then that red message appears over a black screen, and there’s nothing that the user can do about it.

Ernest de Leon:
Sounds about right.

Jolie Hales:
Yeah. I mean, I can’t even imagine working in a building with multiple people at multiple computers and then just seeing every monitor turn black with this scary red message all at the same time. It’s like something out of a horror movie.

Ernest de Leon:
Or like something out of another movie that’s so bad it’s good. I’m of course speaking about the movie Hackers from 1995 and the end scene where they move to attack the mainframe at Ellington Mineral Corporation. Another must-watch for those with exquisite taste in film.

Jolie Hales:
This is the third episode where you’ve talked about that stupid movie. I still haven’t seen it. I mean, I’m going to be honest, the trailer was enough enlightenment probably for me.

Hackers – trailer:
Hackers.

Jolie Hales:
All of this brings us back to the SolarWinds hack. SolarWinds is different in that there doesn’t appear to be any destructive malware placed on the 18,000 or so organizations’ systems, but rather it appears that the perpetrators were either just hoping to gather information over time or-

Ernest de Leon:
Prepare to hit the United States like they once hit the Ukraine.

Jolie Hales:
That’s the craziest thing, because while the Ukrainian breach, which by the way is called NotPetya.

Thomas W. Bergeson:
NotPetya.

Thomas W. Bergeson:
NotPetya.

Thomas W. Bergeson:
NotPetya.

Jolie Hales:
While it had a global impact, Ukraine is, I mean we know significantly less connected to the world’s economy and business infrastructure when compared to a country like the United States or some country in Western Europe or something. In theory, an attack like this on even just a few select organizations out of the 18,000 could really kind of break the world for a bit. I mean, the Associated Press stated in an article that I’ll link to in the episode notes on BigCompute.org, they said, “What saved the world from digital mayhem was its limited business-to-business connectivity with Ukrainian enterprises, the intended target.

Thomas W. Bergeson:
It began as an assault on Ukraine, but its effects were felt around the globe. It was described by cyber experts as the equivalent of using a nuclear bomb to achieve a small tactical victory.

Ernest de Leon:
Yes, launching this type of attack against a world superpower like the United States would require a delicate balance. They would need to cause enough pain to be recognized, but not so much that they ended up a nuclear wasteland.

Jolie Hales:
Yeah, that’s a good way to put it. I mean, you definitely are thinking like a cybersecurity expert where, for example, I’ve happily become basically a paperless person. I don’t write checks. I barely ever have cash at hand. I manage bills, banking, all of that stuff completely online. I think that a lot of people these days are probably in the same boat, so it was alarming to me to read about the effects of this NotPetya attack on Ukraine, because it really made me realize how unprepared I am for something like this. I mean, I probably should sew some cash into a pillow or something as a precaution, because my ability to purchase food or gas or anything is all tied to credit card processing systems, which are all susceptible to these digital catastrophes.

Ernest de Leon:
As a Texan, I’ve hedged against a lot of this, but I totally understand what you mean. We’re increasingly in a digital world, and our exposure increases as the percentage toward complete digital increases.

Jolie Hales:
Yeah, and that brings us to now. What we do as individuals, as organizations, as people in the high performance computing industry, what can we do to protect ourselves and to protect our way of life? I want to go back to the Senate and Congressional hearings that we referenced in the last episode, and there were a few themes that came up that were about what we need to do for the future. The first key that’s brought up is modernization of infrastructure.

Brad Smith:
I think we have a lot of work still to do, certainly across the United States, when it comes to the modernization of our IT infrastructure and to the application of IT best practices.

Jolie Hales:
That’s Brad Smith, President of Microsoft, speaking at the Senate Intelligence Committee hearing.

Brad Smith:
Until we modernize and move more people to the cloud, we’re going to be operating with less visibility than we should.

Jolie Hales:
In fact, we mentioned in earlier episodes that Florida water facility hack, but I saw you tweet a story, Ernest, this week about a similar water facility hack in Kansas. I mean, what happened there?

Ernest de Leon:
Yeah, so this one is a great example of one of the areas we look at in cybersecurity called insider threat. In this case, a 22-year old former employee hacked the public water utility in a rural Kansas area and turned off the facility’s cleaning and disinfecting faculty.

Jolie Hales:
Really?

Ernest de Leon:
Yeah. The best part is that the indictment claimed that he did this from a Samsung phone.

Jolie Hales:
So this 22-year old former employee. I mean, why?

Ernest de Leon:
I didn’t see that in the indictment, but I can tell you that dealing with insider threat, it’s typically one of a few things. The first one is disgruntled former employee. They were fired for whatever reason. This individual wasn’t. The other reason is usually something to do with money. So you have an employee who is compromised in one way or another for whatever reason, and a third-party offers to pay them if they will either give them access or do something for them. That’s one of the reasons why during security checks for people trying to get security clearances, they interview friends, family and-

Jolie Hales:
Really?

Ernest de Leon:
Yes. More importantly, they look at things like your credit history, because while it may not seem relevant, somebody who makes poor decisions in that regard often finds themself in a place where they’re more easily compromised than somebody who doesn’t.

Jolie Hales:
That’s so interesting. I didn’t know this. So it’s just people’s level of temptation to earn a quick buck by agreeing to work with the Russian intelligence or something?

Ernest de Leon:
Sometimes it’s as tacky as doing it for a quick buck. Other times, people are in a very compromising position in terms of finances-

Jolie Hales:
Interesting.

Ernest de Leon:
… and this is their ticket out of that situation, or at least they think it is, without realizing that they’re going to be much worse off when the FBI comes knocking on their door.

Brad Smith:
Especially when we’re talking about publicly owned critical infrastructure in this country, a lot of it is too old. It needs to be modernized. That’s why you see these ransomware attacks, which need to connect with us. They so often target municipalities. We’ve seen Baltimore. We’ve seen New Orleans. They target hospitals. So that is in critical need of improvement.

Jolie Hales:
Another theme that continuously came up throughout both hearings was the idea of disclosure obligations and then liability protections for the private sector when they’ve been hacked. Apparently, a challenge that we’re having in the United States is that often when private companies discover a breach, they then don’t disclose it to customers or officials because they’re in fear of litigation, that they’ll be sued for allowing a breach to occur or whatever and then they’ll be held accountable for damages, let alone the worries that they have about earning a bad brand reputation. Now, the problem is that not disclosing a breach then means preventing pertinent information from protecting other potential victims. Additionally, many private sector organizations just aren’t equipped to fully understand and resolve some breaches. So not disclosing means they could still be vulnerable without even knowing it.

Ernest de Leon:
Right. So there are two facets to this scenario. One you mentioned, which is purposeful foot dragging due to fear of litigation or PR damage. The second is that it sometimes takes a while to do a full security audit and forensic examination to understand the full depth of the breach as well as to ensure that it has been mitigated. I agree though that something must be done to incentivize full disclosure sooner, perhaps some sort of legal immunity from damages that result from non-negligence so that the fear of disclosure is reduced. But in general, it is never the right answer not to disclose. You will always end up with a worse situation down the line, because that information will get out at some point and it will be discovered that you knew about it. That’s when the litigation really becomes negligence and purposeful deceit of either your customers, or even worse, investors that you have a fiduciary obligation to.

Jolie Hales:
It’s why there are arguments that the solution to United States cybersecurity lies both in technology but also in diplomacy. Senator John Cornyn summed up his thoughts at the Senate Intelligence Hearing.

Senator John Cornyn:
It seems to me that there should be an obligation of some sort on the part of a victim of a cyber attack like this to share what they know, what they’ve learned with the appropriate authorities. I can only imagine the chills that run up and down some people’s back when I say that. Think about liability concerns, other reputational risks and the like. If we’re going to get our arms around this at all, it seems to me we need to know a lot more than we know under the current practices in terms of the obligation of the victims to step forward.

Jolie Hales:
Just as you’ve suggested, those leaders testifying at these hearings recommended that the government create a reporting system so that organizations are then obligated to immediately notify a central government agency of a breach while also being protected from litigation, at least within reason for doing so. Brad Smith at Microsoft summarized his perspective.

Brad Smith:
It starts with identifying who needs to report, what they need to report, to whom they need to report it and how. I do think one thing that is worth touching upon that we really haven’t perhaps talked about at this hearing is the critical need to enable people who have this information to report it easily and in a streamlined manner, because we are acting as the first responders. In a sense, when an incident is unfolding, we’re fighting a fire. You don’t want to take people away from the fire, so they’re filling out a lot of forms and doing things that are going to detract from their ability to respond. So I would hope that one design principle that would be built into this would be the need to do it simply, efficiently and in a manner that is sensitive to the work that is needed while an incident is unfolding.

Jolie Hales:
Undercover superhero and CEO of FireEye, Kevin Mandia, said you have to offer protection, or the lawsuits that would follow would just be another win for the hackers who thrive on creating chaos.

Kevin Mandia:
You don’t want the attacker to win twice. Once they broke in … Well, actually it’d be three times. They broke into SolarWinds. They had what looks to be a very successful deep blast zone type of cyber espionage campaign, and then they harm American companies both in shareholder lawsuits, liabilities and investigations. It’s like a trifecta for the adversary against us. So we’ve got to think of a way where we play team ball as a nation where we all come together. I do believe the fastest thing we can do, we’ve been talking about a lot today, ma’am, get the threat intelligence into an agency in the government. Then from there, it gets pushed out to the security community so we can go shields up a lot faster. Best we can do, ma’am, is maybe somebody’s a victim, but we’re all as secure as the very last victim in cyber crime.

Ernest de Leon:
He is 100% correct. We need more from our government in terms of real-time threat intelligence, but also incentives for companies to come forward faster when a potential breach has occurred.

Jolie Hales:
Another theme that came out at the hearings was around attribution and accountability, or in other words, that it’s important for the United States to not only sniff out and then openly name the perpetrator, but also hold that perpetrator accountable with some kind of consequences to discourage them from repeating this kind of attack in the future.

Brad Smith:
Is it a diplomatic issue or is it a technical issue? Yes, that’s a way of saying it’s both. We need to deal with it on both levels. I don’t believe for a moment that we live in a world where our adversaries are more capable than our own government, but we do live in a world where there is an asymmetry. It is easier to play offense than it is to play defense. When you play offense, you can scan the horizon and look for the weakest point, and then that is where you direct your energy. When you’re on the defensive, that means you need to scan and secure the entire horizon.

Brad Smith:
So on the technical side, that means that there’s this enormously important work to strengthen all of our cyber defenses and it equally makes it a critical diplomatic and international legal issue, because it simply must be the case that there are certain acts that are put off limits and for which there are international and diplomatic consequences. This kind of indiscriminate and disproportionate attack on the software supply chain is and should be one of them.

Kevin Mandia:
We’re all playing goalie, and we’re taking slap shots from Wayne Gretzky. I mean, the puck’s going to get in the net sooner or later, and that’s what’s happening in cyberspace right now. Folks are taking slap shots, and literally there is no risk for repercussion to the folks doing it. So we’re all fighting a losing battle over time.

Jolie Hales:
Right now, there’s concern that there haven’t been consequences for nation states that pull off these kind of hacks, or at the very least, consequences that haven’t been consistent. As we talked about in the case of the SolarWinds and NotPetya attacks, experts say that there’s solid evidence that both were perpetrated by the Russian Government. Though of course, Russia has denied any involvement in both cases, but experts at this point just don’t believe them, which is why they recently announced sanctions on Russia.

Ernest de Leon:
Yeah, it’s kind of like when you catch a kid with melted chocolate chips all over their mouth, and you’re like, “Did you eat the cookies?”, and they’re like, “No, I didn’t eat the cookies.” That’s exactly what happened. I will point out that there’s a delicate balance that has to happen with respect to international diplomacy. It can’t be stick, stick, stick, or there will never be peace. Foreign policy has to be a blend of carrot and stick. You have to find common interest and build from those such that it doesn’t make sense for the other side to pursue bad behavior.

Jolie Hales:
Oh man. I mean, this is why I’m grateful that I’m not a politician, because these are pretty complicated scenarios. I honestly just can’t imagine trying to make these decisions. In fact, Lieutenant General Thomas Bergeson, who I mentioned earlier, put it this way in 2019.

Thomas W. Bergeson:
Russia had been using their years long conflict with Ukraine as a scorched earth testing ground for their cyber tactics.

Jolie Hales:
That’s why even though the SolarWinds attack and the NotPetya attack in Ukraine were very different and separated by a few years. I kind of group them together in my mind wondering if somewhere down the line what happened in Ukraine could have happened in the United States given that the hackers had access to a ton of information and backdoors into a lot of organizations in the SolarWinds situation. Also, we don’t know 100% what the hackers actually did and what information they actually got.

Brad Smith:
Who knows the entirety of what happened here. One entity knows. It was the attacker. The attacker knows everything they did. Right now, the attacker is the only one that knows everything they did. We have pieces. We have pieces at Microsoft, SolarWinds, FireEye, CrowdStrike, others, we all have slices, people in the US Government, but we need to bring those slices together. Until we do, we’ll be living and working and defending on an uneven playing field. That is not a recipe for success.

Ernest de Leon:
I agree 100%. We need a lot more effort around data correlation across these breaches as well as threat assessment and mitigation that constantly evolves as we gain new data.

Jolie Hales:
Another problem that needs solving that kept coming up during the hearings, and this directly applies to a person like you, Ernest, is the fact that there’s apparently a big cybersecurity talent shortage in the United States, which is interesting because there’s also a shortage of talent in the HPC industry right now.

Brad Smith:
I don’t think we can secure the country without investing in more cybersecurity people for the country. There’s really a critical shortage nationwide of cybersecurity professionals. I think we can put our community and technical colleges to work in part to get more people into public agencies, into small businesses and others.

Jolie Hales:
So if you know anyone trying to decide what career they should go into and they happen to like computing, it sounds like well paying jobs in cybersecurity are quite available and people in those fields are in strong demand. Though I wonder how well you guys sleep. I might be scared to know what’s going on all the time.

Ernest de Leon:
That’s a good point. Our profession is one that pedals in paranoia, so-

Jolie Hales:
I would rather be ignorant.

Ernest de Leon:
I would say that in general, this is true and the need is dire. Part of the problem is that cybersecurity is a specialty area of the larger computer science or technical field, kind of like a neurosurgeon versus a general practice doctor.

Jolie Hales:
Got it.

Ernest de Leon:
You have to be comfortable enough with computing to take a hyper-specialized area and make that your practice. Like anything else in the world, if it were easy, everyone would do it.

Kevin Mandia:
I don’t think it takes a lot of people to test your networks on how secure they are. I do believe that’s the best way to get unvarnished truth in security. Kind of like you do crash test dummies to test the safety of a vehicle, shoot real bullets at of bulletproof vest to determine how effective it is, in cybersecurity, you need to test your security. That’s a couple folks. There’s a great assymetry between offense and defense. To have somebody perpetrate what would be perceived as offense, not a lot of resources. The problem is, the 52 card pickup you play on the other side because of that assymetry. One attacker can create work for hundreds of thousands of defenders. It’s a bad assymetry is cyberspace I think other nations have picked up on where they can’t beat us with tanks, won’t beat us with planes, but in the cyber domain, if they train folks, the A team can create work for potentially millions of defenders. So the bottom line, that assymetry is the problem. It’s hard to answer your question without cataloging offense, very few people, defense, you have to pitch a perfect game every day and put a lot more people on it.

Jolie Hales:
Then of course, there are the steps organizations can actually take to prepare themselves against these kinds of hacks. This is an area that you, Ernest, are, I would say, pretty well versed in. In fact, I’ll even put a plug in here for a blog that you recently posted on BigCompute.org. You walk people through steps that they need to take in order to make sure that they are secure.

Ernest de Leon:
That’s right. Key areas of focus for organizations and those in the HPC space specifically should be around physical security, network security, operating system security, application security, user security, file security and integrity, logging and monitoring, and lastly but most importantly, redundancy and backups.

Jolie Hales:
So if anybody wants to read about those details, take a look at BigCompute.org.

Ernest de Leon:
I know I’ve said it before, but it’s not a matter of if you’ll be hacked, but when.

Kevin Mandia:
I’d like to remind folks that this was a foreign intelligence service that hacked into 17,000 different organizations. I would ask the members of Congress to think, “Is it reasonable for our companies to defend themselves from a foreign intelligence service? Is that the bar that we want to set for this nation’s private sector?

Ernest de Leon:
Since breaches are so common, everyone can and should take steps to not only lessen the chances of being hacked through being proactive, but also how to best mitigate the damage done should someone breach your system.

Kevin Mandia:
To me, the attacker that did the SolarWinds implant, they’ve already moved onto whatever’s next. We’ve got to go find it. This attacker, maybe their pencil’s down for a few months, but the reality is they’re going to come back. They’re going to be an ever present offense that we have to play defense against. How they break in will always evolve, and all we can do is close the window and close the security gap better next time.

Jolie Hales:
The chairman of Maersk summed it up, I think, pretty well after his company recovered from the NotPetya attack.

Jim Hagemann Snabe:
It is time to stop being naïve when it comes to cybersecurity. I think many companies will be caught if they are naïve, even size doesn’t help you. I think it is very important that we are not just reactive but proactive. I think we can’t be average. We’ve got to be the best we can.

Jolie Hales:
That’s going to do it for this episode of the Big Compute Podcast. To join the Big Compute community and learn more about upcoming events, go to BigCompute.org and type your email address in at that bottom of the website to sign up for our newsletter.

Ernest de Leon:
If you want to help us spread the word, you can leave us a five-star review wherever you get your podcasts like Apple Podcasts.

Jolie Hales:
Google Podcasts.

Ernest de Leon:
Spotify. There’s many of them.

Jolie Hales:
All right everyone, thanks for joining us.

Ernest de Leon:
Always use multi-factor authentication and follow the 3-2-1 Backup Plan. Goodbye.

Jolie Hales:
Bye.