The SolarWinds Hack: Worst Case Scenarios

Ernest de Leon:
We’ve mentioned Firefly who had hacking tools stolen in this breach.  

Jolie Hales:
Firefly or FireEye?

Ernest de Leon:
Oh man. Why does it say Firefly?

Jolie Hales:
I don’t know. I wonder if I wrote that. I think I did.

Ernest de Leon:
I was like-

Jolie Hales:
That’s my fault. I’m sabotaging you. Hi everyone. I’m Jolie Hales.

Ernest de Leon:
And I’m Ernest de Leon.

Jolie Hales:
And welcome to the Big Compute Podcast. Here, we celebrate innovation in a world of virtually unlimited compute and we do it one important story at a time. We talk about the stories behind scientists and engineers who are embracing the power of high-performance computing to better the lives of all of us.

Ernest de Leon:
From the products we use every day to the technology of tomorrow, high-performance computing plays a direct role in making it all happen, whether people know it or not. So Jolie?

Jolie Hales:
Yes.

Ernest de Leon:
We left the last episode on a little bit of a cliffhanger, didn’t we?

Jolie Hales:
Yes, I should say so. I mean, in fact, if any of our listeners missed the last episode, I want to take this moment to highly recommend that you go back and listen to that one before continuing with this episode, so that you have all the background information that you need to understand what’s coming next. It’s basically, this is part two of that episode.  

Ernest de Leon:
Right. Or you might be a little bit confused about what’s happening in this episode because we will definitely reference some things that we talked about in the last episode.

Jolie Hales:
And for everyone else who’s already listened to that one, just a quick review on where we are. So SolarWinds, who has been in the news a lot lately.

News Clip:
SolarWinds.

News Clip:
SolarWinds.

News Clip:
SolarWinds-

Jolie Hales:
-is an IT management software company that provides products to tens of thousands of organizations, including a cybersecurity company called, FireEye. And just a couple months ago, FireEye was the first to discover that they had been hacked. And eventually the source of that security breach was traced back to a commonly implemented SolarWinds software update on their Orion product.

Ernest de Leon:
Yes, the malware was actually attached to an update that rolled out to countless organizations, ultimately breaching the systems of at least 18,000 organizations.  

Jolie Hales:
Right. And FireEye was just the first to catch it. And the other organizations that were breached include major tech behemoths like Intel and Microsoft, as well as United States government agencies like the Department of Defense.  

Ernest de Leon:
And actually, I’ll note something here. There was a congressional committee hearing that happened not too long ago where they questioned some of this.  

Jolie Hales:
I’m so glad you brought that up, because after we recorded our last episode about this hack, I got really curious and I ended up watching and taking scrupulous notes on all eight hours of both the US Senate Intelligence hearing and the US congressional hearing that took place at the end of February, just a few weeks ago, where leaders of SolarWinds, FireEye and Microsoft all spoke about what happened.  

Senator Marco Rubio:
One of the hallmarks of this operation was the great care that was taken by this adversary to use bespoken infrastructure and tradecraft for each victim.  

Jolie Hales:
That’s Senator Marco Rubio, who serves as vice chair of the Senate Intelligence Committee.  

Ernest de Leon:
For those who haven’t seen that news clip, you should really watch it because there’s an interesting exchange between one of the congresswoman, and I believe it’s the CEO…  

Jolie Hales:
Oh, I know exactly what clip you’re talking about. The congresswoman was Katie Porter, who actually represents my district here in California. And let’s just say that she didn’t go easy on the new SolarWinds CEO.  

Rep. Katie Porter:
Is it true that some servers at your company were secured with this crackerjack password, solarwinds123?  

Sudhakar Ramakrishna:
Congresswoman, I believe that was a password that a intern used on one of his GitHub servers back in 2017, which was reported to our security team, and it was immediately removed.

Katie Porter:
I’ve got a stronger password than solarwinds123 to stop my kids from watching too much YouTube on their iPad. You and your company were supposed to be preventing the Russians from reading Defense Department emails.  

Jolie Hales:
Oh man-

Ernest de Leon:
It was pretty brutal.

Jolie Hales:
That would be a hot seat to be in, I think.

Ernest de Leon:
Even if it’s true that it was an intern, let’s just hypothetically say it was. This is a failure at the organizational level in terms of security, and that’s where it lies, like the individual who did it, yes, they are obviously culpable for doing it, but there are many layers that had to have failed beyond just the one person.

Jolie Hales:
I agree with you. And to be frank, I actually have a lot of sympathy for SolarWinds, because I mean, being hacked these days is so easy. It often just takes one person being duped, through social engineering or whatnot, and then a whole organization can end up paying the price from what I understand. But at the same time, general security measures concerning high stakes passwords should be common sense implementations and an intern really shouldn’t be given that level of control.  

Ernest de Leon:
That’s right. As a matter of fact, I often say, it’s not a matter of if you’re going to be hacked or breached or compromised. It’s a matter of when and how. Right? And in the security world, that’s the assumption we always work on. Our goal is obviously to try to stop that from happening, but our secondary goal is to notice when it does happen and rapidly respond to it.

Jolie Hales:
Well said. And when we left our last episode, I still had a lot of questions about this hack and what it means. So specifically I want to know what kind of information was actually taken if we can know that and what can it be used for, in like a worst case scenario?

Ernest de Leon:
There are a lot of things to unpack here. So let’s start with what information the hackers made off with.

Jolie Hales:
Let’s.

Ernest de Leon:
This matters for many reasons, not the least of which is, what further damage can be done with the stolen data?

Jolie Hales:
Yeah. And that’s probably my number one question at this point.

Ernest de Leon:
Understandably, we’ve mentioned FireEye who had hacking tools stolen in this breach.

Jolie Hales:
Yes, FireEye. And I’ve got to say when I watched hours of those hearings, in my perspective, their CEO, Kevin Mandia, I mean, he might as well have been wearing a cape, because he came across as not only very credible, but I would even go so far as to say heroic for leading this effort that found the hack first, and then for sounding the alarm immediately. I know that probably sounds cheesy, but he clearly knew his stuff during the hearings. And he seemed to authentically have the nation’s best interest in mind from the very beginning. So he definitely won me over and he explained the hack in a way that a lay person could understand.

Kevin Mandia:
Whoever this threat actor is, and we all pretty much know who it is. This has been a multi-decade campaign for them.  

Jolie Hales:
That’s the FireEye CEO/undercover superhero himself, Kevin Mandia.  

Kevin Mandia:
I won’t explain how we found this implant, because there was no magic wand to say, “Where’s the next implant?” When we were compromised, we were set up to do that investigation. It’s what we do. We put almost a hundred people on this investigation. Almost all of them had 10,000 hours there, so to speak, 10,000 hours of doing investigations and we unearthed every clue we could possibly find. And we still didn’t know. So how did the attacker break in? So we had to do extra work and at some point in time after exhausting every investigative lead, the only thing left was the earliest evidence of compromise was a SolarWinds server. And we had to tear it apart. And what I mean by that is, we had to decompile it. Specifically, there was 18,000 files in the update, 3,500 executable files. We had over a million lines of assembly code. For those of you that haven’t looked at assembly, you don’t want to. It’s something that you have to have specialized expertise to review, understand, piece apart. And we found the proverbial needle in the haystack and implant, but how do we get there? Thousands of hours of humans investigating everything else, and that’s one of the reasons I share that, as you wonder why people missed it. This was not the first place you’d look, this was the last place you’d look for an intrusion.

Jolie Hales:
And honestly, I don’t know about you, but I kind of feel like FireEye deserves a lot more praise than they might be getting just for making this discovery, and then immediately alerting the public.  

Ernest de Leon:
If I had to sum it up in like one sentence, FireEye did the right thing. They noticed that they had been hacked. They noticed the types of things that had been taken and they immediately and publicly broadcast that to help mitigate the damage that could have been done. Had they done, what’s typically done in this situation where a company plays it very close to the vest, primarily for legal reasons, but-

Jolie Hales:
PR reasons, a lot of times.

Ernest de Leon:
Right. Legal and PR and they try to keep it to themselves until they’re sure of what happened. And then they go public with some spun statement. In this case, FireEye just came out with the truth right away, put the list of tools that have been compromised and also listed a bunch of mitigations to help stop potential attacks with those tools. So they should be praised in the situation because they absolutely did the right thing.

Kevin Mandia:
We did not have a legal requirement, at least based on the legal advice that I got to disclose at the time that we did. So we did so based on, we’re a security company, we work to a higher order. Yeah, it’s all built on trust and, you got to report.  

Jolie Hales:
Now, when you say hacking tools were stolen, what are you referring to, exactly?  

Ernest de Leon:
So if you want more detailed information into what was actually stolen from FireEye, you can go to their website. They have a blog post that actually covers the entire incident. And it lists the specific tools that were potentially stolen and mitigations, they put in place.  

Kevin Mandia:
Take some of our red teaming tools that we use to assess people’s security programs.  

Ernest de Leon:
Let’s just say, there were many tools that are used in a toolkit for doing all kinds of hacking, to find and probe for known vulnerabilities in software, and these were all taken.  

Jolie Hales:
Okay. So tools that the ethical hackers typically use were taken by the not so ethical hackers.

Ernest de Leon:
That’s right. So Orion is what got them into the FireEye environment. And then they stole these tools.  

Jolie Hales:
And that brings up a really interesting point. We’ve been told that the Orion software compromised 17 or 18,000 organizations, but it’s my guess that once the hackers had access to that many, they probably had to pick and choose who to focus on from there. I’m thinking that maybe not all 18,000 organizations are going to be a value to some nation state, but a cybersecurity company like FireEye or a US government agency. I can see why those would be targets.  

Ernest de Leon:
That’s right, because there’s only so much bandwidth, right? They probably had a list of 18,000 companies they could have gone into and looked for stuff. But when they looked at the list of companies, they targeted specific ones, like you said, US government agencies, FireEye, a very prolific security research company, Microsoft, which handles a lot of enterprise email, enterprise directory services, which include authentication and access. So yes, unless they had an unlimited amount of people to put eyes on all these, they had to focus on the highest value targets first.

Jolie Hales:
Right. So if like a florist in Wyoming uses SolarWinds to manage their IT, I don’t… That is a terrible example, because I don’t think they have a lot of digital flowers.  

Ernest de Leon:
Or a high school in Ohio. Like-

Jolie Hales:
That’s a better example.

Ernest de Leon:
A high school has enough assets that they would use a product from SolarWinds, but they’re not really a high value target, like a government agency or a Fortune 500 company.  

Kevin Mandia:
After stage one, the attackers had a menu to over 17,000 companies that had downloaded the implant. But that doesn’t mean the attacker stole anything from 17,000 companies. The stage two victims are where the attacker decided I want something. And the attackers manually engaged with about a hundred different organizations. In stage two, the attackers did three things. First, steal your keys. They came in through the trap door in the basement that you didn’t know about. They took your keys. And with those keys, they accessed your information. The same way people and employees do.  Second thing they did is they did very specific and focused targeting of documents and emails. And the third thing these attackers did, I put in other category based on the victim, they stole source code or software. And in the case of FireEye, they stole assessment tools that we use to assess security of organizations.  

Ernest de Leon:
So when these hacking tools were taken, the immediate concern was, what will the hackers use these tools for in the future? And the truth is, we don’t know the answer.

Brad Smith:
One is espionage, obviously to obtain information, especially say from US government and other agencies.  

Jolie Hales:
That’s the voice of Brad Smith, President of Microsoft, talking about three suspected purposes of this hack.  

Brad Smith:
Second, to learn more about technology, because obviously technology is the plane on which this organization’s activities take place. That’s why 50% of the victims that we identified are in communications and technology companies. Third, I think there’s an aspect of this that you’d almost put in the context of counter-intelligence. They focus on red team tools, so that they know how to withstand attacks. They look for what a company like Microsoft may be knowing about them, so that they’re able to try to circumvent what we’re going in the future. And that’s true for other tech companies as well.

Ernest de Leon:
If I had to speculate, I would guess that they will use these tools to exploit known vulnerabilities in software and systems that other more lucrative targets have. However, you want to define the word “lucrative.”  

Jolie Hales:
So then what do you mean by known vulnerabilities? This is going to sound completely simplistic, but if a company has a digital security vulnerability, especially if they’re a lucrative organization, shouldn’t they just fix it if it’s known? Or are we talking about companies like the small public water facility we mentioned in the previous episode in Florida that was hacked because maybe they don’t have the budget to upgrade their digital systems.  

Ernest de Leon:
Yeah. So there’s two definitions we look at here for known vulnerabilities, right? Typically, the one that most people are familiar with is the one that you’re referencing here, which is it is known publicly. It is perhaps on a public bug tracker of a software project, something like that. It has what we call a CVE attached to it.  

Brad Smith:
There’s one thing I consistently find today, it’s that many of the public sector computers and information systems software, especially at the state and local level, are not as modern as they should be. Just to give you one example, one department of health at the state level that we’re working with on the distribution of vaccines, we want to help them strengthen their work. And when our consultants looked at the manual for the software program they were using, it was for a company that Microsoft acquired more than 20 years ago. So the software was more than two decades old. So part of what I think we need to do is strengthen CISA. But I think part of what we need to do is really across the country at the state and local level embrace the modernization of our IT infrastructure and in so doing embrace the modernization of our cybersecurity protection.

Ernest de Leon:
But there’s also known vulnerabilities that might be known within the ethical hacking community, or even more specifically known within an organization like FireEye, that they’ve not released to the public, because they are using it for their own penetration testing.  

Jolie Hales:
So then the hackers, once they got into FireEye, had access to the tools that expose these vulnerabilities that they would not have otherwise known about.

Ernest de Leon:
Right. All different kinds of known vulnerabilities, right? Because there could also be the case where customers were running certain types of software that had known vulnerabilities that were public, but the customers themselves are not running aggressive scanning and vulnerability assessments against their own infrastructure. And therefore, while the vulnerability is known to the public, the customer may not know that they are affected by that vulnerability.  

Jolie Hales:
Okay. So these black hat hackers who stole hacking tools from FireEye could now use these tools to further breach other targets. Now, what other targets are we talking about? These are the government agencies or the high value targets that we were talking about earlier?  

Ernest de Leon:
That’s right. It could be government agencies, it could be Fortune 500. And depending on the type of tools that were stolen, the type of vulnerabilities that they want to exploit, it could be anybody. So that’s really the issue here, is that FireEye had a very large wealth of tools that were used for this. So the tools could be used against just about any target. But as you mentioned earlier, they’re going to target very specific high value targets first. And that is primarily because of bandwidth, but also it is not uncommon for an organization to notice when a larger attempt is being done at a breaks like extrication of data from the environment. And at that point, they muster all of their resources and shut it down. So if you’re going to be found out, you want to get the highest valuable data out first-

Jolie Hales:
Before you’re discovered.

Ernest de Leon:
Before you’re discovered.

Jolie Hales:
Make the most of your time. Interesting. Okay.  

Ernest de Leon:
Ironically, had they not tipped their hand with the FireEye breach, where FireEye noticed it and then put out that PSA, the larger SolarWinds breach could still be going on today, compromising many more enterprises and agencies over time. It’s already suspected that the SolarWinds hack was in place for months. So it would have been a very long time before it was noticed outside of the context to FireEye.  

Kevin Mandia:
For us being a stage two, we had firsthand account of what they do. The attackers came in through the SolarWinds implant. And the very first thing they did is went for your keys, your tokens, basically they stole your identity architecture so they could access your networks the same way your people did it. And that’s why this attack was hard to find these attackers from day one. They had a back door, imagine almost a secret door into your house. And the first thing that happens when it come through that secret door is all your keys are right there. They just grab them and now they can get into any locks you have in your house, the same way your people do. And I think during a pandemic where everybody’s working from home, it’s way harder to detect an attack like this, where the only indicator of compromise was just somebody logging in as one of your employees. And there was nothing else far-fetched about that.

Ernest de Leon:
And more than likely, the SolarWinds hack was intended to slowly gain access to as much critical infrastructure as possible before finally launching a larger attack, that would have been nearly impossible to defend against at that point.  

Jolie Hales:
I mean, that’s pretty frightening to think that it was all in preparation potentially for a massive attack. I’m the kind of person who likes to know what the worst case scenario is. I’m one of those people who likes to plan for worst case scenario, but had this breach gone unnoticed, practically speaking, what are some of the worst things that could have been done down the line with everything being as digitally interconnected as it is?  

Ernest de Leon:
That’s actually an excellent question to ask, because the answer to that changes every couple of years, every decade, maybe that goes by because more and more systems are attached to the internet. And a lot of times the command and control systems of critical infrastructure are not updated or not patched correctly. Just like we talked about the water utility down in Florida. So the worst case scenario, there’s several different things that could happen, right? So one of them is essentially stealing state secrets that have to do with defense, right? So if this was a nation state actor, one of the things they’re going to be after is secrets to our military capability, aircraft, sea craft, satellite surveillance, intelligence that we gather. They want to have all of this, so that if there’s ever a confrontation between that nation state and ours, they have the upper hand in terms of the data, they may not have the technology to overcome what we have, but they at least know what our capability is.

Jolie Hales:
Interesting, and they could even make defensive preparations in advance if not offensive.

Ernest de Leon:
Correct. And in some cases with other nation states, they’re so far behind us, this could catch them up on decades worth of research in either weapons, design or whatever the case is, right?

Brad Smith:
As we look at the world, we have espionage threats, we have disinformation threats, and then ultimately we always have the threat we were talking about before of actually damaging a society or a country.

Ernest de Leon:
The more likely scenario is something like what happened in Florida, which is an attack against the actual people of the country, as opposed to the government or the military.  

Rep. Jamer Comer:
You may not have heard about this attack because it hasn’t affected your daily life. You still go home to a warm house overnight, and you still flip all the television at night and watch TV. You still FaceTime with your friends and family, but that’s only because the attackers chose not to disrupt those activities.

Jolie Hales:
That’s Congressman James Comer, of Kentucky, during the congressional hearing.

Rep. Jamer Comer:
Now imagine if an adversary had the ability to take our electric grid offline in the dead of winter or the peak of summer. Now, imagine if this took place, you’re in a national crisis. Imagine if an adversary wanted to toy with our financial markets. Imagine if an adversary had the ability to control the supply chains and manipulate whatever they wanted.

Ernest de Leon:
Think of a scenario where a hacker figures out that the electric grid and a certain portion of the country doesn’t have the right security in place for the various systems that manage the power delivery throughout the grid. And they figure out a way to overload the grid and essentially shut it down. That is something that they would be looking to do, because what it does is it essentially creates chaos, right? If you look into the world of disaster preparedness, right? There’s an entire… FEMA deals with this a lot, but there’re entire think tanks and organizations that deal with things like, how do I prepare for X? How do I prepare for a hurricane? How do I prepare for an earthquake? Some of the scenarios though, are how do I prepare for the electric grid failing? It would create chaos. And that’s really what they’re after in those types of attacks.  So they’re looking for vulnerabilities in critical systems like power, water, and also things like affecting supply chains for food. That’s another area that disaster preparedness, think tanks and whatnot work on. So if there’s a disruption to the supply chain, with respect to food, to where you halted, like what happened in Texas, where the trucks couldn’t move because of the ice on the freeways, you have about 72 hours to recover from that before you get into a situation where there’s no food on the shelves anymore. And then people get to the point where they start resorting to violence, because they don’t have access to food or water or whatever the case is. right? So again, it’s not an attack for the sake of just attacking, it’s an attack for the sake of creating chaos.  

Jolie Hales:
What’s the goal? I mean, this is all very apocalyptic, but what’s the goal in creating chaos? What do they gain? What does the hacker gain from that?  

Ernest de Leon:
It destabilizes governments. That’s what chaos does. At the end of the day, it destabilizes the government.  

Jolie Hales:
I always have this vision in my mind of someone hacking… This is ridiculous, but someone hacking all of the traffic lights and making them turn green at the same time.  

Ernest de Leon:
Well, you must have watched the critically acclaimed, Hackers movie from the 1990s.

Hackers Movie Trailer:
Hackers.

Jolie Hales:
Wait, that happens in the movie?

Ernest de Leon:
That happens in the movie.

Jolie Hales:
I thought I was the first person who came up with this idea. It would be insane, but I don’t even think that’s possible because the traffic light systems are probably not connected.  

Ernest de Leon:
You would think, but as I said this story evolves over time where let’s say 20, 30, 40 years ago, these systems were all independent and controlled in small little grids. A lot of that stuff is connected to networks nowadays, so-

Jolie Hales:
Oh my gosh, this is all so crazy.

Ernest de Leon:
Yeah. And-

Jolie Hales:
I’m going to go live in a cave.

Ernest de Leon:
Right. So jumping back to the tools that were stolen from FireEye, that’s just what was stolen from FireEye. As you know, the hackers gained access into many other companies and networks as a result of the SolarWinds breach. So what else did they steal? I’ll let you guess.

Jolie Hales:
I really have no idea here because I don’t know exactly what information is accessible in this situation. I mean, are we talking personal citizen information like social security numbers and tax info and all that, that we are afraid of being taken or bank fraud?

Kevin Mandia:
This breach to me from what I can observe, and I was a first-hand victim of it, wasn’t about stealing the information of consumers, PII. This was about stealing documents that were relevant to the collection requirements of another nation.

Jolie Hales:
That’s Kevin Mandia, FireEye again, answering my question. So if it isn’t about stealing consumer information, is it more along the lines of proprietary code, like owned by a high profile company or, I mean, I don’t have the slightest clue what information the US government connects to the internet regarding the nuclear weapons we were talking about, or top secret national security. You probably know more about that than I do.

Ernest de Leon:
I do and while I can’t go into a lot of detail on that for obvious reasons, a lot of that information is compartmentalized and not accessible to the generic internet.

Jolie Hales:
Whew, that’s a relief.

Ernest de Leon:
We have some people who know better, but ultimately the most accurate answer to your question is, and I know this is going to sound like a cop-out, but it’s that we don’t know.

Kevin Mandia:
Emails and documents were taken. And quite frankly, the people targeted all that information that was taken. I believe the threat actor is still learning how they can use that information. It’s going to emerge over years and it’s going to take months and months organizations to get their arms around all the possible uses of the stolen documents.

Jolie Hales:
Man, we don’t know a lot of things.  

Ernest de Leon:
And that’s the problem here, right? So we know of the specific names of the companies and agencies they hacked, but most of those entities did not disclose what exactly was accessed.  

Jolie Hales:
In fact, from watching the hearings, Microsoft is one of the few big tech companies that agreed to appear and answer questions while it appears that other large companies declined. And Brad Smith of Microsoft, who came across to me as being pretty credible and authentic, was asked about the potential of other companies being hacked and maybe not disclosing it.

Brad Smith:
Some the largest companies in our industry, that are well known to have been involved in this that still have not spoken publicly about what they know. There’s no indication that they even informed customers. And I’m worried that to some degree, some other companies, some of our competitors even, just didn’t look very hard. If you don’t look, you won’t find, and you’ll go to bed every night being blissfully ignorant, thinking you don’t have a problem when in fact you do.  

Ernest de Leon:
We know they got access to some of Microsoft’s source code. Microsoft says they got access to a small number of repositories. One had to do with Azure, Microsoft’s premier cloud offering.  

Brad Smith:
Our build systems were secure and they were not penetrated in any way that we had no customer data that was touched in any way. And that we found no evidence that any of our services or products were used as a vector of attack to launch an attack against anyone else. What we did find in certain instances was once this intruder was inside a network of, say a federal agency, one of the things that he was able to do, was get access to an account that had what we call elevated privileges like an IT administrator.  It was able to find the password or get the key to get into that account. When he was in that account, we found that that individual had access, say, to the Office 365 email of a portion or all of say, a customer. And so once they were there and then they went into the Office 365 cloud service, and that’s when we identified their presence.

Ernest de Leon:
If the hackers had access to code that specifically dictates who is able to access what resources, what data they can see, whether or not they can download, say that data or extricate it in some other format that can be a problem.  

Jolie Hales:
So, then that means if somebody had those kinds of tools, they could then control who had access to what. They could actually say, “Okay, well, John, over here is now going to access this level of security.” And give themselves basically access. Is that what that means?  

Ernest de Leon:
Right. So what’s going to happen in this case is they may not have access to those systems directly right now, but if they have access to the code base that handles identity and authorization management for those systems, they can look for ways to exploit that code and then come up with a vulnerability, which they use to gain access to the system. So this doesn’t directly give them access. But what it does is it gives them essentially the code base to look for ways to break it.  

Brad Smith:
At this stage, we’ve seen substantial evidence that points to the Russian Foreign Intelligence Agency. And we have found no evidence that leads us anywhere else. At Microsoft, as we worked with customers that had been impacted by this, we stepped back and just analyzed all of the engineering steps that we had seen. And we asked ourselves how many engineers did we believe had worked on this collective effort? And the answer we came to was, at least a thousand. I should say, at least a thousand very skilled, capable engineers. So we haven’t seen this kind of sophistication matched with this kind of scale.  

Jolie Hales:
From supersonic jets to personalized medicine, industry leaders are turning to Rescale to power science and engineering breakthroughs. Rescale is a full stack automation solution for hybrid cloud that helps IT and HPC leaders deliver intelligent computing as a service and enables the enterprise transformation to digital R&D. As a proud sponsor of the Big Compute Podcast. Rescale would especially like to say, thank you to all the scientists and engineers out there who are working to make a difference for all of us. Rescale, intelligent computing for digital R&D. Learn more at rescale.com/bcpodcast.

Ernest de Leon:
Now keep in mind, there were other enterprises also compromised and various data was stolen from them as well. Considering the 18,000 customers were exploited, we could spend months just combing through the data to see what all was stolen, but we don’t have all that time on this podcast episode, so let’s focus on the more serious fallout here. So as we mentioned in the last episode, several federal agencies were hacked as a result of the SolarWinds vulnerability.  

Jolie Hales:
And to me, that’s probably the scariest part of all of this.  

Ernest de Leon:
Right. It would have been very concerning if just one minor agency had been hacked, but that was not the case. Several agencies were hacked. Some of those dealing directly with national security.  

Jolie Hales:
In fact, after our last episode, I decided to do a little more digging into exactly which United States government agencies were compromised. And this is based on articles that I will link to in the episode notes. And it is again, important to mention that this situation is very fluid, it’s still evolving. And this is just what the public has been told so far in this collection of news articles and publications. So first, the Department of Commerce, they had their high ranking officials’ emails breached, and it seems that the hackers were probably spying on their email communication. Also the Department of Defense, as we’ve mentioned, which includes parts of the Pentagon. The good news here is that apparently the SolarWinds Orion software wasn’t particularly popular. So while we don’t publicly know the extent of the breach, there’s optimism from what I’ve read, that it wasn’t incredibly extensive.  The Department of Energy was also breached, which includes the National Nuclear Security Administration, which houses the country’s nuclear weapons stockpile, which sounds really scary. Now the good news here in an ongoing investigation is that it seems that the department of energy attack was isolated to business networks only, and didn’t reach their mission essential type national security functions. So that goes back to what you were talking about with your knowledge of government agencies too, Ernest. And hopefully that’s true.

Ernest de Leon:
We hope.

Jolie Hales:
Yeah. We hope, exactly. Also the Department of Homeland Security. And all I could find was that they had been a part of the breach, but details are not publicly available, which I guess if any agency is going to be tight lipped about it, it makes sense that it would be the Department of Homeland Security.  

Ernest de Leon:
Yeah. There’s two, right? DHS and DoD, are going to be two that would not publicly disclose.  

Jolie Hales:
Yeah. And that makes sense. And the Department of Justice includes the FBI, the ATF, the Drug Enforcement Administration among others. And they reported that around 3% of their Microsoft office email accounts were potentially compromised. But again, some good news here is that they don’t believe any classified systems were impacted. So hopefully that’s the case. And then the State Department, they apparently had their email servers hacked by… What they’re saying, looks like the same Russian state hackers that hit them back in 2014. So this could be the same people.

Jolie Hales:
And next, the Department of Treasury, which manages national finances, which includes the IRS. They actually, from what I read may have been hit relatively hard. And while it appears that tax payer data weren’t breached, the hackers likely stole encryption keys from government servers, which correct me if I’m wrong, but I would think that having access to an encryption key would make it possible for a hacker to read and access information that was originally encrypted for the sake of security, right?  

Ernest de Leon:
That’s right. But in this case, joke’s on them. The country’s actually broke. So actually more than broke it’s in the hole by like $20 trillion. So they’re not getting much out of that hack.  

Jolie Hales:
And then the final US government agency I looked into was the National Institutes of Health. And this one was really interesting to me, where it’s thought that Russia’s Foreign Intelligence Service was actually after the coronavirus vaccine research. And that’s fascinating because if you remember, Russia started vaccinating people in their country surprisingly early, I’m just saying.

Ernest de Leon:
That’s right. As matter of fact, when they announced that, I think they called the vaccine Sputnik, if I’m not mistaken.

Jolie Hales:
Oh, did they?

Ernest de Leon:
Yeah. Which is a jab at us from a little bit of a cold war perspective, but the Twitter-verse was a fire when that happened. And I remember seeing one like prominent, let’s just say scientist, saying, “Good luck with that.”  

Jolie Hales:
Interesting. And then in addition to these major agencies, apparently there were local and state governments that were also part of the breach, as well as critical infrastructure entities and private sector organizations. We’ve talked about a few of those. So it’s definitely wide reaching. It’s even being called the biggest US hack in history. Not that we’ve had the internet for that long, but still it’s pretty menacing.  

Brad Smith:
It’s a little bit like a burglar who wants to break into a single apartment, but manages to turn off the alarm system for every home and every building in the entire city. Everybody’s safety is put at risk. And that is what we’re grappling with here.

Ernest de Leon:
And as someone who worked directly within the Fed Gov space in the past, I can tell you that when something like this happens, the assumption is that all networks and all systems are potentially breached, right? And I want to emphasize that word — potentially. One of the things that concerns me the most in my role as a security person is when a breach happens and the company immediately comes out the same day saying no user data was stolen or nothing was stolen, because there’s no way you can know that in that amount of time, right? So in this case, CISA issued an emergency directive to mitigate the SolarWinds hack.

Jolie Hales:
Interesting. And for our listeners, CISA is-

Ernest de Leon:
CISA is the Cybersecurity and Infrastructure Security Agency, dealing with all things US cybersecurity. So what that means is, anybody who’s doing business with the federal government, including the federal government agencies themselves, are bound by this agency. Anything that they put out in terms of cybersecurity, both awareness and preparations, and mitigation should a breach or something of that type happen. And the directive issued by CISA is pretty lengthy, but the TLDR is that pretty extensive measures now need to be taken to secure affected systems and networks. When I read the directive, it was quite shocking because it essentially said that.

Cyberbot:
Any system or network where the SolarWinds, Orion product was used is considered to be compromised.  

Jolie Hales:
So even if there wasn’t direct proof that a customer using Orion was hacked directly, we have to err on the side of caution and assume that they were because sometimes these fingerprints aren’t left behind.  

Ernest de Leon:
So yeah, in addition to considering all Orion customers compromised.

Cyberbot:
All of the command and control infrastructure for SolarWinds is to be immediately disconnected or powered down.

Ernest de Leon:
Furthermore, we’re to-

Cyberbot:
Treat all hosts monitored by SolarWinds Orion as compromised by threat actors and assume that further persistence mechanisms have been deployed.

Ernest de Leon:
What this means is that the hackers would have deployed measures to persist the compromise past attempts to remove or contain the hack.  

Kevin Mandia:
When the implant in the SolarWinds software ran, one of the first things it did 11 days after it installed, mind you, it slept for the first 11 days, is it looked at the system it was running on and it looked for common safeguards like Windows Defender, like CrowdStrike, like FireEye’s endpoint and it shut them off. And again, the implant ran at system level, it had the permissions to do whatever it needed to do. So it just said, “What security is running? Kill it.” And that’s why we couldn’t detect it in the first stage of the attack.  

Jolie Hales:
So in other words, we need to assume the worst case scenario and act accordingly, because these hackers probably put something in place that would protect them, even if their hack was discovered.  

Ernest de Leon:
Exactly. So imagine this right, you get a virus on your computer, your antivirus sees it and it goes and removes it. The next time your machine reboots, the virus is back, right? That’s a persistence mechanism. So what it does is it conceals itself somewhere. And in the event that it’s removed, it’s able to reinsert itself.  

Jolie Hales:
Oh man, we do have to assume the worst case scenario.  

Ernest de Leon:
Absolutely. And in addition, in the CISA directive.  

Cyberbot:
All hosts monitored by the SolarWinds Orion product, are to be fully rebuilt using trusted sources.

Jolie Hales:
Oh, wow.

Ernest de Leon:
Now, this is insane, because any enterprise or business using SolarWinds Orion in their infrastructure likely had most, if not all systems connected to it, which meant system administrators would essentially have to rebuild their entire server infrastructure from the ground up.

Kevin Mandia:
There’s no doubt in my mind, this was planned. It was an operation. There was a lot of people involved. And the question really is where’s the next one? And when are we going to find it?  

Ernest de Leon:
Additionally–

Cyberbot:
All credentials used by or stored in SolarWinds software should be considered compromised.  

Ernest de Leon:
Now, you can imagine how much work this caused already for short-staffed and overloaded system administration staff.  

Jolie Hales:
Mm-hmm. I imagine a lot of late nights and long hours.  

Ernest de Leon:
Oh yeah. I’ve been there. I’ve done that many times. While this might seem like an extreme response. When you are dealing with matters of national security, you have to assume the worst in this kind of scenario.

Jolie Hales:
Now I’m curious, obviously with Intel and Microsoft being effected among probably a number of others also dealing in high-performance computing, how does this Orion hack affect the HPC industry in general?  

Ernest de Leon:
Yeah. I want to call back to a former podcast episode with one of our undercover superheroes, Dan Stanzione, executive director of the Texas Advanced Computing Center or the TACC, who we interviewed a couple of episodes go. I really love how he referred to the on-prem supercomputers at the TACC.

Dan Stanzione:
In many ways, I think our overarching stance on that is just simply that we are the cloud, but we are a very special kind of cloud.

Ernest de Leon:
In the HPC world, we still use many of the same types of technology that traditional on-premise enterprise computing or cloud use mainly compute, network and storage, the big three. The fundamental principles of cybersecurity are the same across technology stacks. At the end of the day, we’re still working tirelessly to secure compute resources, network resources, and storage resources.

Jolie Hales:
Whether it’s HPC in your own data center or HPC access via commercial cloud services, the cybersecurity approach is virtually the same.  

Ernest de Leon:
Right. Mostly the same. There are some key areas where we have security opportunities and HPC that are not as prevalent in the non-HPC world. One of those is that HPC demands certain performance levels across many different servers simultaneously and typical security measures, we might take an enterprise computing for example, end-to-end encryption everywhere, may not make sense in the HPC context. Sure, there are ways to offload certain things to hardware when you’re talking about standard ethernet-based networking, but when you’re a working with InfiniBand, you’re in a whole different ball game. You’re often trading a more hardened security posture enforced in the kernel for faster connection that is out of band from traditional ethernet networking. Many of the existing tools that we use in the cybersecurity space were not designed to engage with this type of traffic, and the few tools that do have a significant performance penalty.  

Jolie Hales:
Does that make HPC harder to hit by a black hat hacker or harder to protect?  

Ernest de Leon:
I would say it’s a wash, in some ways it is harder to protect, but also some of these systems, because they’re out of band, they’re out of reach of somebody who doesn’t have, either hands-on or-

Jolie Hales:
Like direct access.

Ernest de Leon:
…direct access to the out-of-band method that we’re doing it. So, this is just one example. My larger point is that while we have some significant differences between HPC and traditional enterprise computing, the majority of the technology is similar. And at the end of the day, any malicious actor seeking to extract data will have to do so via traditional networking channels. That makes it easier to monitor, but at the same time, some attacks do not have the explicit end goal of data extraction. Some attacks just want to lock up resources and cause a denial-of-service. Some attacks seek to encrypt the critical data and hold it for ransom.

Jolie Hales:
I know a little bit about that one, as we heard on our last episode.  

Ernest de Leon:
Yes, you do. Some attacks seek to exploit hardware firmware, vulnerabilities and cause hardware failure like Stuxnet. In the HPC space, we have had several malware waves that attempt to hijack HPC clusters and mine cryptocurrency.  

Jolie Hales:
Oh, that’s so interesting. And it makes so much sense. You see people hoarding GPUs to mine cryptocurrency these days on their own personal computers, but if they could just tap into a supercomputer that’s maybe full of GPUs, dang, I imagine you could make a lot more money much faster, and of course be a much bigger jerk.  

Ernest de Leon:
That’s right. And securing these HPC systems is very similar to how we secure traditional enterprise computing systems with a caveat that we have to do so with little to know performance impact to the infrastructure.  

Jolie Hales:
You mentioned the cloud, which I wanted to point out was also mentioned quite a few times in the senate and congressional hearings. I especially found it interesting because I’ve heard how some organizations have been hesitant to move to the cloud in the past and even in the present in part, because maybe they’re not convinced that it’s secure enough. By listening to these hearings, both Kevin Mandia and Brad Smith advocated for the opposite, and did so stronger than I expected.  

Brad Smith:
All 60 of the Microsoft customers who were attacked had their networks penetrated on premise, meaning in their server room, in their building, it was not in our cloud services. It’s like, if someone broke into your house, but not my house, I would not know until you told me, or in this case, what they did was they went into your house. They found the keys, the passwords, so that they could go into the service in the cloud. Once they got that, once they stole your keys, once they entered our cloud service, we saw them. And then we called you and we said, “Did you know that they’re in your house? Did you know that they’ve stolen your keys?”  

Jolie Hales:
He really makes the case that Microsoft’s cloud, Azure, wasn’t the problem here, but more of a potential solution. And if we’re hesitant to take the word of a president of a company that offers cloud services, Kevin Mandia of FireEye, actually also didn’t disagree.

Kevin Mandia:
After 30 years in IT security, I believe it will be easier to secure the cloud than the last 30 years of us trying to secure everybody’s home offices and secure inside four different walls all over the place.

Jolie Hales:
And as congress continued to ask about cloud, Kevin Mandia got more specific.

Kevin Mandia:
Migration cloud’s going to happen whether we want it to or not. It’s rare in history where something costs less and is better. Cloud is actually costing less and is better. For example, if I wanted a server set up at FireEye, I could ask an IT staff to do it, or I can go to an infrastructure as a service provider and get it in five seconds. So the cloud is coming and then you add the pandemic to it, and the work from home, all the major enterprises, all the major organizations are going to the cloud. The upside is, it cuts both ways, but you should get better visibility and better controls in the cloud. And the reason why you’re putting all your decentralized IP and value into one place, it’s easier to monitor it, easier to safeguard it. You don’t have distributed security controls at that point. I think we’re in the middle of the cloud migration, but over time, what we will see, is organizations recognizing at least the infrastructure portion of a cloud will be more secure because these companies have to secure it. Meaning the providers have to secure it.

Jolie Hales:
As we’ve been talking about this, I can’t help but find it a little bit painfully ironic that this digital virus type malware infected so many computers over the exact same timeframe that a human virus, the coronavirus that you may have heard of was spreading over the globe. It seems like 2020 just wasn’t a good year for health for people or machine.  

Ernest de Leon:
That’s right. And one of the things I will remind our listeners here, and I say this all the time, I may sound like a broken record, but hackers often use times or situations where humans are under duress to do what they need to do. For example, at the very beginning of the pandemic, when panic was very high and curves had spiked up really high, there was a surge in social engineering type hacking attempts across the board because they knew they could take advantage of humans already being in a state of duress globally this time, not just, in a certain region.  

Jolie Hales:
That’s interesting because everybody’s guard’s down, because they’re so focused on stress in another area.  

Ernest de Leon:
That’s right. And people, if they’re working from home, they’re increasingly isolated from the rest of their coworkers. Whereas if they had been in an office and something suspicious happened, they could just walk over to the security department and be like, “Hey, this suspicious thing happened. Is this normal?” And they could have quashed it right there. When someone’s working from a home, they now have to either draft an email, go on Slack or something, wait for an asynchronous response. It just turns into a longer process that can be exploited.

Kevin Mandia:
We may never know the full range and extent of damage. And we may never know the full range and extent as to how this stolen information is benefiting an adversary.  

Ernest de Leon:
At the end of the day, it’s up to us to do our due diligence here and perform the correct forensic analysis on all of these systems and ensure that any potentially compromised systems are rebuilt from the ground up. Now that’s both time consuming and expensive, but we can’t just hope our systems are ineffective, right? We have to proactively go after this and prove to ourselves and our customers or whoever the other stakeholders are, that those systems are in fact, unaffected, uncompromised, and functioning as they should.

Jolie Hales:
That makes sense when the stakes are this high.

Ernest de Leon:
That’s right. And that’s where we’re going to leave it for this episode.  

Jolie Hales:
Well, wait, I still have some questions. So if you don’t mind, so we have guidance on what to do to eradicate the effects of this particular hack. But who’s to say that it couldn’t happen again, or if something isn’t eradicated in a certain place that it spreads out again, right? Like these companies that have been hacked, if they don’t get the SolarWinds Orion vaccine, right? What if it could still spread? I guarantee that there’s probably some so-called intern out there just waiting to accidentally launch another cyber crisis. So is there a way to really prevent this thing from happening? Have we learned of maybe a new technology or something that can be put into place? I know that’s pretty vague, but-  

Ernest de Leon:
That’s a great question, and one we will address in the next episode.

Jolie Hales:
Oh, we’re making this a three part series.

Ernest de Leon:
It looks like it.  

Jolie Hales:
Well then in the meantime, I’ll avoid clicking email links to doordash.lol. That was a phishing email that came in just a few days ago. And I did not click.

Ernest de Leon:
Yes, please don’t. And for our listeners, if you’d like to help spread the word of the Big Compute Podcast, like a carefully crafted virus, you could leave us a five star review wherever you get your podcasts.  

Jolie Hales:
A good virus. Is there a good virus out there? Oh, and apparently at the end of our last episode, the “please sir, may have some more” was from Oliver Twist, not Christmas Carol. My husband fact-checked me on that one.

Ernest de Leon:
Well, I’m glad somebody knew because I certainly didn’t.

Jolie Hales:
Oh, well, thanks to everyone for supporting the Big Compute Podcast and stay safe out there.

Ernest de Leon:
Right. Make sure you use multi-factor authentication and we’ll catch you in the next episode.