The SolarWinds Hack: What Happened?

Ernest de Leon:
FireEye, contrary to popular belief, is not the same as Pink Eye. FireEye…

Jolie Hales:
Do people think that?

Ernest de Leon:
No, but your eye is red and it’s burning, so I figured it would… Anyway.

Jolie Hales:
Hi, everyone. I’m Jolie Hales.

Ernest de Leon:
And I’m Ernest de Leon.

Jolie Hales:
And welcome to the Big Compute Podcast. Here we celebrate innovation in a world of virtually unlimited compute, and we do it one important story at a time. We talk about the stories behind scientists and engineers who are embracing the power of high-performance computing to better the lives of all of us.

Ernest de Leon:
From the products we use every day to the technology of tomorrow, high performance computing plays a direct role in making it all happen, whether people know it or not. So, Jolie.

Jolie Hales:
Yeah, what’s up?

Ernest de Leon:
Have you ever been digitally ripped off?

Jolie Hales:
Like hacked?

Ernest de Leon:
Hacked.

Jolie Hales:
I have. Can I tell you a story?

Ernest de Leon:
Yes.

Jolie Hales:
Okay. Okay. A few years ago, I was doing video production for a company in Southern California, and I was working on a computer in their office building. And at one point I wanted to, I think it was, I wanted to add an image to a video that I was editing, so I did what I always did, right? As you can imagine, I navigated to a specific video production network drive, went into the assets folder, and then I double clicked an image so that I could look at it before importing it. But unlike every other time I’ve done this, instead of the image opening, I got this error message saying that the file format was unsupported and could not be read.

Ernest de Leon:
Oh, this sounds great.

Jolie Hales:
Yeah. It gets even better. The file could not be read, which was super weird since I had literally used this image in the past, and I couldn’t imagine what would have corrupted it. And it was a network drive, so it wasn’t like it was this physical hard drive corruption thing. I was curious and a bit suspicious, so I tried to open a couple other files in that same folder. And again, I was given the same error message and the files just wouldn’t open. And that was really weird because I had just opened a few of those image files like literally an hour ago and they had worked just fine. And now all of a sudden, it was almost as though they had become corrupted, but I did not know why.

Ernest de Leon:
That sounds terrible, especially considering it was on a network drive.

Jolie Hales:
Yeah, exactly. And since I’m a filmmaker, I’ve unfortunately lost a lot of work over the years because hard drives have been faulty or the computer is crashed and they weren’t backed up properly and all of that. When this happens or something like this happens, I get kind of concerned. In this case, I was determined to get to the bottom of what was going on with these files so that I could then make a plan of what to do about it, because any filmmaker out there knows losing creative work is the worst. I backed out of the assets folder, and then I took a look at the main drive itself in Windows Explorer. And I scrolled and down at the bottom, sitting underneath all the folders, were these three files that I did not recognize. And they all had the same like gobbledygook kind of file name, but they had different file extensions. And as the hyper detail oriented, organized control freak that I am, I knew that I hadn’t put any of those files there, and I was pretty sure no one on my team had done it either. And I noticed that one of these three files was a plain text file. And with the understanding that plain text files can’t execute viruses, I ran it through a quick virus check to be safe anyway, because I’m obviously not a cybersecurity expert, but try to get my ducks in a row. And when it came back clean, I then opened that text file. And there staring me in the face were the words, “Oops, your files have been encrypted.” And then there were these instructions on how to pay a bunch of Bitcoin to get the files restored.

Ernest de Leon:
A beautiful ransomware attack. So let me ask you this, do you remember what the number of Bitcoin was to get them restored?

Jolie Hales:
I think it was $500 or something. 500 American dollars is what it translated to. I don’t remember exactly. It wasn’t an insane amount. It was almost like this was targeted at an individual computer, not a corporate network drive.

Ernest de Leon:
Yes. And let me tell our users right now, never ever pay for this stuff. It doesn’t matter what the consequences are of it. Do not pay for these things. All it does is encourage them to continue doing this.

Jolie Hales:
Exactly. In this text file, I see the words written in there for what I recognize at the time to be a ransomware attack, but I wasn’t supposed to notice that text file in order for the attack to work. That text file was actually supposed to help generate the graphical pop-up message that would appear after all the files on the drive had finished being encrypted by the virus. And encrypting every file on a two terabyte network drive takes a lot of time. I imagine it probably takes hours. So it hit me at that moment that the ransomware encryption must’ve been happening right then. So obviously, yikes. That put my button to gear. And in order to verify how much of the drive was affected thus far, I quickly clicked around and checked other familiar files on the drive. And it soon became clear to me that the encryption must’ve been happening in alphabetical order by folder and file name, which is why the assets folder, which I just happened to be working in at the time this encryption started, was the first to contain corrupted files, right? Because if you go back to grade school, assets obviously starts with A. And then I clicked around a little bit more and I realized that the files inside the assets folder toward the end of the alphabet were still openable at the time, which meant that the encryption must’ve just barely begun and had so far apparently only affected a few dozen files out of thousands upon thousands on this drive. And at the time, ransomware attacks were somewhat new. They were just becoming popularized. But I had happened to have heard about them on one of my favorite podcasts, which some of our listeners may know, Radiolab, just a few days before this happened. And I’ll include a link to that awesome episode on bigcompute.org, by the way. But because of listening to that podcast, the subject of ransomware was fresh in my mind and that’s why I recognized what was happening so quickly.

Ernest de Leon:
So at that moment, did the song come in your head, “Who you going to call?”

Jolie Hales:
No, but it probably should have. And it’s funny because all of this happened so fast, right? I was able to draw the conclusion within probably 60 seconds just by doing some quick clicks. And obviously I am not a cybersecurity expert. And so yes, I needed to call for help at that moment. I had no idea, let’s be honest, how to stop this encryption process at all. And I don’t know if this helped at all, but I disconnected my computer from the network just through the physical cable, although I’m pretty convinced the infection came from some other machine in the building. And then I called up our in-house IT team, and then I was literally like frantically yelling into the phone. I can just imagine their faces as they’re hearing me say, “Get over here now! Russian hackers are encrypting all the files on my drive in hopes of getting a Bitcoin ransom.” And they thought I was joking and they’re all making fun of me, because I guess I’m known for joking around for some weird reason, but eventually I finally convinced them I wasn’t kidding. They came over to my office. They worked their magic. They isolated the virus, and they stopped the encryption.

Ernest de Leon:
So what happened to all the files that had already been encrypted?

Jolie Hales:
Well, see, that’s the thing. Because I was lucky enough to catch the virus early, it had only encrypted the files in folders that started with A. And since that network drive had been backed up 36 hours prior, all of the files, except three Photoshop files, were able to be restored. And I was bummed because I had done a lot of creative work on those particular files, but then, and this is the kicker, the next time I opened up Photoshop, recovered versions of those three exact files popped up because apparently I had had them open on my machine when we abruptly shut it down.

Jolie Hales:
So in the end, I legitimately lost zero files and zero Bitcoin, which I considered to be a complete miracle.

Ernest de Leon:
And let’s be honest here, since it never made it to the C folder, none of the cat pictures were taken.

Jolie Hales:
You know, I’m more of a dog person, so the D folder would be more at risk on my personal computer, but it didn’t reach the D folder either. So we’re lucky.

Ernest de Leon:
Yeah. And this is the thing with viruses or hacks in general, right? The person perpetrating them obviously has a goal in mind. Who knows what it is? Sometimes it’s something corny like money. In this case, Bitcoin. Other times it’s much more nefarious than that. But at the end of the day, these hacks can cause a lot of trouble for a lot of people.

Jolie Hales:
Oh, it’s so true. I mean, even if the ransomware attack that hit my drive had worked, the consequences would have been minimal when compared to what can happen if like larger or more important systems were hacked. And when I start to think about what hacks really have the potential to do, it’s kind of scary. I mean, it feels like cyberspace is kind of the new war grounds because everything is so digitally interconnected. There’s a lot of opportunity to really kind of mess things up.

Ernest de Leon:
That’s right.

Jolie Hales:
I mean, just a few days ago, do you remember that public water utility hack in Florida that was caught?

Ernest de Leon:
I do.

News Clip:
An investigation underway after a hacker tried to poison a Tampa Bay area water system. The bad actor increased the amount of sodium hydroxide or lye in the water supply from 100 parts per million to more than 11,000. Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners. Thankfully, a plant operator noticed someone on the system making the potentially harmful change in chemicals and reverted the levels before any water was contaminated.

Jolie Hales:
Which I imagine could have been incredibly dangerous had the supervisor not been paying attention and reversed it. Thank goodness. And people may not know this, but you, Ernest, are a cybersecurity expert. I mean, that’s basically your job when you’re not recording podcasts.

Ernest de Leon:
Yeah. I hesitate to use this word expert. I think anybody in my field would, but I’m a practitioner of cybersecurity on a regular basis. Yes.

Jolie Hales:
So yeah, you know your stuff when it comes to cyber security. And I thought it was crazy because you had even told me a few days before the water facility hack that you suspected that something exactly like that would happen someday at a public water facility.

Ernest de Leon:
That’s right, and that’s mainly because very early in my career, I worked for a public water utility as well. And one of the issues we had was that many of the pumps that are being used that pump water essentially into the water towers were managed and/or run with what we call SCADA systems. And those systems at this point, even back then, were very archaic, often ran on very old versions of Windows. Think Windows XP, Windows 3.1. Things that are long since out of support by Microsoft, have no security patches being made for them, and they should never be connected to the internet. However, a lot of companies are connecting these things to networks for essentially convenience reasons. And that is a massive, massive security vulnerability just waiting to be exploited.

Jolie Hales:
That’s so interesting. And you would think that anything that has to do with our public, that that would be at the forefront of making sure that it wasn’t hackable, like a public water facility. At least that’s what a naïve person like me would think. When you told me that you thought that somebody would try hacking into a public water facility system, and then literally a few days later it actually happened, that was enough for me to go out and immediately dropped like 200 bucks on a 55-gallon water storage tank. Though it is still sitting in my garage and it doesn’t have any water in it, because it’s a whole to-do to prepare the water to put in the tank. So it’s not really going to do me any good.

Ernest de Leon:
Yeah, I agree. It’s these aging infrastructures, right? You often hear people talk about our aging infrastructure with reference to bridges that are very old and starting to fall apart or things like that. But we also have very old computing infrastructure or management infrastructure for some of these critical systems like power and water.

Jolie Hales:
Why? Is it just because they’re government run or government assisted and government doesn’t update very quickly?

Ernest de Leon:
Well, that’s actually an interesting question to ask, because in some cases they are and in some cases they are not. I think what it comes down to at the end of the day is public utilities in general are not working on high margins. They have very tight budgets.

Jolie Hales:
I got you. So updating the software is an expense that doesn’t necessarily make it to the top of the list.

Ernest de Leon:
That’s right, especially when you’re dealing with priorities, right? As an example, at the water utility I worked at, we often had just call them minor disasters where a water main would break under a street or something like that, and these things routinely happened because the infrastructure was old. A bunch of those add up in a year and you’ve essentially used up all of your budget. And it’s not like you can just make money up, right?

Ernest de Leon:
You either have to raise rates to the rate payers, which they’re not going to be happy with, or you try to use some other financing tool to get it done. Yeah, at the end of the day, you’re right. It’s that the cybersecurity aspect of a lot of these things are lower on the totem pole than they probably should be.

Jolie Hales:
That’s very interesting. Very eye opening too. I mean, it’s crazy to see how much cybersecurity threats really are real even to our public life more and more, right? We saw the giant Twitter hack that was done by some kid in Florida, right?

Ernest de Leon:
Yup.

News Clip:
Several high profile Twitter accounts appear to have been hacked. Accounts include Jeff Bezos, Bill Gates, Elon Musk, Uber, and Apple. “Due to COVID-19, we were giving back over $10 million in Bitcoin. All payments send to our address below will be sent back doubled.” They then post a Bitcoin address and say that, “This is only going to be going on for the next 30 minutes.” NBC News reporting that over 250 transactions have already been sent.

Jolie Hales:
If that had been a very organized effort with some really sinister intentions, that could have been something really bad and that’s just Twitter.

Ernest de Leon:
Yes. And one of the things that we’ll point out is that there are many areas that are prone to attack or vulnerable in terms of cybersecurity, but the weakest link in any of these is always the human being. And there’s a specific, if you want to call it, area of study within cybersecurity that has to do with what we call social engineering. And that is the art of conning or-

Jolie Hales:
Tricking somebody.

Ernest de Leon:
Right. Tricking somebody into giving you elevated access to a system that you should not have access to.

Jolie Hales:
Like even with the Twitter hacks, I remember it was incredibly interesting to learn that they literally made phone calls in order to get verbal information transferred to them that they then used for the hack. I had never considered the fact that it was like person-to-person verbal communication in some instances when it comes to hacking. I thought everything had to do with behind the scenes code, but you’re right, it’s that human element.

Ernest de Leon:
And that’s true. You bring up the Twitter hack and there have been quite a few high profile ones lately and some of them have actually hit the news. Jolie, tell me, what do you know about the SolarWinds hack?

Jolie Hales:
Oh, SolarWinds. I don’t know very much about it. I know that there was a hack, and I know that it was apparently humongous, like absolutely massive. And it was relatively recently discovered I want to say a couple months ago, but I really don’t know much more than that and I’ve always been curious.

Ernest de Leon:
Yeah, it’s a pretty big hack, and there’s a lot to cover. It’s going to take a few episodes. There’s no way to cram it into one.

Jolie Hales:
Totally fine. Worth it.

Ernest de Leon:
It is. So many moons ago, I mentioned this earlier, I worked at a public water utility company.

Jolie Hales:
Just like the one in Florida that was hacked.

Ernest de Leon:
That’s right. And when you work in security in these types of areas, you know where your vulnerable points are. This is kind of why when we talk about the one in Florida, when I see certain patterns happening, it immediately brings to mind, “Ah, the SCADA systems.”

Jolie Hales:
Well, you definitely predicted it, to the point where I was like, “Man, if I didn’t know earnest really well, I would have thought he did it.”

Ernest de Leon:
Yeah. So at this public utility company, we used SolarWinds to manage our static IP addresses. I don’t remember the name of the product. This is a very long time ago, right, before all the acquisitions and new products that comprise the portfolio of products that SolarWinds offers today.

Jolie Hales:
So when you say SolarWinds, just want to back up for anybody who isn’t really familiar, it sounds like the SolarWinds company is a software company.

Ernest de Leon:
Sure. SolarWinds basically creates it infrastructure management software. They’re based out of Austin, Texas, and they have a few thousand employees.

Jolie Hales:
Okay. So not like a solar company, not a wind energy company, and not a company that has anything to do with streams of charged particles coming from the sun. SolarWinds is an IT software company. Got it.

Ernest de Leon:
Correct. Which by the way, you and our listeners know I’m dying to talk about an actual solar wind and why that probably wasn’t the best name for a company, but I’m not going to do that here. We’ll save that for one of our future space episodes. Anyway, when I was working for the public water utility company, we use this basic product from SolarWinds to manage IP addresses. It had the ability to scan a network and tell us which static IP addresses were in use at any given time and what the host names were for the machines at those IPs. That’s about all it did. It was a very simple product that really didn’t do much that Microsoft domain services wouldn’t eventually use SERP entirely in the interim.

Jolie Hales:
Okay, interesting. So you’ve had some hands-on experience with SolarWinds in its earlier days.

Ernest de Leon:
Right. Again, it’s a very simple product when they were probably a very small company. Fast forward to December 2020 when I was working in high performance computing cybersecurity, right? We in the cybersecurity industry suddenly had a bomb shell dropped on us. We had long expected something of this scale to happen, but we had no idea it would be so soon. And it all started with FireEye.

Jolie Hales:
Wait, what’s FireEye?

Ernest de Leon:
FireEye is actually a pretty large and pretty relevant cybersecurity company. They basically detect and prevent major cyber attacks for companies. For instance, they’ve been called in to investigate a lot of the high profile attacks we’ve heard of like with Target and Sony Pictures.

Jolie Hales:
You mean like when North Korea retaliated after Sony made a movie they didn’t like?

News Clip:
The North Korean regime has called the movie terrorism.

Ernest de Leon:
So on December 8th, just a few months ago, FireEye announced that they had been hacked.

Jolie Hales:
Uh-oh.

Ernest de Leon:
Yeah. And many of the tools they use for white hat hacking had been stolen. And in case you don’t know what white hat hacking is, it’s a term used for hackers who basically choose to use their hacking powers for good rather than evil. A lot of times they’re security specialists who attempt to find security holes by hacking the system.

Jolie Hales:
So white hat hackers are like ethical hackers?

Ernest de Leon:
Exactly. In the industry, we kind of have three main groups, if you will. We have white hat hackers, black hat hackers, and gray hat hackers, and it’s exactly what it sounds like. The white hats are the ethical hackers doing it for good, trying to secure things. The black hat hackers are doing it for nefarious purposes, trying to steal things sometimes just to create chaos. A lot of times they don’t even want money. They just want to sow discord.

Jolie Hales:
They just want to be jerks?

Ernest de Leon:
Yes. And the gray hats are in the middle. They waver from one side to the other, depending on however they’re feeling that day.

Jolie Hales:
Ha, I like the idea of the white hat hackers. I feel like they’re definitely undercover superheroes.

Ernest de Leon:
Now, white hat hackers use typically the same hacking methods and tools as those who hack for sinister reasons, which we talked about a second ago, we call them black hat hackers, but white hat hackers typically have permission from the system owner first, which makes the process legal.

Jolie Hales:
Oh, okay.

Ernest de Leon:
These ethical hackers test the possibilities of hacking company systems and basically perform vulnerability assessments that they then turn over to those companies to help the companies secure their systems better. And in other cases, big companies like Apple, Google, Microsoft have what they call bug bounties.

Jolie Hales:
Oh, I’ve heard of this.

Ernest de Leon:
These hackers will often find vulnerabilities and then submit them into the Bug Bounty Program. And if it gets accepted as a valid vulnerability, they get paid a certain amount of money based on the type of vulnerability it is.

Jolie Hales:
That’s brilliant. It really encourages white hat hacking.

Ernest de Leon:
Yes. It tries to take what are traditionally black hat hackers or gray hat and turn them into white hat by giving them money directly as opposed to-

Jolie Hales:
Through some sinister method.

Ernest de Leon:
Right. Because typically the black hat hackers are not doing it for themselves.

Jolie Hales:
Really?

Ernest de Leon:
Not all of them, right? But typically they’re doing it and then trying to sell whatever hacks they come up with to nation states or whoever’s paying the highest dollar, to be honest, for that vulnerability.

Jolie Hales:
Wow, that’s interesting. So then FireEye is a cybersecurity company that basically employs a fleet of white hat hackers?

Ernest de Leon:
Yes, that’s correct.

Jolie Hales:
Just to make sure I understand, the black hat hackers are what we normally think of when we think of that typical computer hacker. On the internet, they’re always depicted in pictures as being these faceless dudes in black hoodies hunched over a computer in a dark room.

Ernest de Leon:
Yes, that’s absolutely right. As a matter of fact, I’m going to interject something here. If our listeners have never watched the movie Hackers from the 1990s, the one that had Angelina Jolie in it, you have to see this movie. It is another one of those that is so bad, it’s great. It’s great.

Jolie Hales:
A hacking movie made in the 1990s, I mean, how well could that hold up?

Trailer:
Hidden beneath the world we know is the world they inhabit. Tate? Yeah, mom? What are you doing? I’m taking over a TV network. Finish up, honey, and get to sleep. They’re hackers. But this time. Come here look at this. It’s some kind of virus. Unless $5 million is transferred to the following account, I will capsize five oil tankers. They just hacked no wrong guy. Game’s over.

Ernest de Leon:
It’s amazing. If you’re a fan of bad movies that are so bad, they’re good.

Trailer:
Hackers.

Jolie Hales:
So back to the story, so it’s December 2020 and this cybersecurity company FireEye announces that they’ve just discovered that they’ve been hacked.

Ernest de Leon:
Right. So they essentially sound the alarm and they start digging into the details of the hack, how someone could have breached their systems being that they’re super vigilant and always looking at this kind of stuff.

Jolie Hales:
Yeah, I would imagine.

Ernest de Leon:
Yeah. And what kind of data they had access and for how long?

News Clip:
Big news in the tech world this week as FireEye, a top cybersecurity firm based out of Silicon Valley, announced that hackers made off with tools that the company says can be used to mount future attacks.

Ernest de Leon:
At that time, those of us in the cybersecurity industry thought FireEye was the sole target of the attack. But we also know that anytime the tools of the trade are stolen, meaning the tools that are used to hack and/or information or data about known vulnerabilities that are used for hack, anytime that stuff is stolen, we have to assume that those will be used to hack other entities in the immediate future.

Jolie Hales:
So in other words, black hat hackers usually hack one system in order to access another system and so forth. You have to remain consistently suspicious.

Ernest de Leon:
Yes, that is one of the things they do, right? In a large environment, they’ll have to hack one system to get access and elevated access. And then from there, they look for other systems. However, in some cases, they go after whatever system they want to start with. In that case, they only have to get one, but that’s not usually how it happens. It’s usually a progressive thing where they find a backdoor somewhere, get in, and then figure out how they’re going to spread out from there.

Jolie Hales:
Okay.

Ernest de Leon:
In the sales lingo, we often call that “land and expand.” That’s exactly what they’re doing, but from a nefarious perspective.

Jolie Hales:
Interesting. Okay.

Ernest de Leon:
It turns out that while FireEye was definitely targeted for the wealth of security tools that could be used for nefarious purposes, they were not the only target. Enter our friends at SolarWinds. So as the FireEye team investigated, they discovered that the breach had originated from their use of SolarWinds. But the question was how. SolarWinds has a product called Orion. I know, the space puns are all over the place today. Anyway, Orion is a platform used by tens of thousands of companies to manage their information technology resources. It’s not only common, but fairly ubiquitous for software companies to push updates to their systems, often to enable new features or patch known security vulnerabilities. Ironically, in this case, it was this update mechanism that enabled the proliferation of the hack.

Jolie Hales:
Okay. So let me get this straight. FireEye is a cybersecurity company that uses SolarWinds’ IT management software in their business.

Ernest de Leon:
Right.

Jolie Hales:
And SolarWinds pushed out a patch or some kind of update for their Orion that was used by FireEye, And they were completely unaware that it had malware attached to it until FireEye I noticed something suspicious and made the discovery.

Ernest de Leon:
That’s right. Someone hacked into SolarWinds themselves and crafted a malicious software update package that was then injected into the update server from which SolarWinds used to distribute updates to the customer.

Jolie Hales:
And then that malware ended up breaching the systems of, I mean, I would imagine just about any company that used the Orion product.

Ernest de Leon:
Exactly.

Jolie Hales:
Dang, that’s pretty scary.

Ernest de Leon:
And it’s even scarier how the hack first happened.

Jolie Hales:
Uh-oh. Okay. Now you have to tell us.

Ernest de Leon:
I will after the break.

Jolie Hales:
Oh, now it’s your turn to leave us hanging.

Ernest de Leon:
Got to get paid.

Jolie Hales:
From supersonic jets to personalized medicine, industry leaders are turning to Rescale to power science and engineering breakthroughs. Rescale is a full stack automation solution for hybrid cloud that helps IT and HPC leaders deliver intelligent computing as a service and enables the enterprise transformation to digital R&D. As a proud sponsor of the Big Compute Podcast, Rescale would especially like to say thank you to all the scientists and engineers out there who are working to make a difference for all of us. Rescale, intelligent computing for digital R&D. Learn more at rescale.com/bcpodcast.

Ernest de Leon:
Okay, back to the hack. Every SolarWinds customer who used the Orion product was now compromised by doing something they should have been doing, which is updating software. The beauty of the hack, if you want to call it that, was in the distribution method chosen. They didn’t try to hack a web server and upload a compromised installer file or anything so mundane. They found an opportunity to attack the updates supply chain and thus remained completely under the radar.

Jolie Hales:
That is so crazy. We always think… I know that I do, at least. I’m always thinking that software updates are a way to protect ourselves from security threats, right? Whenever there’s an update that comes out for my phone or my computer or my operating system, I’m always getting that update installed immediately. So whoever these attackers were, it’s like they used our own psychology or something against us.

Ernest de Leon:
Right. And I’d love to give them the credit to say they did that, but I think as we’ll find out later, it was more of a crime of opportunity.

Jolie Hales:
Okay.

Ernest de Leon:
So because Orion was used to manage and, in many cases, update the many IT systems of SolarWinds customers, the attackers now had a vector to install even more malware on these customer systems without their knowledge. This was truly an inception level of hack.

Jolie Hales:
How many customers were actually affected by that?

Ernest de Leon:
So the truth there is that we don’t know.

Jolie Hales:
Wait, really? Not even a little bit?

Ernest de Leon:
Well sort of, but not really. SolarWinds claims that as many as 18,000 customers installed the compromised updates, many of those being Fortune 500 companies and several agencies of our own U.S. federal government.

Jolie Hales:
18,000 customers. So we’re talking 18,000 different companies, not 18,000 individuals, right?

Ernest de Leon:
That’s correct.

Jolie Hales:
Oh my gosh, that is so bad.

Ernest de Leon:
It is. Now, from what I’ve gathered from various media reports and through my own sources in the cybersecurity community, we know that FireEye, Microsoft, Cisco, Intel, and a few other tech companies where among those attacked.

Jolie Hales:
Oh, they’re big names.

Ernest de Leon:
Big names. We also know that the Department of Energy, the Department of the Treasury, the Department of Homeland Security, the Department of Defense, and even the-

Jolie Hales:
What?

Ernest de Leon:
Yeah, and even the National Nuclear Security Administration were all hacked as well.

Jolie Hales:
Even if you don’t live in the United States, the reach is global.

Ernest de Leon:
Yes. The reach of the attack is stunning.

Jolie Hales:
That’s crazy. And not just in the high performance computing industry, which we’re focused on, but pretty much everyone in every industry.

Ernest de Leon:
That’s right. So the worst part of this that we know of is that the hack was not discovered immediately.

Jolie Hales:
Oh boy.

Ernest de Leon:
It’s thought right now, again, because the story is evolving, that the purpose of this hack was to do reconnaissance on all of these companies, agencies, and systems, and to prepare for a future cyber attack. But we don’t know that for sure. As far as we know, this attack was in place for months before it was discovered.

Jolie Hales:
Dang! I mean, if it’s been in place for months, who knows how much information they had access to in that time.

Ernest de Leon:
Right. We’re still operating on a lot of hypotheticals here, and it will take months more to figure out the full extent of this hack. It’ll probably take years to ensure that all the possible compromised systems have been rebuilt from the ground up and all the networks purged of possible malware.

Jolie Hales:
So when you say rebuilding all possible compromised system from the ground up, what do you mean exactly? Are we talking fresh installs and updates or completely new software development or something completely beyond my scope of cybersecurity understanding?

Ernest de Leon:
No, you’re absolutely right. We’re talking about fresh installs and updates, decommissioning of systems that were potentially compromised, what types of systems you should consider compromised, whether or not you know they are, and just kind of a general operating guide as to what to do. This obviously sets policy for the entire federal government and many private industries follow it as well, just as a precaution. We will get into that in the next episode, but yes, absolutely, that’s exactly what we’re talking about.

Jolie Hales:
Interesting. That’s got to be a big job.

Ernest de Leon:
Yeah. So now we get to the two questions that our listeners have probably been asking this entire time, who did this and how did they manage to hack the update system of SolarWinds?

Jolie Hales:
That’s exactly what I was thinking.

Ernest de Leon:
Yeah. Because once you’ve built up a story like this, the first question in my mind is, okay, who pulled this off, right?

Jolie Hales:
And then how does it affect me?

Ernest de Leon:
Right. What’s the fault for me? To the first question, we do not have definite proof of who did it. We do, however, have a lot of indicators, things we call fingerprints, that points in the direction of Russia’s Foreign Intelligence Service.

News Clip:
Experts believe Russia was behind the hack of a company called SolarWinds, sending malware to 18,000 private and government organizations. Russia has denied any involvement in the hack.

Jolie Hales:
Dang it, Russia! I swear that the same people who put that ransomware on my computer, they’re probably not the actual Foreign Intelligence Service. Probably just some dude in an apartment.

Ernest de Leon:
Perhaps. Now, you may be familiar with the names Cozy Bear or Fancy Bear. These are terms that are used for units believing to be operating out of Russia at the behest of the Russian government.

Jolie Hales:
How dare they defy such cute and cuddly names?

Ernest de Leon:
Now, you might be thinking to yourself, if this hack was this massive and this pervasive in nature, it must have been an epic undertaking, right?

Jolie Hales:
I mean, you would think so.

Ernest de Leon:
It immediately starts bringing things to mind like Mission Impossible and the kind of dropping in from the ceiling through a vent that had lasers looking for you and hacking some mainframe.

Jolie Hales:
Yeah. Wearing sunglasses indoors.

Ernest de Leon:
Right. It’s something where the perpetrators had to find the right window at the right time and exploit a known zero day vulnerability to gain access to one system in the SolarWinds network then work their way around to the update server to inject the compromised update software package. Sounds reasonable given the breadth of the hack, right?

Jolie Hales:
I mean, I don’t know much about hacking, but I would assume it would be an intricate and complicated process. Yes.

Ernest de Leon:
Well, as is often the case in life, the reality of the situation is far less sensational than it might seem.

Jolie Hales:
Really?

Ernest de Leon:
You see, SolarWinds was using an FTP server to host the update files that its customers used to update their systems automatically via Orion. And our users out there, FTP is file transfer protocol. It’s an archaic protocol that people should not be using any more or just about any… I realized that there are some very unique edge cases where you might use it, but please stop using FTP if you’re using it.

Jolie Hales:
I was going to say, I used that maybe 15 years ago.

Ernest de Leon:
Yeah.

Jolie Hales:
Haven’t seen it since.

Ernest de Leon:
There’s a reason why. In 2019, a security researcher notified SolarWinds that the password to this FTP server had been leaked on GitHub in plain text.

Jolie Hales:
GitHub? There’s a lot of people on GitHub. This isn’t like some underground dark web.

Ernest de Leon:
That’s right. And mind you, this was a public GitHub repo, which means anyone could see this password, not just SolarWinds employees.

Jolie Hales:
Oh my gosh.

Ernest de Leon:
But that isn’t the best part of the story. Even if it had not been leaked, can you guess what the password was?

Jolie Hales:
Please tell me it wasn’t password.

Ernest de Leon:
No, it wasn’t, but it’s pretty close.

Jolie Hales:
Oh no.

Ernest de Leon:
The password was actually SolarWinds123.

Jolie Hales:
It was not.

Ernest de Leon:
Yes. Ladies and gentlemen, the password was the name of the company followed by the numbers 123.

Jolie Hales:
That password led to a ginormous hack of major companies and agencies, including the Department of Homeland Security and the Department of Defense?

Ernest de Leon:
That is absolutely correct.

Jolie Hales:
Oh my gosh!

Ernest de Leon:
Some of the most secured agency networks on the planet were hacked by way of a password that was SolarWinds123.

Jolie Hales:
That is so incredibly insane. And of course, this makes me think up so many other questions, like what kind of information did the hackers get away with? And what does that mean for Americans or people in other countries? I mean, what does that mean for us in high performance computing? You mentioned Microsoft, which is a sponsor of the Big Compute conference. I mean, what can this information be used for? Was the plot for a larger cybersecurity attack simply thwarted? I mean, do we know the answers to these questions? Or could some nation like Russia or China now have a bunch of information they could use to hurt other countries? I mean, do we have any idea what the hacker’s intent might’ve been? I know this is a thousand questions. But if I have them, I imagine our listeners also have them. Ernest, please guide us down this path. We need to know.

Ernest de Leon:
So I want to explore all of those answers with you, but we’ve already run out of time for this episode, so you’ll have to ruminate until the next one.

Jolie Hales:
Oh man! Talk about a cliffhanger. Everything everyone’s digital life has been compromised. Tune in next time to learn more about this cyber apocalypse. That’s like what it feels.

Ernest de Leon:
And in the meantime, you can help us spread the word of the Big Computer Podcast by leaving us a five star review on Apple Podcasts, Google Podcasts, Spotify, or wherever else you listen. And you can check out bigcompute.org for more information about the SolarWinds hack.

Jolie Hales:
I like that transition. It’s like we go from cyber apocalypse to please leave us a review, five stars.

Ernest de Leon:
What’s that Uncle Scrooge thing where he says, “Please, more food,” or what is it?

Jolie Hales:
“Please, sir, may I have some more.”

Ernest de Leon:
Yeah.

Jolie Hales:
No. Are you talking about Christmas Carol?

Ernest de Leon:
Yeah, there you go.

Jolie Hales:
Disney Christmas Carol?

Ernest de Leon:
Yeah.

Jolie Hales:
No, wait. Is it that? No, that’s God bless us, everyone. I don’t know.

Ernest de Leon:
No, it’s the one with Uncle Scrooge.

Jolie Hales:
Please, sir, can I have some more? That one.

Ernest de Leon:
That one. That one.

Jolie Hales:
And it’s like, “No. You got to go leave us a review first, kid.”

Ernest de Leon:
Yeah. You got to leave us reviews, and then we’ll continue with our disparagement of large corporations and government agencies.

Jolie Hales:
Well, this is definitely a cliffhanger. I’m excited to learn more, so we got to get that recorded ASAP.

Ernest de Leon:
Thank you all for listening and we’ll catch you in the next episode.

Jolie Hales:
Stay cyber safe.