Speaker Series

Engineering Superpowers in the Cloud

Post by
Jolie Hales

Speaker:

Please welcome to the stage CEO of MSBAI Allan Grosvenor


Allan:

Five years ago. And I’m driving North from Los Angeles to the Mojave desert. And I’m really excited because the small company I worked for Masten space systems has just won this massive opportunity from DARPA to design a next generation space launch system. We were chosen for the Access One program. The goal of Access One was to design a space launch vehicle to stage vehicle that would launch the booster together with the upper stage, the edge of space, the upper stage would continue to orbit with the satellite. The boost stage would turn around and fly back and land. And not only would it be landing like today, the Falcon nine does from SpaceX, but it would also be ready to fly again. The next day they wanted so-called aircraft like operations, super exciting projects. I was also really nervous. We were a very small company, total number of people in the company about 30 people.


Allan:

I was leading an aerodynamics team of two people, including myself. Now, the reason I was driving from Los Angeles to Mojave is Mojave is a magical place. You can fly experimental vehicles there. You can light off rocket engines. You can fly to space, but it’s the desert. You don’t want to live there. So I was going to have this long commute every day, an hour back and forth. I knew in this program that we were going to have access to department of defense supercomputers, and I knew that I wanted to somehow make use of this drive time. So I tethered my laptop to my phone. I wrote a series of scripts that enabled me to initiate a lot of compute jobs while I was connected remotely to the DOD supercomputers. And then I connected speech control to those scripts. And for the following several months, I had ongoing conversations with my laptop on the drives to, and from Mojave. I got an incredible amount of work done while I was on these drives up and down the 14 highway using high performance computing to run design cycles, to check in on jobs. And it made a tremendous difference in my ability to get work done.


Allan:

But I want to tell you how I got there when I was a 12 year old boy. I was not so interested in school, but I loved science fiction. I loved watching. Nightrider where the super intelligent car kit was helping Michael Knight fight crime. The crew of the Starship enterprise with the super intelligent ship’s computer and the Android named data. And I loved both the Terminator and Terminator one in Terminator, two movies in this scene. The reprogram Terminator is about to tell young John Connor, some pretty bad news. Well, I come home from school and I would go to my dad’s work bench and I would make stuff. I would try to copy what I saw in these these shows and movies that I loved. I made robotic arms custom model rockets out of metal, all sorts of crazy stuff. And I got parts from some pretty interesting places.


Allan:

I had a buddy that knew how to get into a local junk yard, and we would go there and pull apart things like photocopiers and take out AC motors. And I blew a lot of fuses in my parents’ fuse box and almost set the house on fire a few times, but my parents understood that I was passionate about this, and I was lucky to have a few teachers that did too. And when they introduced me to engineering and I realized that that could be not just a hobby, but a job. I knew that this is what I needed to do. Now. I ended up going to school for mechanical engineering at Carleton university and then aerospace engineering there as well. And this was the nineties. Now I was seeing change happening with computational science codes like computational fluid dynamics. Things were starting to happen.


Allan:

And new technologies were starting to be developed that were blowing everybody away. I realized, okay, now it’s actually possible to simulate concepts on the computer before you build it and you can try new things. So I thought this is what I want to get into. And so I did, and I particularly focused on computational science in both my undergrad and my masters. And then I continued that focus in my career. I have worked in aviation related companies on energy technology projects and also space launch. And in those projects, I’ve been extraordinarily lucky to have access to supercomputers like these. On the left hand side, you see department of defense supercomputers. I have run simulations on each one of these systems on the right hand side, you see department of energy Oakridge leadership, computing facility, supercomputers. Jaguar was the first one there that I started running on then Titan and most recently summit. And we’re very lucky to have a very close relationship with Oak Ridge today.


Allan:

In all these, the projects we have come up with design concepts and we have been able to test them by running thousands upon thousands of different scenarios, different operating conditions, different geometrical, configurations, full design optimizations, to be able to test brand new concepts and turn them into working prototypes before we ever built anything. It’s a total game changer. This is one example. It’s the Access One project I was telling you about earlier. What you’re looking at here is a simulation of an entire launch and return trajectory. The boost stage has just separated from the upper stage. The upper stage is continuing to orbit. The boost stage is getting ready to return back into the atmosphere. And what you’re going to see is it’s coming back a very high angle of attack when you’re designing a reentry vehicle. One of the really key things you have to look out for is the maximum temperature that you reach in this vehicle.


Allan:

If you are not extraordinarily careful, you’re going to come down at very high speeds, very high so-called dynamic pressures. And if you come down too fast, as you’re hitting the thickest, the densest part of the atmosphere, you’re going to get too hot. You’re going to exceed melting temperatures of metal. Now that’s why in old school, a reentry, the vehicles you see you know, black thermal protection system materials that they install on the bottoms of the vehicle and those work great, except they’re extraordinarily expensive. If you have to use you know, carbon, carbon and ceramic based materials, it becomes so expensive that it drives the entire cost of the vehicle. Now, if you’re designing a vehicle that you want to have aircraft like operations for you don’t want to do that. So what we did is, is we use the department of defense supercomputers to run simultaneous trajectory and vehicle configuration optimization so that we could slow down at the top of the atmosphere.


Allan:

So that by the time we got into the densest part, we were slow enough that we could basically get, get away with stainless steel. We also put carbon on deleting edges, but Mmm, as a consequence of using the department defense supercomputers, we came up with a launch and reentry design configuration that has never been done before. And when we presented it to the DARPA, they responded by being very nervous. You know, when you’re designing high speed flight vehicles, whether an aircraft or a rocket generally they don’t want you to try something brand new. They want something with so-called heritage. The reason is because a lot of test pilots have died because they discovered the, in the absolute worst time, you could discover that there’s a yaw roll, a coupling, for instance, which means if you want to turn right, the vehicle actually turns left.


Allan:

You don’t want that. And you certainly don’t want to discover that when you’re testing it, for real, also tons of rockets have blown up. So generally they don’t want you to try a brand new configuration. They want to have you choose configurations that have already been fully studied and characterized like crazy. But the problem is, you know, the way we respond to them is we said, look, you want to achieve something that is completely unprecedented, something that’s never been done before. We’re not going to get there with the fully studied older configurations, but we knew that we had done our homework. We had run a very large number of simulations. We had full fully characterized aerodynamic databases that we had run on the supercomputers. And so we had some confidence. And when we built the wind tunnel test model and ran a series of winds, haul tests, the measurements compared very well with our predictions, they matched DARPA switched from being concerned, worried about this configuration to loving it.


Allan:

They told us it was one of the favorite things that we did in this program. I have been very lucky to work on a few high stakes programs like this, where we’ve had outcomes of this, this nature, but this was a particular, particularly special moment where we tried some really new things and it worked, and it was high performance computing that changed the game, but it’s not just exotic space flight vehicles that you should be using high performance computing for. We think every engineer needs an assistant, some help. And I want to give you some examples, look at this traffic light. Are you going to be able to see that from the street? Do you think somebody could have benefited by placing that bridge, that overpass and the signal together with a model of the road in a computer model and run some different configurations to see if there is a better way to place that probably we’ll get this electrical outlet. Don’t use that electrical outlet. If you, if your laptop’s running low and you need to plug in somewhere, if it’s squirting water out, look for another one. It’s very likely that they would have benefited from running a few different configuration scenarios and installations and shielding scenarios before they installed this thing.


Allan:

And this is an example where you’ve gotten too far into the build before you discovered some problems.


Allan:

This MRI example is a scenario where you really needed to train all the users around the machine a little bit better before you gave them the opportunity to discharge her, to turn it on. And a lot of people in this room have been in this situation. I know I certainly have, you know, you have one compute node that you’re ready to install in a brand new rack. You’re super excited about getting it up and running. So you throw it in, cable it up and started it up. Then you start getting some work done, you throw some more compute nodes in and wire them up. And by the time, you know, it, you are wishing you had spent a little bit more time planning, how you’re going to cable this thing. And a little while later, maybe you end up like this guy.


Allan:

Well, Woodrow Wilson more than a hundred years ago said that he not only used all the brains that he had, but all that he could borrow. And I’m sure he never envisioned the kinds of brains that we have access to today with high performance computing. And particularly with cloud, we have an unprecedented opportunity to borrow a huge number of brains to focus on the biggest problems that we needed to solve. And it is so exciting 10 years or more ago when I was talking to a variety of different cloud computing companies. And I was exploring the possibility of using some kind of cloud service compared to on prem systems that we had and government supercomputers, they sucked, they were too expensive. They were clunky. There were not, there were not many providers when you ask them what kind of interconnect speeds they could guarantee for you.


Allan:

They didn’t have good answers and you couldn’t run it either today. That has been totally solved. There is this amazing cloud infrastructure available and it is just waiting to be taken full advantage of. We also have amazing engineering software to run in the cloud. So we had these two incredible layers available to us today. You know, 2020 is this incredible time where super powerful cloud computing actually affordable. And then, you know, like a thousand different engineering software packages to do every specialized problem that you could imagine, not just structural analysis, thermal analysis, fluid dynamics, aerodynamics high speed, low speed, Aero, acoustics, vibration electromagnetics. I could go on and continue the rest of my talk, just listing all the different options. The one drawback is that it’s not one vendor, that’s building all these codes. It’s a very large number of vendors, and they’ll have different philosophies about how you design these user interfaces.


Allan:

And it is always the case that to become a productive user of any of these pieces of software, you need to spend a lot of time getting trained up in them. So there’s a natural barrier to you actually using the best tools for the latest project that you’re working on. What you’re going to generally do instead is you are going to get really good at using like a few packages and then just fit that into every project that you’re running. Now it’s not just a problem for the end user engineer, the designer, it’s also a problem for the engineering software vendors. At the beginning of my career, I actually was an engineering software vendor. And I have first hand experience with this. I also have a lot of colleagues in industry and all the big engineering software companies who collectively tell me the same experience I had generally, any of these ISBNs are tending to sell to like one, 100th of the number of customers that they actually could be selling to in terms of the number of companies they know are out there that could be taking full advantage of their software.


Allan:

And the primary reason for this is because their software is too hard to use. So we think that now is the perfect time to add yet another layer, a so-called intelligent layer, a management layer, or an AI driven assistant layer that helps engineers take advantage of engineering software and to deploy it to cloud.


Allan:

And we have been developing an AI driven assistant that we call guru. Now guru begins with a client that you install on your device. We don’t just design this to run on your desktop computer or your laptop. We also intentionally designed it to run on your mobile device. And it’s not just because of experiences like the one I told you about driving up the 14 highway to the Mojave desert it’s because engineering in general is asynchronous. And certainly computational science based engineering is a very asynchronous process. Here’s what mean by that. Usually you’re using some kind of queued system. It’s very unusual that you’re going to have full access to a system that you can just set up a job, deploy the job immediately, and it just starts running right away. But even if you did have that if you’re running full design studies maybe you’re running, you know, a hundred different versions of some design.


Allan:

Some of those simulations are going to continue running for, you know, an hour, two hours. It depends on how you’ve run it and the size of the model. And a lot of them are going to fail or stop running for a variety of reasons. Sometimes there are just really bad designs. So the performance is actually so bad that the physics that you’re simulating is screwed up and the job just crashes. Other times, there are just very nasty problems that you run into. That’s just a part of the process of running very large number of, you know, parametrically very jobs that there are infinite, number of reasons why the job’s gonna fail. And then sometimes compute nodes go down. And so to be able to actually do this kind of work, there’s always some time that elapses between when you intended for the job to start and when you actually get all the results that you need.


Allan:

So it’s very asynchronous. And what that means is particularly at the beginning of my career, I didn’t have a life too much if I had designed deadlines because my simulations weren’t just running during business hours, I would generally spend a good part of the day, setting up jobs, setting up them up to run, and then I’d find out in the evening that some of them crapped out and I have to go and diagnose why they were and get them back up and running the same would happen on the weekends today. We have enough tools available to us that you should be able to check in on your mobile device and find out what’s going on and give simple commands to the assistant, to get the jobs that failed back up and running for you. So that’s where it begins. And it’s a very modular system.


Allan:

Each of the specialized capabilities that the assistant has is contained in an individual module or what we call agents. And then the agents reside on our servers in what we call an agent society. And without going into too much detail, we’re using a hybrid intelligence methodology at the agent level where we’re using techniques that enable us to capture procedural symbolic rules, and also learn from relatively sparse datasets to give the agent the learning ability to be able to actually run these kinds of jobs and then connect to services like speech services. You can talk to this assistance.


Allan:

Once the goal manager of the assistant has decided what tools is going to enroll in your job, how it’s going to set up the workflow. Then it deploys it to cloud systems. And the reason that we’re doing this is because we know right now, there’s a very small fraction of engineers that are actually making use of high performance computing to explore brand new concepts and solve hard problems out of a crowd like this. You have a very small fraction of users who actually have all the expertise combined that that’s needed to do this kind of work. And we want to change that. You know, if you imagine, what does a creative person look like? And you imagine an engineer, not everybody is going to come up with the same image in their mind. Maybe this is a person who has some really fun, cool, imaginative ideas.


Allan:

Wouldn’t it be great to give this person the ability to harness the power of cloud computing and actually try those ideas out. And this guy doesn’t look happy. We’ve seen this guy in the office somewhere along the way, right? Well, wouldn’t it be nice to give this guy some tools that would make his job a little bit more fun? That’s what we’re doing. We’re making, using cloud computing easy in engineering. We’re extremely passionate about it. And I want to give you a glimpse of what we’re talking about. This is a working prototype. That’s actually running jobs at Oakridge national laboratory. And it’s operating based on speech control.


Allan:

Try this. It’s going to change your life guru. I need to run a new simulation, engaging convergence robustness agent. I love it, how it takes you through this.


Allan:

So that’s what we’re working on. We are extremely passionate about giving engineers the super power to harness the power of cloud computing to invent the future. We think the number of engineers using cloud today is nothing compared to what it’s about to be. Thank you.