[00:00:00] Luke: From privacy concerns to limitless potential, AI is rapidly impacting our evolving society. In this new season of the Brave Technologist podcast, we’re demystifying artificial intelligence, challenging the status quo, and empowering everyday people to embrace the digital revolution. I’m your host, Luke Malks, VP of Business Operations at Brave Software, makers of the privacy respecting Brave browser and search engine, now powering AI with the Brave Search API.

[00:00:29] You’re listening to a new episode of the Brave Technologist. And this one features Shubham, At Kodiak, he’s the mastermind behind all machine learning endeavors steering the development of the safest autonomous driving technologies. His leadership and innovations at Ford Autonomy, particularly in 3D perception for vehicles, have set new industry standards.

[00:00:47] Shubham’s numerous patents and publications echo his pivotal role in advancing autonomous vehicle technology and underline his commitment to transforming the future of transportation. In this episode, we discuss the role of AI in enabling [00:01:00] self driving trucks. And how it can solve the increasing challenges with the long haul trucking industry, supply and safety, practical examples of their safety strategy and redundancies Kodiak is putting in place.

[00:01:11] How they’re navigating regulation within the space of both the national and state levels across the U S And how soon we can expect to see autonomous trucks on the road, and now for this week’s episode of the brave technologist.

[00:01:26] Hi Shabama, welcome to the Braid Technologist podcast. Thanks for joining us today.

[00:01:31] How are you doing?

[00:01:31] Shubham: I’m doing great, Luke. Thanks for having me. Excellent. So

[00:01:35] Luke: let’s get the audience a little familiar with you. Um, Can you tell us a little bit about how you ended up doing what you’re doing? Was it something you just kind of had a keen interest in or something you kind of fell into?

[00:01:44] Shubham: Yeah, I would say it was both.

[00:01:46] I was fortunate enough to start my career in the automotive capital of the world, Detroit, where I immediately started noticing prototype autonomous vehicles and really got hooked. It was [00:02:00] fascinating to see these vehicles loaded with so many sensors on top. You know, thereafter I moved towards working completely on computer vision problems and autonomous vehicles, which was pretty exciting.

[00:02:11] My keen interest in machine learning and computer vision kept growing. And this eventually brought me to Ford Greenfield Labs in Palo Alto back in 2019, where I was leading the perception team in the autonomy division. And from there to leading all machine learning efforts here at Kodiak. Awesome.

[00:02:27] Luke: Was there anything kind of in your background that really helped you get to where you were or what kind of skills would you recommend folks that might have an interest in this

[00:02:35] Shubham: focus on? For me, it was a mix of curiosity and the hunger to learn more. It’s really important to focus on the fundamentals and think from the first principles.

[00:02:46] I’ll give you an example. Machine learning is evolving rapidly. New technologies are being built every single day. current state of the art are really being knocked out of their feet as we speak. And there is this big push for moving [00:03:00] towards unsupervised and semi supervised learning. We’ve seen methods such as segment anything and most recently depth anything emerge and shake the whole computer vision community.

[00:03:11] But the point here is Most folks interested in these machine learning methods want to spend the least amount of time and energy and simply try these methods on their data. Obviously, these models work great and everybody’s super excited. But if you truly want to learn how these models work, get an intuition on their inner mechanics, and if you really want to take it a step further and go build something by yourself, then you really need to take extra efforts to read the paper, read the technical reports and Really dive deeper into the open source code.

[00:03:45] So if I had to advise people on the kind of skills to focus on, it would mostly be towards building the foundation of how things work, getting your hands dirty, poke holes into the code basis, you know, try out, new ideas and [00:04:00] as you do these, something will spark in you and it’ll help you go build extraordinary

[00:04:05] Luke: things.

[00:04:05] This is one conversation I’ve just been super interested in kind of getting into because so much of what we talk about on here is kind of limited to the digital world. And we’re talking about, you know, software or how certain hardware and software interacts. But like What you guys are doing is so embedded in everyday life, but there’s so many variables, right?

[00:04:25] Like I think about the first time you try to get that driver’s license when you’re young and like going through all the material. Okay. I got to know, speed limit. I got to know the right signals. I got to know all the prior protocol with all the analog things. Then you’re dealing with software too, right?

[00:04:38] Like, and then all of the variables on how software and hardware connect and other drivers and all of this, it just seems like overwhelming, right? You guys are leading in the industry on this kind of self driving technology. Like, how are you guys using AI enabling kind of the self driving and like, is it different teams working on different parts of this puzzle together?

[00:04:57] Or is it like, give our audience kind of a [00:05:00] sense of, how this all kind of comes together. I think that would be interesting.

[00:05:03] Shubham: Yeah, I mean, there are lots of moving pieces that needs to come together. But before I actually get into the technical side of things, let me. Briefly talk about why self driving trucks are important.

[00:05:15] You know, we at Kodiak really believe that autonomous trucks are going to change the lives of so many people. Really two thirds of all the products that are shipped each year are delivered by trucks. There is a saying in trucking, if you bought it, a truck brought it. Today, we really face a shortage of about 80, 000 long haul truck drivers.

[00:05:35] And that number is. only going to increase and it’s estimated to increase up to 160, 000 by end of this decade. And it’s really because, you know, driving a long haul truck is difficult, dangerous job that usually keeps people away from their family for days, weeks, and even months. And fewer and fewer people want to do that.

[00:05:57] So We think that by [00:06:00] automating the long haul truck driving, we can solve one of the world’s biggest economic problem. And anybody who has lived in the United States over the last four years know about the supply chain problems and how the pandemic has impacted Right. Even like autonomous trucks has a lot of benefits that traditional trucking does not.

[00:06:20] They can operate 24 7 only stopping for refueling or picking up new loads. They’re going to be so much safer. They don’t drive distracted. They don’t have a bad day. They don’t drink and drive. And in making all of these happen, AI plays a huge role, right? These semi trucks weigh tens of thousands of pounds.

[00:06:41] And in order for them to operate safely on highways at high speed, we need to be able to see far and not just see far, but you need to identify every single agent on the road, estimate where they are in the world and track how they have been moving and even forecast their trajectory in the future. [00:07:00] And to do this in a way such that you can ensure safety every single moment in time, you need robust and redundant sensing at Kodiak.

[00:07:10] We truly believe in redundancy. We have More than 40 neural networks that’s running in parallel extremely, extremely efficiently consuming many cameras, lidars, radar as its inputs and producing holistic understanding of the world. We really leverage state of the art in machine learning research and build on top.

[00:07:32] We have built cross modal spatio temporal neural networks that can fuse features in a high dimensional neural space for a complete 3D scene understanding. So, there is so much going on and all of that needs to really come together to make it all happen. Yeah, it

[00:07:48] Luke: is. And it’s super disruptive. I mean, I would imagine this impacts things like rail transport to in some context.

[00:07:55] There’s a lot to unpack there too. It’s just, I think it’s really smart to going at it from the trucking [00:08:00] angle too, because if you can get into the commercial side of this and, and on the business benefits and the ways that this can scale out to the broader market too, it’s like got a lot of those proof points and, a lot of that kind of commercial business running through it, which is really, really smart.

[00:08:14] Do 5G and things like that kind of play into what you guys are doing in being able to transmit all of this data? Is it real time? Is it stuff that is happening locally in real time? I’m trying to kind of unpack how all this stuff kind of works. Right. Is this thing constantly just kind of checking via all these centers making sure that the soccer mom next to you is not too close or something like that?

[00:08:36] much infrastructure is in place for something like this to scale across the Continental US, for example. So

[00:08:44] Shubham: this is really a safety critical application, right? And every single milliseconds matter, which is why you cannot depend on doing the processing off board. All the sensor data that comes in, they come in in real time.

[00:08:59] We need to make [00:09:00] sure that we process those sensor data really, really efficiently of the orders of. let’s say tens of milliseconds. And there are multiple neural network models that are processing data from multiple sensors, cameras, lighters, radars, and all of those need to process the data independently.

[00:09:21] They all produce their predictions and they all get fused. And all of that needs to happen in pretty much real time. So everything that we do does happen on board and we need to take real time decisions about what trajectory we should be following. How do we avoid getting too close to the vehicle in front?

[00:09:41] How do we make sure we switch lanes in a safe manner and so on? When

[00:09:47] Luke: you’re talking about the roads and safety and, all of this, you know, machinery, like people are going to kind of react to that. Do you think that people tend to misunderstand or underestimate the issue or, where do you think that this, some of this kind of stems [00:10:00] from?

[00:10:00] Shubham: Trucking is. Often underappreciated, right? And with expected shortages in number of long haul truck drivers, there is really going to be a huge dent in the economy. So there is no doubt in my mind that this problem needs to be solved. And autonomous trucking is really the first step towards solving it.

[00:10:20] The solution often requires. hardware, perception, planning, and controls, all these teams to collaboratively work together and build a product that really works well. Machine learning is critical in solving perception for autonomous driving. You need to model not only what you have seen, but also the unknown unknowns.

[00:10:41] As humans, we have learned an extraordinary model of how the world works, and we are able to quickly apply those learnings to a new and unseen problem. And This is simply not possible with machine learning. Neural networks cannot really generalize to unseen domains. The best ammunition we have gotten towards [00:11:00] generalization is to build foundation models through auto labeling and pre training of our models.

[00:11:05] At Kodiak, we are looking at a scale of Parabytes of auto labeled data on which we can train those foundation models and then fine tune on smaller scale human refined data. This is essentially what allows us to first generalize and then specialize. In addition, we are also building machine learning cycles where we are able to source the rarest kind of data and go after long tail cases and tackle the toughest of problems.

[00:11:32] Luke: AI is just kind of everywhere, right? And it’s transforming a lot of things. What gets you excited about what you’re seeing out there right now?

[00:11:39] Shubham: Right. I mean, AI has been booming and there has been so much hype recently around generative AI, both in natural language processing and computer vision. I personally am most excited about multimodal temporal neural networks.

[00:11:56] These models tend to learn something [00:12:00] fundamental about the world. They seem to reason about What information is available in the model’s input and their interaction, right? A simple vision and language models may be pre trained on just the data available on internet by simply training the network to try and predict the next token in a sequence.

[00:12:18] Most data humanity has Often seen on internet are just text and visuals. And really the advent of transformer architecture now allows us to model any arbitrary data. And in the context of vision and language, the data can just be serialized and interaction between the data can be modeled by means of self and cross attention.

[00:12:40] If objective is formulated correctly, then we can do so many cool things with it. As an example, we can feed an image of a scene from the highway and ask. Whether or not it is okay to proceed in my current lane at a speed of let’s say 65 miles an hour. And really answering these kind of questions [00:13:00] require these neural networks to have a mental model of the world, understand what actors are driving on the road, what does safe mean, and so on.

[00:13:10] In a sense, I would say this is a foundation model that understands the underlying physics and can be fine tuned on certain specialized tasks. Impact of this technology obviously is tremendous and in every possible field. I’ll give you some examples. It’s already transforming robotics by building comprehensive understanding of the world and how to operate within them.

[00:13:34] In healthcare, Multimodal networks can interpret medical imagery, understand patient’s history and parse through clinical notes simultaneously. In education, these models can help you with the learning process by breaking down the problem and then working as your personal teaching assistant. Even in environmental impact, it has huge environmental impact, right?

[00:13:55] Where these models can now analyze satellite imagery along with [00:14:00] Climate data and predict natural disaster, track changes in ecosystem and even inform strategies to combat climate change.

[00:14:08] Luke: I want to go back to a little bit to multi modeling. You were giving a really good example that I kind of want to drill down into a little bit.

[00:14:14] Let’s walk the audience through, I’m driving on the freeway. This is all happening like really quickly. Right, right. Maybe you can kind of give a really kind of a naive point of view as to like, okay, here, I’m just driving down the road. here are the different, it doesn’t have to be every single bit, of course, but, a bit of a high level as to like, what’s going on.

[00:14:32] Shubham: Right. So there are many kinds of sensors that we at Kodiak use as part of our sensor suite on these trucks, right? Some of those sensors include cameras, lidars, radars, and they’re all beneficial in different kind of conditions, right? If the weather is perfect, the lighting condition is good. Cameras are going to give you enough information.

[00:14:56] To understand the surroundings, but if it is not [00:15:00] good, one of the places where cameras don’t do super well is when it’s raining or when it’s fog, you just cannot see through them. And at the same time, you don’t necessarily perceive things in 3d from camera, unless you do some processing on top. If you have, let’s say, stereo vision with wider baseline, you’ll be able to see far.

[00:15:22] But otherwise you won’t be able to perceive the world in 3D. So that’s where LiDAR really shines, which is one of the sensors that we could use to perceive the world in 3D. Similarly with radar, again, LiDAR and radar are useful in, you know, different weather conditions and whatnot. LiDARs will have some of the downsides while driving through.

[00:15:44] dust and fog, but radar can really see through them. So we take all of those sensor inputs and really try to make sense of what does the surrounding look like, right? And to be able to do that, we have many kinds of models that [00:16:00] operate independently and together on some of these sensors, and they really.

[00:16:06] are trying to extract higher dimensional features that can then be fused together to really understand what’s going on in the world to really understand where does the lanes lie how far laterally and longitudinally we are driving from other agents on the road how far can we go at the current speed before we approach too close to the vehicle Ahead of us, right?

[00:16:34] Things like those. And once we have a really holistic representation of what the world around ego vehicle looks like, we can really plan our path and we can control our vehicle accordingly so that we drive through a predetermined path that we refine every certain number of time steps, right? That’s really what self driving stack overall looks like.

[00:16:58] And in [00:17:00] trucking, it’s no different.

[00:17:01] Luke: That’s awesome. It’s just mind blowing how far along some of this stuff has come over the past ten or so years and how much it’s even scaled to where it’s able to get now. When I normally talk about regulation with folks, it’s a little bit more abstract, but I would imagine that what you all are dealing with is much more granular and also across the board.

[00:17:21] I would imagine you all are having to deal with the Department of Transportation. Are you guys dealing with this like on a national level or is it national and state, you know, interstate trucking, right? Like you’re crossing state lines and did the state’s regulators have different opinions on this from the national?

[00:17:37] Like, is everybody just so new to this they don’t really understand it? Like, how has it been as somebody building technology in this area that obviously, like, The safety implications are huge, right? Like around this it’s a very kind of a sensitive thing that people probably get emotional around for good reason, but it’s also leading technology.

[00:17:54] Right? Like it is very hard for a lot of people. I mean, like to probably wrap their heads around this, like how, how [00:18:00] has it been trying to kind of navigate the regulatory side of this as a technologist?

[00:18:04] Shubham: Wrong use of AI can really be catastrophic. And we are already seeing these adverse effects with technologies, even in terms of, I think deepfakes has recently been really popularized by the impact it can have in someone’s persona and things like that, right?

[00:18:23] Misinformation on social media through AI bots. Privacy invasion, money embezzlement from elderly. All of these are really bad results of, you know, bad use of AI. And I’ll say this, the only reason humans are at the top of the food chain is because of our intelligence. And at the rate that we have been making progress in AI, we may see the point of singularity soon, beyond which they can be perceived as more intelligent than us.

[00:18:51] And what happens after is not really clear. So the balance between AI innovation and regulation is really delicate. And [00:19:00] on one hand, too much regulation can stifle the innovation and slow down the research. And on the other hand, too little regulation can lead to unchecked development that can harm society and even pose existential threats.

[00:19:14] I do think that AI and robotics are beneficial for things that are really dull, dirty, and dangerous. And those are really the places where it should be applied the most by. Automating manufacturing processes and transportation, for example, will be able to create better jobs for the society.

[00:19:32] Luke: What are some of the trickiest problems that you’re having to solve around self driving trucks?

[00:19:38] Shubham: I mean, one of the major assumption is that every other agent on the road is also following the traffic rules. Every once in a while, you often see erratic behaviors and believe it or not, you have seen. Humans run across highways while we are driving and we have been able to detect them from pretty far [00:20:00] out and we have been able to Operate or like slow down change lanes and things like that accordingly.

[00:20:06] But yeah, one of the biggest challenge is Just dealing with those and the way we do that is to really source the rarest kind of data on the highway. Just make sure we get labels on them. We have simulations to ensure that we can deal with all kinds of scenarios that can happen on the highway. We have rigorous testing and all the processes around it.

[00:20:31] But yeah, it’s a difficult challenge to deal with the unexpected.

[00:20:35] Luke: I would imagine. What’s your outlook on the future? Are you optimistic about where we’re heading with, with us in the next five or 10 years?

[00:20:41] Shubham: Yeah, I am extremely excited about the prospects of AI and robotics, and I’m eagerly waiting to see how it all unfolds the rapid development we have been seeing in this field.

[00:20:54] I’m highly optimistic about where we will end up in the next two, five, or even [00:21:00] 10 years, right? I’m really hopeful of the transformation it can bring in the industrial segments towards creating better jobs for people who put a lot on the line today to earn daily wages. In trucking industry, it will create better and sustainable jobs.

[00:21:15] You know, it’s potential on healthcare and climate control are unparalleled. So I truly believe that the future outlook in this space is radiant.

[00:21:24] Luke: Putting it in our audience’s perspective, you know, far out do you think that they are from looking over and seeing a self driving truck next to them on the highway, give the audience a sense of like how far.

[00:21:36] Your work is from like just being a normal observable thing on the road,

[00:21:41] Shubham: right? So there are two parts of it one is hardware itself And then there is software to be able to put something like a semi truck on the road We need hardware that is ready to go driverless, right? And this is one of the things we announced early this [00:22:00] year.

[00:22:00] Our next generation, the gen six trucks are ready to go driverless in 2024. Yeah, so realistically, we will be able to see some of these driverless trucks very soon. I don’t want to put a timeline on it, but we are ready to go today.

[00:22:17] Luke: Really soon is close enough. Are these electric trucks or are they, they got a diesel power?

[00:22:24] What’s the powertrain like on

[00:22:25] Shubham: these trucks? We work with all kinds of trucks, but yeah, mostly non electric ones. Cool. Cool.

[00:22:32] Luke: That’s interesting. What advice would you give to researchers and students that want to have an impact and want to work on things like this what you’re working on at Kodiak around that are really going to be transformative that involve AI in the space?

[00:22:44] Like, is there anything you would recommend they dig into as far as like research programs or anything like that you could

[00:22:50] Shubham: recommend? Yeah, I really think we are fortunate enough to be living in an era where GitHub and archive is a thing. Authors of most research papers end [00:23:00] up open sourcing their code and the results are very easily reproducible.

[00:23:05] It allows researchers to sort of build up on existing work and build even better things. So my recommendation would simply be to like. Dig deeper into those code bases, tinker with those parameters, learn from plethora of research papers, and try to ask yourself why certain things work. Build intuition, implement ideas, experiment, and analyze results.

[00:23:28] I really think this Really getting your hands dirty and going trying to build things yourself really teaches you a lot.

[00:23:37] Luke: It’s awesome to hear like how many times you’ve said open source in this conversation. I think it seems like one of the big differentiators too between this recent wave and past ones is just how much of this is like open source that would you say like the majority of what you’re working on all open source is it kind of a mix between some proprietary, some open?

[00:23:58] Shubham: So all the things we [00:24:00] do are proprietary, but open source really. Helps you, you know, start experimenting with what is state of the art. And I think I’ve mentioned earlier state of the art these days are getting knocked off of their feet as we speak, right? In academia, researchers are really focusing on building best of the best.

[00:24:19] We within the industry can really take a look at some of those ideas, see, understand why they work and really try to. Think about what can we do in order to take those state of the art stuff and make it work in really a time constrained application such as autonomous trucks, right? And that is also one of the gaps that I have personally seen between academia and industry is where academia has really been pushing towards making state of the art technology, pushing them forward.

[00:24:53] And there has to be. I feel there has to be more efforts put into making cheaper models that do equally [00:25:00] well I’m not seeing so many of those out there, right? But I really think building on open source ideas open source models open source repositories and Really building it, transforming it into something that can work well with your application that can work even better for your data, even faster while achieving the same level of performance is really the key to unlocking all industrial AI.

[00:25:29] That’s awesome

[00:25:30] Luke: and I really appreciate it. You know, you’ve been super generous with your time here and I want to respect that. Is there anything we didn’t really touch on that you would like to let our audience know about or anything that while we’ve got their attention that you want to put out there

[00:25:43] Shubham: to them?

[00:25:44] No, I’ll just say I cannot stress enough. That if you really want to learn, you have to go and build things. You learn, you only learn by doing them yourself. That, that’ll just be my messaging.

[00:25:58] Luke: Awesome. somebody wants to follow [00:26:00] your work reach out to you, are you out there on social media? Is there, we can put it in the bio too, or in the show info, but is there anywhere you’d recommend people give you a

[00:26:08] Shubham: follow?

[00:26:09] Yeah, absolutely. I think LinkedIn would be the best place to reach out to me. Awesome.

[00:26:13] Luke: Awesome. Thank you so much. I really appreciate your time. I learned a lot today, a ton to unpack, and I’m sure our audience is going to love it. Love to check back in with you too as things progress. And hopefully we see some of these self driving trucks on the freeway a day soon, you know?

[00:26:29] Shubham: Absolutely. Thanks for having me, Luke. It’s been really good. so much. Thank you have a good

[00:26:33] Luke: one thanks for listening to the brave technologist podcast to never miss an episode make sure you hit follow in your podcast app if you haven’t already made the switch to the brave browser you can download it for free today at brave dot com and start using brave search which enables you to search the web privately brave also shield you from the ads trackers and other creepy stuff following you across the web.