Luke: [00:00:00] You are listening to a new episode of The Brave Technologist, and this one features Danit Naone, who is a senior engineering manager in Meta’s applied AI division with two decades of experience and an academic background in computers, neuroscience, and neuroscience. At Meta, she pioneered AI enabled coding interviews, making it the first major tech company to adopt AI native evaluation for engineering talent.

She also coaches leaders through AI driven change and helps new hires integrate AI skills into Meta’s engineering environment. In this episode, we discussed the internal resistance and public misunderstanding they had to overcome when pioneering their AI enabled interviewing process, how they’ve changed the way they evaluate engineering talent and interviews, and what they’ve learned about skills are most needed today.

What personal values to prioritize and how to test for this before hire is even made. And the importance of learning from and with the market through open source AI models and development at Meta. And now for this week’s episode of the Brave Technologist.[00:01:00]

Danit, welcome to the Brave Technologist. How are you doing today? I’m good. How are you? I’m doing well. I’ve been looking forward to this one.

So, so meta became kinda the first major tech company to run AI enabled coding interviews, and, and you were part of that rule out there, if I understand correctly.

Can you kind of walk us through what you were trying to solve with, that, that traditional, uh, interviews, weren’t able to do? So the funny thing is, when we started this is this wasn’t the problem we were trying to solve. When we started, it was, , 2024, AI just started to kick in and like AI as we know it today, and we started to see instances of candidates who use AI to practically to cheat.

So they see interviews were the traditional interviews, right? You have a problem, you need to have the problem exploration and then coding and then validation. Uh, very clear steps. But we started to assist signals of candidates who [00:02:00] useche pt, uh, at that time. , That was the only tool we saw.

And then at some point it. Tools that now it called Interview Code back then it had a Wall’s name. But basically with those tools you couldn’t spot when a candidate is using that tool. It created an overlay over the screen of the candidate. Uh, it was able to listen to everything, read everything, and you have full AI capabilities to search through the internet for any questions exist.

And it created an overlay of the solution for the candidate, including my thoughts and things, but that it couldn’t be detected when we share the screen. So we started from something very, very different. As we evolved, as we started discussing it and try to practically solve that problem, we realized that the problem is not that [00:03:00] candidates are cheating their way to the system.

The problem is that the system is checking for the wrong things. Interesting. The Brave Technologist is brought to you by the Brave Search, API access billions of indexed web results from a simple API call with the Brave search API join the leading names in AI and tech using the Brave search, API to power Agen Search.

Keep LLMs current with real-time data, train foundational models, and bring the best of the web directly to the leading edge. Get started today at brave.com/api. Yeah, it, it, it’s one of those things where like, uh, you know, when people think about this use case, uh, they mostly think about it around like, you know, uh, school and academics and then, and then, uh, homework and things like that.

But like, you know, I, I, I think for maybe for less technical users, there are these like cases with interviews where this is like, these types of tests are like actually pretty important for, you know, qualifying candidates. Yeah. When original coding interviews were developed. The, it started from let’s assess the skills we need an [00:04:00] engineer to have and try to simulate and see how many signals we can find in an interview.

And a big part of that was I would give you all the tools you need in order to succeed. How far can you go? So yes, we measured the way they practiced and prepared themselves for the interview, but that was by design and it was okay. But in the new world, you don’t need to measure that anymore. ‘cause knowledge starts to become, it’s, it, uh, it’s not how we say knowledge about Google.

It’s more around how knowledge become today. So. AI can solve all of those for you. You don’t need to practice as many questions as you can until you will get to a level that you can take any problem and, and break it into smaller problem and so on. So what we have left, and that’s when we start discussing about the [00:05:00] New World Engineering will.

Look at, we started to speculate about the world that that is right now. Started to define things like what is the difference between traditional engineer and, and AI native engineer or, um, applied AI engineer. There’s tons of names to that, but how coding will look differently and. Again, remember the year it was very early.

It’s not like we could have, look what happens around the industry or predict something. It was very much speculation and that was also part of the challenge here. A lot of engineers who are in metaphor a while and they’re like, we know how it is, how it’s going around here. I mean, ai fine, but, but. It’ll pass.

Not, not like you are speculating. It’ll go. So a lot of, [00:06:00] um, struggle around bringing people towards the, the mindset between shadow and it’s not like I had confidence that. This is where it’s gonna go. I was mistaken in a lot of things that I assumed we were gonna be there, but this process of coming back to the roots, reassess the skills, try to understand how the new skills will look like, and then try to understand , how can we evaluate that in an interview?

What should we guide the interviews to look for? And, and all, all of those other angles of it. was that the major point of resistance that you ran into kind of like getting people to, you know, look past it, maybe becoming like kind of a fad and, and being more of a, breakthrough type of, shift or were there other like types of resistance you had to work with, in order to make this happen?

So, yeah. The, the, the first one was resistance to you to adopt AI to begin with. and it [00:07:00] was from people in between in inside our team. So they were focusing more on people using this tool to cheat. Uh, they are average, uh, engineers and we, if we’re gonna encourage them to use ai, we’ll have average engineers who can, basically all they can do is park, and those are the engineers we’re gonna hire to the company.

And then at some point, this evolved a little bit when we started to see to add restrictions to it. Okay. This part of the skill is not only the prompting side, it’s also the validation side and how can you see, and then the, we started to see there was resistance about which skills exist, do we wanna add more skills or not?

I think at that point, I, I don’t wanna even call it resistance. It was more just lack of clarity. We were still figuring out things. like adaptation, right? it also makes me like kind of wonder too, like were, were people that are going through the process then kind of like sharing with others like what the [00:08:00] process was.

Is that kind of part of the challenge too, or is that you’re kind of having to update the or or adapt the curriculum to a bunch of different types of, . I don’t know, threats is the right word, but just like, , uh, challenges, right? With how AI is kind of, you know, people are, are, are using it in ways that, to try and kind of, get past the test.

But, , are people also sharing these things more broadly in the developer community when they happen? I’m just kind of curious ‘cause I think, uh, a lot of folks that aren’t devs that might be listening might be carrying curious. everything I was sharing right now is just like the first step.

You go and you develop and you have an assumption, okay, this is how an interview gonna look like. And then you try in, within the small team mock interviews and, and you sharpen that. Now you gather into something that you feel enough confident that, okay, those, this, this thing can actually measure skills.

And then. You have the part where you go and check that on in production. and what we started at the beginning, engineers were already hired into meta. , We [00:09:00] run them through the same process. It’s kind of like a mock interview. I see. Yeah. And then, and then we check that, um. Then you’d need to start and scale.

So you need to train all the rest of the interviews, or at least a significant group in meta to interview to those interviews. And the training has two phases. It is the mindset part where. You help them catch up with all of those debates we had. , And then you have also the technical, practical part.

‘cause many , of our engineers didn’t have those skills that they were expected to measure by. So this is all, all these things is going in the background and then you have the community outside. And you see so many articles about how Meta is interviewing ai.

I, I was reading articles that Meta is using AI in order to interview candidates. It’s like, no, you got it all wrong. [00:10:00] So, yeah, A lot of ion also in there. Yeah, I can imagine. I mean, it good to, because Meta is such a entity, you know, there, there’s such a force in, in the space and especially at the time too.

‘cause like they were doing a lot with open source and, things that were pretty, um, were pretty awesome to see at the time. , So, so is AI. Becomes more capable of, of writing code. How, how should companies rethink what they’re actually evaluating in engineering interviews?

So AI can generate code. This one we all agree on, but can it really write a high quality code? Can you trust AI to not create a new database out of nowhere? Can you trust it to not have logic in your front end not to have, access to the database directly from your phone, all of those challenges around it.

, So the practical things we used to measure that will always stay. Just if we used to measure your coding skills, then we still measure. We need to make sure that [00:11:00] the code produced this clean code, that it is code that can run in production, and this is code that you can maintain All the things we used to measure.

The only thing that changes now, AI is writing it, and we need to be to trust the engineer able to. And fix it if needed. Um, doesn’t have to fix it by hand, can back and forth with the, with ai, that’s fine. But this is a, a very important part of the validation process. And then also validation, just as we used to measure, it used to be a small problem you wanted, we wanted to make sure that you validate that this solves the problem as a whole, think about all those test cases you need to.

Check the same thing also with that code. So I think that overall, the assess criteria or their criteria we have, or we have for high quality code, a code that can run [00:12:00] production. This didn’t change. Um, the, the tools is a bit different. Think about it like mentoring of a junior engineer.

Mm-hmm. Now you have a junior engineer in your team. They are super ambitious and you have this feature you want to develop. You, you will not just let a junior engineer to write tons of code and, and trust it. Right. You will make sure that they do that. Right. You will read through, you will see. This is the skill we need our new engineers to have.

Interesting. Very interesting. I mean, at this point now we’re 20, 26, right? , Are you able to easily tell the difference between somebody who’s genuinely skilled at like writing code, versus someone that’s just good at like prompt engineering or, or, or kind of guiding, or does that even really matter that much anymore?

I dunno if it matters. I mean, why, why isn’t that the same? So, right, right. So generally good in prompting is basically means that you, [00:13:00] you, you know how to give the right context, you know , which context to give at, to which model, maybe to under, to know the differences between the different models.

Fine. There is always the other side of validate the outputs and, and see that things works there. Um, so yeah, definitely good at prompting, but uh, they have to be a good applied AI engineers AI native. Yeah, I mean, I think it’s just one of the signs of how like, kind of transformational it’s been when uh, that’s kind of how fast it changes.

Um, uh, what, what qualities have become more valuable in engineers because of AI and, and not less. Um, so leadership qualities are a bit more from being able to find the right problems to solve, uh, ‘cause now AI can do anything, but I, I’m gonna waste tokens. But does that solve a real problem? [00:14:00] And all the way to know which questions to ask?

So how to break the problem into pieces or to. I mentioned validation and, but, but this is also something, you know, the, if I go back to that example about mentoring, you don’t have junior engineers mentoring other junior engineer, right? It is, it becomes significantly more common as you become more and more seniors.

And that’s why I said those leadership, uh, engineers. Those leadership qualities. Mm-hmm. That makes sense. Yeah. And I think it’s just, uh, it becomes like a compo composing thing, right? Like, um, where, where you’re, like you said with, with breaking the problems down. I think that’s a really good way of putting it.

Um, you know, are there interview questions or signals you think companies should stop relying on altogether in this new environment? Everything that is traditional interviews, you know, the lead [00:15:00] code questions, uh, or any other website that failed so that, um, if you are trying to force your candidates not to use a tool that everyone in the industry is using, something you’re doing is wrong.

Interesting. Yeah, no, that makes a lot of sense. Just throw out the whole playbook. Right. Um, it, it’s practically what we did. We, we worked the whole, there is no interviewing meta at the moment that looks how it used looked two or three years ago. That’s good. That’s probably like I would imagine in this environment for sure.

I mean, I, well and meta like serve billions of people around the world. Um, how do conversations around privacy and responsible AI shape engineering decisions at that type of scale? Because I think very few are at that scale, you know? Mm-hmm. Um, so we did it with culture. Like [00:16:00] privacy or security, cybersecurity.

It’s, it’s very similar topics. You cannot keep one org and say, okay, this is, they, they’re responsible for privacy. They will make sure that, uh, all our product are privacy safe, and, and the rest you can do whatever you want. And then that would be, oh, there are some things in,

Reliability. Reliability. It is privacy, it is cybersecurity, and it’s also, been ai, safe. You have to train everyone and you have to make sure that this is everyone’s problem. And. It’s not like we don’t have a privacy group or all, all, all of those divisions that responsible for those things we do, but they are just yet another layer to protect it.

It’s still the engineers who are accountable for the code, the software that, the components, those are the engineers who are comfortable for all, all of those aspects. [00:17:00] And yes, privacy group can be at another layer. That makes sense. , What type of role do engineers play in kind of identifying potential privacy or safety concerns before AI features ever reach users?

Because I would imagine, , they’re at a vantage point where, you know, a a lot of others aren’t. And, I would imagine you gotta have to kind of imagine different things playing out, right? I mean, is that fair? Yeah. So basically the validation part. I, I mentioned that a bit earlier, but AI can write everything but , it’s not the one accountable.

So every mistake that, and, and we keep reading that right across everywhere. So it is, I don’t know, an AI agent wrote something, put it if in production and, uh, an outage. But. Practically it, it, it’s not the AI agent, it’s there is an engineer here who chose to put that code into [00:18:00] production.

Yeah. . Like the whole validation step and just kind of making sure that the right parameters are set, I would imagine. And I think this, it makes a lot of sense too with what you were talking about earlier around like the leadership qualities and being able to kind of, think about it, how, you know, this is how the machine would do it versus, making everything based off of what a person would do.

I guess, um, it’s all connected. You know, if you have the right culture and, and from day one, you train your engineers to be sensitive to, those are the core values you have in your company. And the, the system has to be reliable and in privacy is important and all of those angles, then this engineer will.

Will not let code to land in production , if it, they didn’t go and check and all of those things. And then eventually when you integrate AI and, and start allowing engineers to use ai, same thing. Keep repeating. You have to make sure you are the one accountable. Those are [00:19:00] the things, you are the one that knows it, it, it has to be in the core of the, of the culture of each engineering the team.

Mm-hmm. That makes a lot of sense. Yeah. I, I think, you know, and, and Meta’s been kind of a major advocate for open source AI models. What advantage do you think that approach brings for developers and innovation from your point of view? Um, so the thing is that you cannot really become an applied AI engineer and engineer who truly knows how to work with the different models and how to work with them without understanding how it works.

I mean, I was talking here very on the surface, on, on think about it like, like a mentor and how you’re gonna coach mentees and so on. But the, the, the strong engineers I seen are engineers who understand the bits and bytes. They understand exactly how the model operates. They understand what LLM means.

They understand what [00:20:00] token means. Not token equals mold, but what token is, um, and. Because of that levels of understanding you are able to do, to do better job. Now we saw the very similar things, uh, what with everything that happens with PyTorch. Um, so that’s kind of like a way to create a talent. One way can be that we need to hire that talent to hire people that we see potential and train them and help them, uh, become what, what we need.

But then another way is to. Give access to those resources, to the industry and, and, and let the industry train them.

It’s one of those things where, you know, learning from, from the community, it, it can be, especially when you’ve got the reach that , that meta has, I mean, the community is kind of part of the, an essential part of , what you all are doing. At least in the culture and in society too.

We’ve talked a lot about. What [00:21:00] AI can do. Where do you think humans will continue to have the biggest advantage in, in the process? I think it these things related to communication. Maybe soft skill, public speaking, storytelling, um, see, see what happens around in LinkedIn that, that can be a great example.

Uh, people are tired of AI generated posts. We had a period in time where everyone were posting posts that looked the same. Um, absolutely. It feels like it is still a lot of, I really questioning whether even how many people are actually in there, you know, on some days because it’s so exactly very repetitive.

Yeah, yeah. But then, which, which posts gather your attention. It was those with the real content. Suddenly posters are not perfect. Suddenly spelling mistakes became, , something that, you know, it, it, it’s that people are thirsty for human connection. When COVID hit and everyone went to, everything was webinar and [00:22:00] everyone, uh, can, can walk on Zoom and suddenly remote was the thing.

And I’m not saying it didn’t live in Mark, of course it did. But when you were talking with people back in 2020, people were like, , we don’t need to leave the house anymore. And now all of a sudden there are conference and people do want to come. And meet a person in person and see them and it becomes something.

So I feel like we’re gonna see a certain version of that. Exactly. With ai, now everyone tells us AI can do anything. Right? Right. But there is some certain of like humanity and emotions that AI cannot do. Yeah. Yeah. And it’s a tool too. I mean, I think, uh, uh, it, it might actually like, make us all appreciate the human things a little bit more, like you’re saying.

‘cause people spot patterns, like better than, than people realize too. And, and even if it’s the, the quality of what the patterns are is improving. There’s an authenticity around like [00:23:00] human what. What’s real, kind of. Mm-hmm. It seems like, um, yeah. Well, we’ll, well, we covered a lot. Thank you for joining. I, I, where can people find your work or follow you online? Um, so the, the magic about my name is that it is very unique name. Uh, there is one at Meta. Um, excellent.

, I’m in LinkedIn mainly. Well, we’ll link that in the show notes And yeah. Denise, thank you so much for, for making the time to join us today. I really enjoyed the conversation and, and, and your point of view. I think it was really eyeopening and, , and I think our audience is gonna appreciate it too.

Thank you. Thank you for the time. It was a great conversation. Thanks for listening to the Brave Technologist Podcast. To never miss an episode, make sure you hit follow in your podcast app.

If you haven’t already made the switch to the Brave Browser, you can download it for free today@brave.com and start using Brave Search, which enables you to search the web privately. Brave also shields you from the ads trackers and other creepy stuff following you across the [00:24:00] web.