Luke: [00:00:00] From privacy concerns to limitless potential, AI is rapidly impacting our evolving society. In this new season of the Brave Technologist Podcast, we’re demystifying artificial intelligence, challenging the status quo, and empowering everyday people to embrace the digital revolution. I’m your host, Luke Moltz, VP of business operations at Brave Software, makers of the privacy respecting brave browser and search engine.

Now powering AI with the Brave search. API. You’re listening to a new episode of The Brave Technologist, and this one features Andrew Rabinovich, who is the VP head of AI and Machine learning at Upwork, and a prominent figure in the field of computer vision and machine learning. He’s known for his work in deep learning, particularly in applications such as object recognition, seen understanding, augmented reality, and multitask learning.

In 2020, he founded Headroom an AI video collaboration platform, which was acquired in 2023. Currently, he leads AI and engineering at Upwork. In this episode, we discussed job security and the age of AI and ways the [00:01:00] freelance economy is already responding to its opportunities and challenges, ways that AI agents will turn workers into managers managing the machines, human-centered ai, and the type of solutions Upwork is building to empower their community amidst ais and certainties.

And now for this week’s episode of The Brave Technologist, Andrew, welcome to the Brave Technologist. How are you doing today?

Andrew Rabinovich: Excellent. Happy to be here, Luke.

Luke: Yeah. Yeah. Thanks for joining. I’m super interested in this conversation. There’s gotta be a lot of interesting things going on at Upwork now.

What’s the most exciting project you’re kinda working on right now that you can share with our listeners?

Andrew Rabinovich: Sure. So I joined Upwork about a year and a few months ago when my startup headroom that was working on video conferencing was acquired. And the main reason for choosing Upwork was because. In the last probably 10 years now, I’ve been working on this concept of human-centered AI as opposed to general artificial [00:02:00] intelligence.

And as you know, Upwork is the world’s largest freelancer marketplace, which I find to be the essential components to achieving a GI through this human machine integration.

Luke: Awesome. how do you see AI transforming the freelance economy? What kind of opportunities are there and challenges for independent professionals?

Andrew Rabinovich: We can step back and think about sort of step function changes in evolution in general, starting from, you know, steam engines and electricity and things like that. And, you know, if you’re a farmer. You do everything by hand with the invention of electricity. It’s not that people stop farming, they just start farming better, more efficiently and effectively.

Right? So this, we think that the same thing is going to happen in all digital work, which is what Upwork helps to facilitate. And that is that freelancers will be able to do much more, much more complex work with. Partnerships with ai, it’s not that a freelancer gets replaced [00:03:00] by a machine, it’s that a freelancer with a machine can achieve much more.

Luke: Got it. Got it. You’re kind of touching on on a good point I was gonna come around to with job security and I mean, obviously Upwork is engaged with a lot of freelancers. Like are there ways that they’re using AI with the platform now that are helping to make their efforts more productive or, or anything you can share on that front?

Andrew Rabinovich: There are two components that relate to your question. First is, are there tools on Upwork that help freelancers find work? And the second question is, are there tools on Upwork that help freelancers collaborate with ai? From external sources.

Luke: Mm-hmm.

Andrew Rabinovich: And the answer to the first question is yes. There is a meta agent inside of Upwork today that’s called uma, that stands for upwards mindful ai.

That helps both clients and freelancers either find [00:04:00] people to help do the work or for the freelancers to help them find the work. Now to the second question, we’re actively working on it and I can’t talk too much about it. In the open. We’re working on allowing freelancers to interact with agents and, and there’s like, 20, 25 is the year of agents, so to speak.

A lot of people define agents in very different ways, but I, irrespective of the definition, a interaction between a human and a machine is. Taken place today very actively. Like the simplest form of it is prompting, right? You go to chat, GPT, you ask it questions, it gives you answers. That’s an interaction between a human and a machine, albeit fairly primitive, right?

So if you assume all possible things that can be done on Upwork from design, coding, writing, music generation, search engine optimization, like. You know, podcasting, like anything you wanna do to all the possible agents that are going to be available in the world. We think [00:05:00] about what is the platform level interface that allows all these freelancers on Upwork to interact with all these up and coming agents that are about to help.

So that doesn’t exist on the platform to have, but there’s more to come there.

Luke: No, it’s awesome. Yeah. It’s really cool to hear kind of how you guys are mindset, you know, around this and how you guys are looking at it. Can we drill down a little bit into this concept of human-centric ai?

Andrew Rabinovich: Yeah.

So it’s a fairly. Loosely defined term, but a few folks, including Jan Koon and Fei FEI Lee at the Human AI and city at Stanford talk about this. And the idea is not to replace people, but to amplify them, right? We all want to be superhuman in some regard, and this will allow us to do that. A simple analogy is we’ve built ai.

To beat humans that chess, and now it go, right? So the question is you don’t want to play against the computer anymore because the computer will, will, will always win. But is it [00:06:00] possible for you to play chess with a computer? Against another human with a computer. Arguably it becomes a different game.

Maybe it’s not called chess anymore, but it’s still something net new, right? So same thing here. We want to build AI that doesn’t like building AI in the realm of. Outside of humans is a bit awkward because we don’t know what objectives to give it, right?

Luke: Mm-hmm.

Andrew Rabinovich: Like if on earth we wanna say, help us solve longevity, you know, hunger strikes and like lack of resources for the whole world, these are very concrete.

Tasks that we can give to the AI But these are things that involve humans and empower humans and better humans, right? If we talk about things that happen on the moon that we have nothing, no idea about, then that would may potentially be general intelligence, but not related to humans.

So I think the things we talk about are how do we build artificial intelligence that. Works [00:07:00] in the realm of human beings in the same way. Why is it that when we build humanoid robots, why do they look like people? Because the world that we build these robots for. It’s built for people. Therefore, these robots must resemble some kinematics and other capabilities there that are consistent with humans so that they fit into the frame of reference that they built for, right?

So, this whole story started with this movie called Her. I’m sure you, you, you’ve seen it. And the goal was to really create, not an assistant that would take tasks or directives from people, but a companion, someone who would be a level footed with a human. And to recognize that humans are very good at certain things.

We’re good at emotions, we’re good at intuitions, we’re good at feelings. Machines on the other hand, are also very good at things, but are fairly orthogonal to what I just mentioned, right? They can [00:08:00] work forever. They have no issues memorizing everything. They have universal knowledge, things like that, but they don’t have an ability to infer or have built intuitions or build relationships that are more.

On the emotional level than purely information gain, right? So if you combine the two together, then you’re able to really get to the next level of intelligence, call it general intelligence, or not like the tags are irrelevant, but the net new capabilities of human machine symbiosis is next level in my understanding.

Luke: Interesting. What do you think the biggest impact of, of kind of this a GI is gonna be?

Andrew Rabinovich: So, the biggest issue today with AI in general is that it’s not gentech, meaning that it doesn’t have an ability to decide what to do. [00:09:00] Nor does it have an ability to decide when the job is completed adequately. So all the mechanics of things we can say, go figure out where to drill the next oil hole, or go figure out how to build the next rocket.

Like these kinds of things machines can do and they’ll get only better. I’m very confident of that, but figuring out what to work on and figuring out when has. The solution then delivered to the desired results. that’s not obvious. So understanding like a, there’s a lot of what we call in, in sort of the human world, a lot of institutional knowledge mm-hmm.

When it comes to work or relationships and so on. But there’s also a lot of institutional knowledge that’s not written down.

And machines learn from data. If there is no data, or if the data is not at super high volumes, then learning becomes very, very difficult. Then that’s why we have [00:10:00] things like we’re human in the loop.

Reinforcement learning approaches, right? Asking people for answers every single time is possible, but it proves to be very expensive and inefficient. Yet there is this sort of. Insight and wisdom that humans possess that I don’t think machines will have in the near term. So an ability to solve very practical problems will obviously come around, but then there’s also these complex social constructs that will require this pairing of humans and machines.

Luke: No, it makes sense. It makes sense. And Upwork seems like a really interesting kind of arena for this given, you know, freelancers are both kind of, I, I would imagine they’d use you guys for like, you know, not only like kind of acquiring work, but also doing the work, right? So you can kind of. Have that kind of feedback, Luke, I would imagine, are there areas that you’re, you’re seeing the AI use really take off on the freelance side that are interesting or is it just across the board right now?

Because I’d imagine you’re getting demand everywhere kind of [00:11:00] thing.

Andrew Rabinovich: That’s a great question, and this very much depends on the quality of the freelancer.

Take software engineering, for example.

There are things like cursor copilot, reflection. There’s a ton of these new tools that allow that, like write code for you.

Like I use them all the time and I write like 10 times more code than, than, than I have in the last 10 years probably.

Luke: Wow.

Andrew Rabinovich: If you, you’re a very advanced software engineer, you will learn how to use these tools and some do. There are still some freelancers who believe that, you know, you open a blank terminal with VI and you just start writing code from scratch and everything is manual.

Can they produce best possible code? Possibly. Will it take forever? Yes. Will it be very expensive? Yes. So what we’re starting to see now is that. There is a very soft but a bifurcation between those who have adopted tools and know how to use them, and those that are still doing things in the classical way, if you will.

Luke: Interesting. As a customer, a user, [00:12:00] right. There’s probably a whole bunch of ways that you can start to quantify these advancements too, and, and you know, hey, you’re saving this much time, or you’ve gotten this much, much of an increase for sure. Are you guys working on metrics around that?

Andrew Rabinovich: Oh yeah, yeah, yeah, yeah.

Absolutely. So you wanna measure things across. Three domains. Number one is obviously quality, right? Because when clients come in, they hope and they request certain amount of quality. Although the way Upwork is structured today, we are mainly responsible for finding a match. But whether that match works out or not, that happens off the platform.

So the work happens off the platform after quality. There’s obviously cost ‘cause everybody wants the best stuff, but cheaper. Then there’s time and there are very interesting metrics around how you can achieve the same quality in a much, much shorter amount of time. Hence, the price goes down dramatically.

And it’s a good example for that is, for example, machine translation.

Luke: Hmm.

Andrew Rabinovich: If you wanna [00:13:00] translate from French to English, you go to Upwork and you ask, the search engine to find you a translator who speaks both and they can do the work, and you can start seeing that there are freelancers who are like, okay, I’ll take the work for you.

And in the background they go use chat GPT to do the translation, and then they just edit. And then they’re able to return the work order magnitude faster than those that who do this payment.

Luke: How is it like measuring quality too? Are there pure metrics you can get from that or is it like there’s still a subjective like element where you guys are kinda making assumptions on your side?

I’m just really curious.

Andrew Rabinovich: Sure, that’s, that’s a great question and the answer to your question lies from the news that came from deep seek. Ironically, hmm. There are tasks that are deterministic, for example. Code completion, or mm-hmm. Compilation, right? You don’t need a subjective metric to determine if the written code [00:14:00] compiles or not.

You just compile it and if it compiles, then it works. And if it doesn’t, then it doesn’t, right? So what Deep, deep seek did is they said, presumably, we don’t have a lot of compute resources because we’re not a US based company. We need to do this fine tuning. Post training process, that usually requires a lot of modeling of models, evaluating models like LLM as a judge type thing.

But those models are very, very expensive to train the judge models. Mm-hmm. And it’s expensive to run them. So once we have a proposal from the the model that we’re training, rather than having a LLM as a judge to evaluate the results, let’s just make up a bunch of rules. That are deterministic, and those take constant amount of time to run, hence cheap.

And then for a lot of the tasks that we want to fine tune our models on, we’ll just use that and assume that that works, right? So same thing applies here. When it comes to [00:15:00] any generative non-deterministic tasks, things are very, very subjective. And aside from getting human expert consensus, it’s very difficult to derive any kind of heuristics.

I mean, it’s very difficult to derive metrics that are not heuristics.

Luke: Mm-hmm. Mm-hmm. Right.

Andrew Rabinovich: Because everybody can just say this is subjective and that’s subjective.

Luke: Mm-hmm.

Andrew Rabinovich: Fortunately, a lot of the work in digital work domain. Is comfortable with that because it’s been like this all along. Even like in fairly deterministic tasks, like writing code, people have different coding styles.

Right? If we, as we expand into this human machine cooperation, a client can come in and say, I need, I want this type of code written, look in my code base and mimic the coding style that I have. So then the machine can do this. And if the exposure of human and machine is front and [00:16:00] center, then it’s very easy structurally to sort of propagate.

Coding styles all the way to the agent, and then have the freelancer evaluated and say, yes, this is written according to the coding styles of Google or whatever, and then it moves forward. But if you do it in an adverse case where you just give the work to a freelancer, the free freelancer may look at the code that you provided as an example and write something that they’re used to doing, and then it’s very difficult to get ’em to change it because unlike a machine like.

The human’s been trained to write code, write code in a certain way and it’s very hard to get ’em to change.

Luke: No, it makes sense. It’s really interesting. Do you all work closely with like testers or freelancers that are kind of like, you know, in like a beta or or you test cohort that that can give you feedback on these things and what are they excited about?

Andrew Rabinovich: We are in the process of doing that and. There are freelancers who are eager to [00:17:00] adopt new technologies because they see it as an opportunity to get more work.

Luke: Right?

Andrew Rabinovich: Right. And the way I think about it is that as agents become more and more capable, you can think of the amount of work. That exists today on Upwork can grow exponentially more because not only would you be able to solve much, much more complex things, you can come to Upwork and say, build me an open source version of Facebook.

Luke: Wow.

Andrew Rabinovich: Now, if you say that an on Upwork today. A, you’ll never be able to probably find a single software engineer who would sign up for this. ‘cause this is crazy. Right? And even if they did, it would take like millions of dollars and years of life, right? Sure. If you are able to find human machine.

Collaborations then, like I can imagine this being done in like 30 days for, you know, a hundred thousand bucks. Mm-hmm. That’s the, on the [00:18:00] complex side of things, but on the simple side of things, we can start thinking about agents being able to solve almost these like microtask in with like. Insane frequency.

Luke: Mm-hmm.

Andrew Rabinovich: Right? As a, as a business or as a consumer, you can go to Upwork and you can just say, I want this and I want that, and every project would be like 10, 50 bucks. But there will just be so many that a given freelancer will be able to manage a whole scope of hundreds of little projects that they may do with different agents, therefore, increasing the demand for their time.

Luke: Yeah, I was gonna ask, I rarely hear people say things like insane frequency. So like, you know, so, so imagine somebody doing tens of projects can scale up to hundreds. Is that, is that kind of

Andrew Rabinovich: Yes. So the, the interesting bit is that their direct involvement or time spent on each of those projects will go down.

Luke: Mm-hmm.

Andrew Rabinovich: Because then they, they will be in this mode of [00:19:00] managing slash verifying

Luke: mm-hmm.

Andrew Rabinovich: Or guidance as opposed to. Doing, you know, and you, and you can sort of see this with people as well, right? Like, let’s say you are a software engineer and then you’re an individual contributor. You’re responsible for, you know, writing the, the design documents to actually coding, to testing, to deployment, whatever.

Then you look at someone who manages a thousand person organization, they don’t do any of it. Mm-hmm. But they make sure that everybody gets their stuff done. So the same type of. Orchestration, if you will, will happen between humans and machines until there are machines that become more advanced than humans.

But I think we can leave that conversation for the next dialogue.

Luke: Yeah. So going from herding cats to herding agents kind of scenario, right? Correct.

Andrew Rabinovich: Yeah. And, and the, and these agents. They’re very interesting because they don’t have any kind of social issues. They don’t have any kind of preferences, they don’t have any opinions.

They just [00:20:00] like do things, right? Mm-hmm. So then it becomes much easier for you as the orchestrator to get really efficient at it so long that there is the right abstraction layers, if you will. Mm-hmm. To connect them and to make the interfaces efficient.

Luke: That’s awesome. I’m wondering too, it sounds like this would be helpful in, in project management and, and even the direct work itself.

Like what about on like the business side? Do you see agents helping with contracts, negotiations and things like that, agent to agent negotiation types of things in the future? Or, or is there just a, a human part of this that’s just gonna always gonna be there?

Andrew Rabinovich: So with humans, it’s literally a marketplace, right?

So the way things work today is that a client comes on the platform, meets with Uma. Tell Zuma what what they want describing the project. Uma puts together a project plan. Then it uses that project plan and the search engine to find a set of possible candidate freelancers.

Luke: Hmm.

Andrew Rabinovich: Then it helps those freelancers write [00:21:00] pro job proposals on their behalf to the client.

Then almost switches the hats and helps the client pick the right candidate and then. The job sort of starts, and then the UMA will facilitate any kind of negotiations and so forth. With agents, it’s, it’s trivial, right? There’s like per token price. Mm-hmm. And there’s nothing to negotiate, right? So long that you have the right interfaces.

APIs, for lack of a better word, then everything is super seamless. You can forecast. Ahead of time how much things will cost because it’s fixed, right? And there’s always going to be a competition where as you see this today, a new model comes out and they’re like, the price per token is a cent cheaper, or the context window is 10 x bigger.

You know? And just through this natural competition, you, you get this evolution in quality and efficiency.

Luke: Wow. Oh, that’s awesome. I mean, it sounds like pretty comprehensive now. Like it’s, it’s interesting to see kind of, [00:22:00] or imagine where this will go.

Andrew Rabinovich: It’s getting there, you know, it’s not a, an overnight change.

We have to remember that Upwork’s been around for 20 years.

Luke: Mm-hmm.

Andrew Rabinovich: You know, so on one hand, internally we build a lot of tools and. Innovations in tech. But on the other hand, back to the human component, there’s a lot of legacy momentum, you know? Mm-hmm.

Luke: Mm-hmm.

Andrew Rabinovich: People are like by inertia, people do certain things a certain way.

Sure. And despite having all the latest and greatest tools available, they still need time to adjust to them and to forego all the sort of things that they have verified in the past that work or not.

Luke: Sure, sure. Keep culture ethos kind of getting exactly that fit within the org. Yeah, that makes sense. I mean, and that’s one of those things that’s, I, it’s interesting having 20 years of, of just the data and experience and, and maturity in the market, and then being able to apply these tools in that context compared to like, you know, being a freelancer or a startup or, or whatever, like it is super interesting.

But sure. A lot of [00:23:00] freelancers, those that haven’t caught the bug yet with this, but are interested, is there anything you recommend or, or, or for, for them getting into kind of. Getting prepared to get into this AI driven job market or, or mindset or tips or anything like that.

Andrew Rabinovich: I would recommend figuring out which parts of their own work they can outsource to the agents.

Luke: Hmm.

Andrew Rabinovich: And then start looking at much more complicated tasks that they themselves can solve with the present presence of an agent. So if, if you used to build websites in Tailwind or whatever, react native or whatever, start thinking about building much more dynamic web applications that incorporate user interactions, search engine optimizations, all kinds of things.

So you can actually tackle much larger project as opposed to just doing this one thing because you rely only on yourself.

Luke: Interesting. That’s awesome. And I know we covered a lot today. Is there anything we didn’t cover that [00:24:00] you want our audience to know about?

Andrew Rabinovich: I think we did speak about a lot of things.

One aspect that I am particularly excited about is there needs to exist an interface between humans and machines that allows us to. Interact in the most efficient ways because if we don’t do that, then it won’t work.

There are startups and larger companies that are thinking about this, and I think this will be the next sort of missing piece of this a GI puzzle, if you will.

So the machines are getting smarter, the need for them to interact with expert humans. Just like you heard from open AI that now you can have a PhD level, Chad, GPT, or a master’s level, Chad, GPT. So it clearly, like the expertise of people go beyond like having an internet connection on a pulse. Like people’s knowledge is really valuable.[00:25:00]

That continues to grow. People go to colleges, although, you know, it’s questionable these days and, but the interface between the humans and machines such that they can do this. In a very organic way. I think that’s the missing link, and I’m looking forward to seeing that come out as soon as possible. That kind

Luke: of breaking it out of the, the chat prompt, right?

Like getting it Exactly. More, more integrated. Yeah. The,

Andrew Rabinovich: the prompt is kind of a very one dimensional version of that, but we need it way more where one of the things that has been very successful, so I come from a computer vision. Background and one of the early on lessons that we learned is that algorithms involving learning by example have always been very, very successful.

Hmm. Right. And this reinforcement learning with human in the loop is just another example of this, right? tell me whether you like A or B more, you don’t have to tell me why. Just stuck rank them. Right. In computer vision, it was like, here’s [00:26:00] an image. With recognized objects.

Here’s another image. Can you like recognize any objects? I won’t even tell you what these objects are, but they have to be similar to the objects in whatever dimension, color, shape, construct, family, like whatever. Can you recognize them? And we, we’ve been able to train machines to do this like 10, 15 years ago.

So this learning by example is critical and. Machines have to start learning from humans by example. Not at the training phase, but during actual inference of problem solving.

Luke: Interesting, interesting. Yeah, this is great. Where can people follow you to learn more about what you guys are putting out or, or just to see what you have to say online?

Andrew Rabinovich: So there is a, an AI blog at Upwork, and whenever we publish things there, I usually cross reference it with LinkedIn. Cool. Outside of that, we do a lot of foundational research and when it gets published, you can see it on on my scholar page.

Luke: Awesome. Well, Andrew, this has been a really enlightening conversation.

I [00:27:00] appreciate you making the time to share about Upwork and in your work and point of view with our audience. And love to have you back too to check back in on things and see how things are going.

Andrew Rabinovich: It was great to talk to you, Luke. Thank you.

Luke: Thank you very much. Have a good one. Thanks for listening to the Brave Technologist Podcast.

To never miss an episode, make sure you hit follow in your podcast app. If you haven’t already made the switch to the Brave Browser, you can download it for free today@brave.com and start using Brave Search, which enables you to search the web privately. Brave also shields you from the ads trackers and other creepy stuff following you across the web.