Devin - The World's First AI Software Engineer

00:00:00: So it happened.

00:00:01: The world's first AI software engineer

00:00:05: called Devin was introduced by

00:00:08: Cognition AI, a company that, well,

00:00:11: builds AI programs as it seems.

00:00:13: For example, Devin, this AI software

00:00:16: engineer.

00:00:17: And that of course means that now the

00:00:20: end is near for us software developers

00:00:24: and engineers, right?

00:00:25: Well, maybe not yet.

00:00:27: And I'll share why I'm impressed or

00:00:32: interested by and in Devin, but why I

00:00:35: don't feel like it'll replace you or me

00:00:39: anytime soon.

00:00:40: And I'll explain.

00:00:41: But first things first, what exactly is

00:00:44: Devin?

00:00:45: What is this thing that was introduced?

00:00:47: Well, it's a program that in the end uses

00:00:50: AI models under the hood to solve

00:00:54: problems that are given to the program.

00:00:57: And with problems, I don't mean, hey,

00:01:00: give me the code for storing a file on

00:01:03: AWSS free with Node.js.

00:01:07: No, I mean, bigger problems, which then

00:01:10: would require multiple steps to solve

00:01:12: them and which would require logical

00:01:15: thinking to solve them.

00:01:17: I mean, these kinds of problems, like,

00:01:19: hey, build me an online webshop

00:01:23: website where users can authenticate,

00:01:27: add products to their cart, and then

00:01:29: proceed to checkout.

00:01:30: I mean, problems like this.

00:01:33: That's what Devin is meant to solve.

00:01:37: These kinds of problems.

00:01:39: And that, of course, sounds pretty

00:01:41: impressive or intimidating, depending

00:01:44: on where you come from, right?

00:01:46: And indeed, Cognition AI, the company

00:01:49: that built Devin, so that built this AI

00:01:52: software engineer, they shared a video,

00:01:55: an announcement video, and actually

00:01:57: they shared multiple detailed videos as

00:02:00: well, where they show what Devin can

00:02:02: do.

00:02:05: And they, for example, show that it was

00:02:06: given the task of analyzing the

00:02:09: performance of different AI models.

00:02:12: And then Devin went ahead and wrote

00:02:14: a program that would do that.

00:02:16: And in that announcement video, they

00:02:18: also showed which tools Devin is able to

00:02:21: use at the moment to solve problems.

00:02:24: Since it's meant to be an AI software

00:02:26: engineer, it has access to a terminal

00:02:28: where it can run commands to, for

00:02:30: example, create a new project or run a

00:02:33: project.

00:02:34: It has an IDE, of course, to write code.

00:02:38: And Devin has access to a browser to

00:02:41: research, to take a look at official

00:02:44: documentation pages, for example.

00:02:46: And of course, also, if it's building a

00:02:49: website, to test that website.

00:02:51: And then Devin goes ahead and in the

00:02:53: end, uses AI models or AI APIs behind

00:02:57: the scenes to solve a problem step by

00:03:00: step.

00:03:01: So it's issuing multiple requests to those

00:03:03: AI models.

00:03:04: The first request might be, hey, how do I

00:03:07: create a new web project?

00:03:10: And then with the response given, it

00:03:12: would execute the command that was

00:03:15: given to it.

00:03:15: And then it would continue asking the

00:03:17: AI models behind the scenes.

00:03:20: How do I use this framework or which

00:03:23: framework should I use for this?

00:03:26: What's the official documentation page?

00:03:28: It would then maybe go to that page

00:03:29: and pass the results it finds there back

00:03:33: into the AI model to understand the

00:03:35: results and deduct the next steps and so

00:03:39: on.

00:03:39: That's essentially how it works.

00:03:41: And in their announcement post or in a

00:03:44: video shared there, Cognition AI, for

00:03:47: example, also showed that Devin was

00:03:49: able to use these tools to, for example,

00:03:52: solve jobs on Upwork.

00:03:54: And in the announcement post on

00:03:57: Twitter, for example, they also shared

00:03:59: that they also used Devin to solve open

00:04:02: GitHub issues.

00:04:04: And there it was able to solve just a bit

00:04:08: below 14% of all the issues it was given

00:04:12: without assistance, which could be

00:04:16: considered a low number.

00:04:17: And I'll get back to that, but which is

00:04:19: actually quite impressive, especially if

00:04:21: you compare it to the numbers of similar

00:04:25: projects or programs, which are far

00:04:28: lower, as they also mentioned in their

00:04:30: post.

00:04:30: I think they mentioned that comparable

00:04:33: programs only achieved around 2%

00:04:36: without assistance compared to the 14%

00:04:39: of Devin, so way less, and around 5%

00:04:43: with assistance.

00:04:44: So clearly Devin is pretty impressive

00:04:46: regarding that, and it has some pretty

00:04:49: interesting capabilities.

00:04:51: So that's all great.

00:04:53: But still, I'm not shaking in fear.

00:04:57: I'm super interested in experiments like

00:05:00: this.

00:05:00: And I do believe that as developers, as

00:05:04: software engineers, we should embrace

00:05:07: technology like this.

00:05:09: And I did already talk about that in

00:05:12: another podcast episode.

00:05:14: So I think we should really be open to

00:05:17: that.

00:05:17: And we should use these tools to make

00:05:20: our job easier and to become more

00:05:22: efficient.

00:05:23: And I would gladly take an AI software

00:05:27: engineer as a coworker and outsource

00:05:30: tasks I don't want to do to that AI

00:05:34: software engineer.

00:05:36: But when it comes to Devin or other

00:05:39: programs like Devin that might come

00:05:42: up in the future, replacing me or you, I'm

00:05:47: not too convinced that this will happen

00:05:49: anytime soon.

00:05:50: And the reason why I'm not afraid is that,

00:05:54: of course, Devin does some impressive

00:05:56: stuff here.

00:05:58: But it's still far off the numbers, the

00:06:03: numbers of issues solved, for example,

00:06:05: that would be achieved by a human

00:06:07: software engineer.

00:06:09: And in addition, there also isn't too

00:06:12: much information available regarding

00:06:14: the exact issues it did solve.

00:06:16: We don't know how well it would, for

00:06:19: example, be able to build entire software

00:06:22: projects that are maybe a bit more

00:06:24: complex.

00:06:25: So that's all unknown.

00:06:27: And you would guess that if Devin

00:06:30: would be able to build super complex

00:06:33: applications, they would share an

00:06:35: example of that as well, right?

00:06:37: Now, you could say, it's just a matter of

00:06:39: time, because these AI models will

00:06:41: improve, right?

00:06:43: And you would be right to some extent.

00:06:45: But if we think about other areas where

00:06:49: AI is used, think about self-driving cars,

00:06:52: think about speech recognition.

00:06:55: And if you then consider when we first

00:06:58: heard about technologies like this, and

00:07:01: when we first started using such

00:07:03: technologies, which is pretty much 10

00:07:05: years ago already, and how far we've

00:07:08: come, then you would probably agree

00:07:10: that, yes, there are impressive demos

00:07:14: there and impressive use cases.

00:07:16: And there are some valid use cases

00:07:18: where AI might be used in our day-to

00:07:21: lives.

00:07:22: But you would probably also agree that

00:07:25: AI hasn't really brought us self-driving

00:07:29: cars yet, no matter if Elon Musk keeps

00:07:32: repeating that it will be available for

00:07:36: everyone soon.

00:07:37: If we think about voice recognition and

00:07:41: AI assistants that talk to us and that

00:07:44: really understand us, also not too

00:07:48: convincing in my opinion.

00:07:49: And therefore, I feel like we got some

00:07:52: interesting numbers here.

00:07:54: We have a very interesting demo here.

00:07:56: And Devin and similar tools might

00:07:58: absolutely become helpful tools very

00:08:02: soon.

00:08:03: I mean, we have helpful tools like GitHub

00:08:06: Copilot already, which also become

00:08:09: better and better over time.

00:08:10: So don't get me wrong, definitely, they

00:08:13: will play an important role.

00:08:15: They will do some of the work we need

00:08:18: to do.

00:08:19: But when it comes to replacing software

00:08:21: developers or engineers altogether, I

00:08:25: think we still have a very long way to go.

00:08:29: Because, for example, if we stick to the

00:08:32: 14% of GitHub issues that's an interesting

00:08:36: number, especially compared to the way

00:08:38: lower numbers of comparable programs

00:08:41: and models.

00:08:42: But of course, it's still far away from what

00:08:45: a human would be able to achieve here.

00:08:49: Maybe it would take a human longer,

00:08:51: but a human, a software engineer who

00:08:53: knows what they're doing, would be able

00:08:56: to solve all those issues essentially, and

00:08:59: not just 14% if they wanted to.

00:09:03: So that is an impressive number, but it's

00:09:07: still far away from what we need for such

00:09:10: a program to actually replace software

00:09:13: engineers and developers.

00:09:16: I also already mentioned in that other

00:09:19: episode, I already mentioned earlier, that

00:09:21: of course, the main thing developers do

00:09:24: in the end is solve problems.

00:09:27: Yes, we write code to solve problems,

00:09:29: and that is a skill we have.

00:09:32: But if you're honest, the problem solving

00:09:35: part and coming up with solutions, and

00:09:39: then writing code, refining and

00:09:42: improving code, that's the main thing

00:09:44: we do.

00:09:46: And that's the thing an AI will need to do

00:09:48: really well in order to replace software

00:09:52: engineers or developers.

00:09:55: And to me, it does not seem as if Devin

00:09:59: or other AI models or programs we have

00:10:03: out there are very close to replace

00:10:07: human developers when it comes to

00:10:10: that.

00:10:11: Now, I'm fully aware that there are a lot

00:10:14: of people out there who will absolutely

00:10:16: tell you the opposite.

00:10:17: And for them, it's very clear that AI will

00:10:21: replace essentially all of the high skilled

00:10:26: work humans do nowadays in the next

00:10:29: five years or so.

00:10:30: And of course, obviously, no one knows

00:10:33: what will happen in the future, though

00:10:35: they know at least if you read what some

00:10:39: people are writing there.

00:10:40: But anyways, obviously, no one knows.

00:10:43: But for the reasons I mentioned, and if

00:10:46: we take a look at other AI developments,

00:10:49: like self driving cars and so on, I still am

00:10:53: pretty confident and I feel pretty

00:10:56: confident about being a software

00:10:59: developer and about staying a software

00:11:01: developer.

00:11:02: But of course, I'm also interested in

00:11:05: hearing your opinion.

00:11:07: What did I forget here?

00:11:08: What do you think?

00:11:09: Where am I wrong?

00:11:10: Or where do you agree?

00:11:12: Definitely share your opinion.

00:11:14: And well, I see and hear you in the next

00:11:18: episode, hopefully.