The NPM Package Attacks - What We Need To Change

00:00:00: There's a good chance you already heard or

00:00:04: read about the NPM package attacks that

00:00:08: happened over the last couple of days, over the last week

00:00:11: essentially. There have been two major

00:00:15: attacks on the NPM package ecosystem

00:00:19: where collectively, packages with billions of

00:00:23: downloads, weekly downloads, have been targeted by

00:00:27: attacks that aimed to distribute and

00:00:30: deploy malicious code, steal

00:00:34: secrets from your machine, for example, from

00:00:38: CICD machines, and so on. Now, in case you

00:00:42: haven't really heard or read about these attacks, I'll briefly

00:00:45: summarize what happened but then, more importantly, I

00:00:49: wanna dive into what that means for you

00:00:53: and me, for us developers, because this of course is

00:00:58: a kind of attack and a scale of attack

00:01:01: that's ... well, I'm sure it has happened before but it's

00:01:05: not something we see every day and it has some

00:01:09: serious impact on the NPM ecosystem and how

00:01:13: we as developers should interact with those packages,

00:01:17: I'd say. But let's start at the basics.

00:01:20: It started essentially a week ago with

00:01:24: the first big attack which, uh, was less

00:01:28: impactful than the second attack to which I'll get.

00:01:31: But that first attack essentially worked by

00:01:35: getting access to an account, an NPM account

00:01:38: of the maintainer of one NPM package which

00:01:42: of course gets used by other packages and so on.

00:01:46: And that attack in the end happened through a phishing

00:01:50: email, so they used a phishing email to get

00:01:53: access to this package maintainer's NPM account

00:01:58: and through that access which they got successfully, they were

00:02:01: then able to deploy a malicious version of that

00:02:05: package, a package which came with some extra

00:02:09: code which in the end deployed a

00:02:12: cryptocurrency, uh, token stealer.

00:02:16: So the goal of this attack essentially was to get into

00:02:19: packages that are then used on websites to

00:02:23: steal cryptocurrency in the end from the visitors of

00:02:27: these websites. So the goal of this attack was not

00:02:31: to compromise the system

00:02:34: of people, of developer, uh, developers using

00:02:38: that package or of, um, machine

00:02:42: CID- CICD machines that deploy,

00:02:46: uh, applications using that package. That was not the goal of this attack.

00:02:50: The goal of that attack was to run on websites and to

00:02:53: target the users of those websites to steal

00:02:57: cryptocurrency, and that wasn't too successful.

00:03:01: Uh, from what I read, they were able to steal like $1,000

00:03:04: or so, but still what this attack shows us, of

00:03:08: course, how dangerous using

00:03:12: lots of NPM packages can be, and I'll get back to that

00:03:15: later, and how vulnerable the entire NPM

00:03:19: ecosystem can be. Because even though this first

00:03:22: attack wasn't too impactful, it was, of

00:03:27: course, a successful attack. And the

00:03:30: second attack about which I wanna talk shows us that

00:03:34: the threat can be much higher than what

00:03:38: resonated or what was done by that first

00:03:41: attack. Because that second attack which happened a

00:03:45: couple of days after that first attack which, as far as I know,

00:03:49: isn't directly linked to it as it seems, but I'm not 100% sure

00:03:53: about that. But that second attack was now not

00:03:57: about stealing cryptocurrency. That second attack was about

00:04:01: getting on your machine or the machines where

00:04:04: applications are deployed, CICD machines, and

00:04:08: stealing security credentials, credentials in general,

00:04:12: stealing AWS access keys, NPM, uh,

00:04:16: access tokens, GitHub access tokens, and all

00:04:19: kinds of credentials, uh, security keys, tokens

00:04:23: you might find on those machines. Now that second attack,

00:04:28: as it seems, as far as I know, did not

00:04:31: start from an phishing, from a phishing mail but instead

00:04:35: from a already compromised, um, NPM account from an

00:04:39: earlier attack potentially. Whatever it was, they were

00:04:43: able to publish a new version of a package,

00:04:47: ironically of a package that wasn't super actively

00:04:51: maintained, so that hasn't seen a lot of updates,

00:04:54: uh, in, in the past. But they now published a new version of

00:04:58: turns out, that package was used by enough other

00:05:02: packages to have an impact because this second attack

00:05:06: actually now was not about including some code that

00:05:09: runs on the deployed website. Instead, that second attack

00:05:13: was about running

00:05:16: malicious software in the end, a malicious script on the

00:05:20: machine where that compromised package was used in

00:05:24: a build process and then that script would

00:05:28: actually search for secrets, for credentials, for

00:05:32: access tokens on that machine and use those to

00:05:36: A, exfiltrate them, to share them with the attackers but

00:05:40: B, also to spread. So it

00:05:44: was a worm. The goal was to then use these access tokens, for

00:05:48: example, to publish the same malicious software on

00:05:52: other packages. So if package B used package A which was

00:05:55: compromised, then during the build process of package

00:05:59: was...... the chance that this malicious

00:06:03: code, which was in package A, would actually get access to the GitHub

00:06:07: repository to the NPM account of package B as well and

00:06:11: then continue spreading, and that's, of course, extremely dangerous and

00:06:15: it was and is an extremely dangerous attack because of

00:06:19: this worm, uh, like behavior of because it was

00:06:23: spreading and it was not just stealing those

00:06:27: secrets from one affected package, but it was using them to

00:06:31: affect more packages. And, therefore, this second attack

00:06:35: had a way larger impact than this first

00:06:39: attack. Ye-, because of this worm-like behavior,

00:06:43: this attack was able to successfully compromise

00:06:46: many hundreds of packages because all these packages, of course,

00:06:50: typically depend on each other and then you might just be using

00:06:54: one of these packages, but because of the packages

00:06:58: internally, suddenly you might be affected as

00:07:01: well. So how can you find out if you are affected?

00:07:05: Well, a couple of things. Uh, for one, you should, of

00:07:08: course, check whether your projects which

00:07:12: you built or where you downloaded, um, new

00:07:16: versions of the packages over the last week or so,

00:07:20: if these projects use one of those, uh, known

00:07:24: compromised packages, and I'll put some links below this, uh,

00:07:28: this episode, to help you scan for that.

00:07:31: So that's, of course, one thing, and when you look for these packages,

00:07:34: not look in your package lock, uh, in your package JSON file, but in your

00:07:38: package-lock.json file because that

00:07:40: package-lock.json file, that's actually the file that

00:07:44: lists all the packages used in your project, including the packages used

00:07:48: by packages and it shows you the exact versions used

00:07:52: there, which allows you to find out that if you are using one of the

00:07:56: compromised packages, if you're using one of the compromised

00:08:00: versions, which, of course, is an important information.

00:08:02: So that's one thing to do. Uh, in addition,

00:08:06: due to how that attack worked, it seems that it

00:08:09: exfiltrated, uh, those stolen secrets

00:08:13: mainly by creating a new public GitHub repository in

00:08:16: your account if you are affected, uh, which had this

00:08:20: name, uh, I, I keep on forgetting it,

00:08:23: shayhulut, which had this name. So you wanna check for

00:08:27: that and you wanna check your GitHub security logs, to which you also find a

00:08:31: link below, you can use it for your account to check

00:08:35: any action related to creating such a repository or

00:08:38: switching one of your existing private repositories to public because that's

00:08:42: also something this attack did and so on.

00:08:45: So these are steps you should take and again you'll find more detailed

00:08:49: resources below. That's one thing that

00:08:52: you should do right now if you fear that you might be

00:08:56: affected. But, of course, this attack also

00:09:00: brings up the question, what can we do to mitigate such

00:09:05: attacks or to prevent such attacks in the first place?

00:09:07: Of course, that's a pretty big thing

00:09:11: with m- with many building blocks because, of course, it all starts with

00:09:15: making sure that if you're a package maintainer that your NPM and

00:09:19: GitHub accounts are secure and are not easily compromised by

00:09:23: using two-factor authentication, though even that is not a

00:09:27: 100% guarantee because with phishing there are ways of getting

00:09:31: around that too with social engineering and so on, but it is, of course,

00:09:34: important. There are a couple of steps you can take

00:09:38: maintainer to help reduce

00:09:42: the, the risk of becoming a spreader of

00:09:46: software, but, of course, you and me, you might primarily

00:09:50: be users of packages, uh, and, therefore, the question is what

00:09:54: can we do? And it starts with

00:09:57: limiting the amount of packages you're using, I'd

00:10:01: say. I'll get to some other recommendations as

00:10:04: recommendation I haven't seen enough in all those blog

00:10:08: posts and articles and so on I read because there is a

00:10:12: tendency for some developers at least to

00:10:16: use too many packages. Um,

00:10:20: y- uh, it... and I get it. It's convenient, right?

00:10:23: Um, you can solve a lot of problems you might

00:10:26: during the development process by bringing in a

00:10:30: package that does what you're looking for.

00:10:33: Um, instead of writing your own code to solve

00:10:37: in a package and you're done with it.

00:10:39: And, of course, for certain tasks, that makes sense.

00:10:43: You're using React because you want a package

00:10:46: that, uh, makes it way easier to reactively

00:10:50: update the DOM and build highly interactive, uh,

00:10:54: browser-based web applications, so yeah, you wanna use

00:10:57: React probably. For authentication, you probably...

00:11:01: you'll wanna use a package. For your backend framework, you

00:11:05: probably wanna use a library. So we

00:11:09: all use them, of course. We, uh, we all probably

00:11:12: rarely build web applications totally from scratch with

00:11:16: just HTML, CSS, and JavaScript and no packages, though

00:11:20: that is actually a valid strategy for certain kinds of sites and

00:11:24: applications. Just saying. But yeah, we all use

00:11:27: packages. But do you really need a package for

00:11:30: checking whether a number is odd or even?

00:11:35: Not sure about that. And, of course, that's an extreme example, but you get

00:11:38: my point. Especially now with AI

00:11:42: available, it's so easy to quickly generate

00:11:46: code for problems you might not

00:11:50: know the solution immediately, like let's say you need to invert

00:11:54: a binary tree, not something you need to do every day,

00:11:58: applications, in most applications, but let's say you, you want

00:12:01: some efficient algorithm for searching or for inverting a

00:12:05: binary tree, whatever.If you need to do something like

00:12:09: can bring in a package that does it for you, you can sit

00:12:13: down and try to learn the algorithm and write it from scratch,

00:12:17: good exercise, or you can use AI to generate it because

00:12:20: exactly the kind of tasks that are super easy for

00:12:23: AI. Now I will say, AI-generated code is

00:12:27: not necessarily secure and AI also sometimes

00:12:31: wants to bring in extra packages, packages.

00:12:34: It can be lazy too. But still, uh, y- you

00:12:38: really can write more code on your own or with help of AI than you might

00:12:41: think. So, that's really one thing that's important

00:12:45: because there is this tendency f- by, for some developers at least,

00:12:49: to solve everything by adding a new package, a new library,

00:12:53: and I would

00:12:55: recommend to try as little

00:12:59: packages as possible. That is something I've done for

00:13:03: years. I like the challenge of

00:13:06: using, yeah, not a lot of packages, not a

00:13:09: lot of libraries. I like the challenge of solving problems

00:13:13: Now with AI, that's even easier. But of course, I'm also using

00:13:16: packages. And if, due to how those

00:13:20: packages work, that they all have their own dependencies,

00:13:24: would get affected by, uh, a worm like this, then we're, of

00:13:28: course, all in trouble. Uh, I don't even wanna think about that

00:13:32: happening. That would probably affect hundreds of

00:13:36: millions of machines if such a package would get

00:13:40: compromised. But yeah. So limiting the number of packages you

00:13:43: use is a good idea. Another thing you might wanna consider

00:13:47: doing is fix your dependency

00:13:51: versions. And I'm not necessarily doing that.

00:13:55: I'll be 100% honest here. I should be doing it.

00:13:58: I will be doing it in the future. I haven't done it, but

00:14:02: I've read it a lot related to these attacks and, of course, it makes

00:14:05: sense. We typically fix the major

00:14:09: versions of our packages. Maybe also the minor versions, but

00:14:13: not the patch versions. So in case you don't know, um, many

00:14:17: packages, or actually, well, pretty much all these

00:14:20: packages, um, on NPM have, like, a major, a

00:14:24: minor, and a patch version number in their

00:14:27: number and depending on how you configured your package

00:14:31: JSON file when you run NPM install and so on, you

00:14:35: either download a very specific version or the latest

00:14:38: patch or the latest minor or even the latest major

00:14:42: version. And in theory, it makes sense to

00:14:46: always download the latest patch version because

00:14:50: patch versions, depending on how the maintainer handles

00:14:54: in theory, they don't change the fundamental

00:14:57: nature or behavior of a package. They don't have breaking changes

00:15:01: they fix security issues or performance issues and so on.

00:15:05: So in theory, you wanna stay up to date with patches,

00:15:09: but in reality, as attacks like this show us,

00:15:13: was spread through a patch update, we might

00:15:16: not want to automatically use patch versions either.

00:15:19: We might want to fix our versions and then deliberately

00:15:23: choose to update our version numbers after carefully

00:15:27: making sure that the new version is safe, not just from a b- breaking

00:15:31: change or performance perspective, but also from a security perspective.

00:15:35: It is more work, but how much work is it to

00:15:39: clean up a compromised system, to rotate all your,

00:15:42: uh, compromised tokens and credentials and

00:15:46: mitigate that, uh, uh, that danger and,

00:15:50: uh, that damage that was done? So yeah, that's one

00:15:53: thing. Uh, if you're using PNPM as a package manager,

00:15:57: example, which I don't, you can even set a certain

00:16:01: setting, the minimum release age setting, to make sure that

00:16:05: you do, for example, automatically download patch versions.

00:16:09: But that the package has to be a

00:16:13: minimum number of, uh, days old, uh,

00:16:16: for example. So that you make sure that you're

00:16:21: not downloading patch versions on day one, a couple of hours

00:16:24: after they were released. Because with those attacks that happened,

00:16:28: was that they had a relatively small attack window.

00:16:32: The attack, thankfully, was detected quite quickly and

00:16:36: those compromised versions were unpublished, so the window of

00:16:40: attack was relatively small but big

00:16:44: enough to affect many, many machines.

00:16:47: Now if you were to set a minimum, uh, release age,

00:16:51: which again, is available with PNPM, but to my knowledge not with

00:16:55: NPM right now, if you were to use this setting,

00:16:59: or with more safety I should say, uh, auto-update

00:17:03: for new patch versions and so on. But you would make sure that you only

00:17:07: do it after a couple of days to make sure that others have

00:17:11: tested the water first and potential security issues might've

00:17:15: been detected. It's not a guarantee, but it's an additional, uh,

00:17:19: step to make your environment more secure

00:17:22: and it allows you to still auto-update, um, for

00:17:26: example. So that's another thing you might wanna

00:17:29: consider. Now another thing you should think

00:17:33: about or you should do in general, to be very honest,

00:17:37: is limit the scope of your access

00:17:40: tokens and credentials. It is, of course, super

00:17:44: convenient to always create access tokens that have full

00:17:48: access to everything. Like, let's talk about AWS.

00:17:51: If you run some deployment

00:17:55: process that also uses AWS or the

00:17:58: deployed app is using AWS and therefore is part of the,

00:18:02: uh, build process, you are exposing your AWS access

00:18:06: key and your credentials to the build process to c- bake it into

00:18:10: the built application, so to say.It is, of course, convenient to have an

00:18:14: AWS access token that gives the application admin

00:18:18: access to all your AWS resources because it's your code, right?

00:18:21: What could go wrong? Well, something like this attack could go wrong.

00:18:25: If that AWS token is ever stolen, like

00:18:29: it could be through that attack, then you're in a huge

00:18:33: trouble. You have a token out there in the wild

00:18:37: attackers full access to your AWS account, and that's just one

00:18:40: example. It's the same, of course, for the other cloud providers.

00:18:43: It's the same for AI tokens. I mean, think of, um,

00:18:47: some malicious actor stealing your OpenAI

00:18:51: access token and then using AI, uh, and you're paying the

00:18:55: bill. Of course, you might have some limits in place there,

00:18:59: that is not a lot of fun. So as a best

00:19:02: practice, you should really try to limit the scope

00:19:06: those access tokens and credentials if it's possible.

00:19:09: Not all providers, not all services give you a lot of

00:19:13: options there, but many do, and if you're, for example, building an

00:19:16: application that only needs to send email through

00:19:20: AWS, then your AWS access token should

00:19:24: only have access for exactly this action, to

00:19:27: exactly this action. Now, in case of AWS

00:19:31: and also other services, there are even better ways than using an access

00:19:35: token in the first place. You can get

00:19:39: short-lived credential stare as well, uh, but that is

00:19:43: beyond the scope of this video. You, of course, might wanna look into

00:19:46: such solutions as well if you're using a service like AWS that

00:19:50: offers even more secure ways of accessing it

00:19:54: than those general access tokens.

00:19:56: But if we're talking about access tokens and credentials, limiting the scope

00:20:00: is definitely something you might wanna do.

00:20:04: And th- of course, if you are affected by the attack, by the way,

00:20:08: you wanna rotate all your tokens and all your credentials.

00:20:11: You wanna change them all because you have to consider them

00:20:15: compromised. That's, of course, clear.

00:20:18: Well, and that's in the end it for my two

00:20:22: cents on the attack here. The most important

00:20:26: takeaway I have from this attack is really the part

00:20:30: with scoping your tokens, limiting the scope, I mean,

00:20:35: not automatically updating to the latest patch versions all the time

00:20:39: automatically, and maybe my very most important

00:20:42: point, not using too many packages.

00:20:45: That is, of course, the best line of defense here because what

00:20:49: us is how vulnerable this entire

00:20:53: JavaScript ecosystem is. If one

00:20:56: package that's used by enough other packages gets compromised and

00:21:00: acts as a worm, as it happened in the second attack,

00:21:03: then we can quickly have a spreading wildfire that's very

00:21:07: hard to contain and that can quickly affect millions or even

00:21:11: billions of machines, um, be that

00:21:15: your laptop or the machine in the cloud where you deploy or build your

00:21:19: application. And that, of course, is

00:21:23: an unbelievable huge threat.

00:21:27: So yeah, that is something we should all consider

00:21:30: these attacks have shown us. We should make

00:21:34: sure that we ramp up our, um,

00:21:38: protection, our mitigation strategies, that we make sure that

00:21:42: we, uh, limit the, the danger we face on

00:21:46: our machines or, uh, regarding to our credentials and so on.

00:21:50: And, uh, yeah, let me know what you think of that,

00:21:54: also want me to mention. As I mentioned, I'll put some important links, um,

00:21:58: in the description, and, uh, yeah, I'll see you in the next video.

00:22:01: Bye.