# 8 Million Requests Later, We Made The SolarWinds Supply Chain Attack Look Amateur
Surprise surprise, we’ve done it again. We’ve demonstrated an ability to compromise significantly sensitive networks, including governments, militaries, space agencies, cyber security companies, supply chains, software development systems and environments, and more.
“Ugh, won’t they just stick to creating poor-quality memes?” we hear you moan. Maybe we should, maybe we shouldn’t – regardless, it’s too late at this stage and so we have to live with it.
From those of you who enjoy our research, to the PSIRT and CERT teams who dread an email originating from `@watchTowr.com`, you are likely aware that we’ve historically delivered research that shone a spotlight on the security impact of abandoned infrastructure in various forms:
– Obtaining the ability to issue valid TLS/SSL certificates for any .MOBI domain (via abandoned domains used for WHOIS servers)
– Hijacking backdoors in backdoors to compromise government networks (by registering domains for backdoors, within widely used backdoors)
Apparently, though, this wasn’t enough to satisfy us that we’d demonstrated just how held-together-by-string the Internet is and at the same time point out the reality that we as an industry seem so excited to demonstrate skills that would allow us to defend civilization from a Neo-from-the-Matrix-tier attacker – while a metaphorical drooling-kid-with-a-fork-tier attacker, in reality, has the power to undermine the world.
Therefore, almost without choice – once again, we’re excited to share our research with everyone and be somewhat depressed by the results – misery loves company, and a problem shared is a problem halved (thank you).
Arguably armed still with a somewhat inhibited ability to perceive risk and seemingly no fear, in November 2024, we decided to prove out the scenario of a significant Internet-wide supply chain attack caused by abandoned infrastructure. This time however, we dropped our obsession with expired domains, and instead shifted our focus to Amazon’s S3 buckets.
It’s important to note that although we focused on Amazon’s S3 for this endeavour, this research challenge, approach and theme is cloud-provider agnostic and applicable to any managed storage solution. Amazon’s S3 just happened to be the first storage solution we thought of, and we’re **certain** this same challenge would apply to any customer/organization usage of any storage solution provided by any cloud provider.
The TL;DR is that this time, we ended up discovering ~150 Amazon S3 buckets that had previously been used across commercial and open source software products, governments, and infrastructure deployment/update pipelines – and then abandoned.
Naturally, we registered them, just to see what would happen – “how many people are _really_ trying to request software updates from S3 buckets that appear to have been abandoned months or even years ago?”, we naively thought to ourselves.
As always, what we didn’t anticipate was how this would turn out (you could argue that we regularly seem to underestimate what is about to happen).
Before you ask, for many reasons, after this research is published we’re resolutely vowing not to touch the subject of abandoned infrastructure again (publicly). We’ve beaten this proverbial horse to death in three different ways, and frankly we don’t want to completely lose faith in the Internet.
As for the research itself, it panned out progressively, with S3 buckets registered as they were discovered. It went rather quickly from “Haha, we could put our logo on this website” to “Uhhh, `.mil`, we should probably speak to someone”.
$400+ USD later (we’ve included S3, CloudTrail, and CloudWatch – because we relentlessly queried the logs), we had some results worth talking about.
When creating these S3 buckets, we enabled logging – allowing us to track:
– Who requested files from each S3 bucket (via the source IP address)
– What they requested (filename, path, and the name of the S3 bucket itself)
These S3 buckets received **more than 8 million HTTP requests over a 2 month period** for all sorts of things –
– Software updates,
– Pre-compiled (unsigned!) Windows, Linux and macOS binaries,
– Virtual machine images (?!),
– JavaScript files,
– CloudFormation templates,
– SSLVPN server configurations,
– and more.
Put extraordinarily simply – if we were villainously inclined, we could’ve responded to each of these requests with something malicious like:
– A nefarious software update,
– A CloudFormation template that gave us access to an AWS environment,
– Virtual Machine images backdoored with ‘remote access tooling’,
– Binaries that deployed ‘remote access tooling’ or scary ransomware, or such,
– etc
to give us access to the requesting system, or network that the requesting system was sat within.
These millions of incoming ‘give me this file’ requests came from the networks of organizations (based on DNS/WHOIS lookups) that some would define as ‘fairly important’:
– Government networks
– USA (inc NASA, numerous laboratories, state governments, etc)
– UK
– Poland
– Australia
– South Korea
– Turkey
– Taiwan
– Chile
– etc
– Military networks
– Fortune 500s
– Fortune 100s
– “Major payment card network”
– “Major industrial product company”
– Global and regional banks and financial services organizations
– Universities around the world
– Instant messenger software companies
– Cyber security technology companies (lol)
– Casinos
– etc
You get the idea.
> Note: We will **not** be drawing any relationship between a specific S3 bucket and ‘which networks we saw requests coming from’ to ensure that no project or software company is subsequently targeted.
Before we start sharing our findings, we want to mention a few things about the nature of supply chain attacks.
As a starting point, the cyber security industry is not new to the challenge posed by supply chain attacks. Over the last few years, we’ve seen real-world supply chain attacks play out, but seemingly only within the grasp of those that apparently most would call ‘nation-state’:
– Jia Tan VS XZ/liblzma and OpenSSH
– SolarWinds Orion VS Cozy Bear
– Every living being VS NPM, roughly every other day
Are these solved yet? Who cares – while these are undoubtedly illegal, malicious incidents, these incidents do demonstrate the potential wide-scale impact of a supply-chain attack, while requiring effort and capability more sophisticated than our proverbial kid with a fork.
Clearly, we’re not as smart as the people who undoubtedly put hours/days/months/years into planning and executing the above incidents – we are hackers in our home offices/hotel rooms who resist Jira – but we did watch.
We want to take this opportunity to give our sincere thanks to the entities that engaged with us (while likely rolling their eyes in the background) when we realized what we’d stumbled into, including:
– NCSC UK (who have helped with introductions and signposting to the correct teams to speak to),
– AWS (who took said ~150 S3 buckets off our hands to sinkhole),
– Major Unnamed SSLVPN Appliance Vendor #2 (who worked with us very quickly and directly to take relevant S3 buckets off our hands), and
– CISA (who very quickly remediated an example that affected `cisa.gov`)
AWS’s agreement to sinkhole the identified S3 buckets means that the release of this research does not increase the risk posed to any party—the same issues discussed in this research could not be recreated against the same specific S3 buckets, thanks to the sinkholing performed by the AWS team.
We believe that in the wrong hands, the research we have performed could have led to supply chain attacks that out-scaled and out-impacted anything we as an industry have seen so far – **or put more clearly, we would’ve embarrassed Cozy Bear and made their SolarWinds adventures look amateurish and insignificant.**
While we suspect some would argue that ‘watchTowr is already the wrong hands’.. actually, you’re probably right.
As a final thought – as an industry, we spend a lot of time trying to solve issues like “securing the supply chain” in as many complex ways as possible and still completely fail to cover the easy things, like **‘make sure you don’t take candy from strangers’.**
If you made it this far, enjoy the research, and we’ll be back again next week :^)
### Where Did This Idea Come From?
Anyway. Before we get too carried away, let’s rewind, and start way back at the beginning.
Picture the scene. A researcher is sitting at their desk, looking at memes for inspiration, surrounded by mountains of empty energy drink cans and disassembled computer hardware.
Ten minutes later, they’re browsing a security vendor’s (let’s call them “Antivirus and MDR Vendor #1”) website looking for a report on a particular APT group, and voila – they find a link to the promised report in the lovely format of a PDF file.
To their dismay though, when they click the link to load the report from `https://s3.eu-west-1.amazonaws.com/nationalcert-content/files/apt-1337.pdf`, they’re met with not the report, but instead the following error message:
“`
NoSuchBucket The specified bucket does not exist nationalcert-content [redacted] [redacted]
“`
Astute readers who are blessed with the ability to read will understand very quickly where this is going – the S3 bucket called `nationalcert-content` no longer exists (and so the poor researcher can’t read that PDF). This does constitute a “security issue”, **but minor at most.**
Could we register the S3 bucket in question, and use it to serve a malicious PDF file a job advert to anyone that requested it (captive audiences, etc)? Yes, yes we could.
While this sounds mischievous and nefarious (we didn’t do this, in this anonymized eample), the reality is that the severity of such an attack is roughly equivalent to hijacking a dead link. It’s not great for the owner of the website, but it’s not the end of the world or significant.
For those in the audience cursed with an inability to threat model – yes, as security “professionals” we can likely all think of scenarios in which this could theoretically be abused, especially when combined with the likely target audience of security researchers or threat intel analysts – but please calibrate yourself, this is not significant nor within the realms of a reasonable threat model.
We know – none of this stuff is new or exciting. Give us a few minutes. We’re really trying our best to build context before we drop the interesting stuff.
### What Is Amazon S3?
Per Amazon, Amazon S3 is:
> _.. an object storage service offering industry-leading scalability, data availability, security, and performance. Millions of customers of all sizes and industries store, manage, analyze, and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and easy-to-use management features, you can optimize costs, organize and analyze data, and configure fine-tuned access controls to meet specific business and compliance requirements._
TL;DR: it is a dedicated file storage solution in the cloud. It has the benefit that it is both very cheap and very easy to use.
Amazon S3 is a good solution in its own right – it allows anyone with an Amazon account to host files and make them accessible to the entire Internet, without worries like ‘what if a hard drive fails’ or ‘do I have enough bandwidth’, or, “should I actually be sharing this with the world?”.
In order to keep things neat and tidy, S3 gives users this magical space and place called a ‘bucket’ – effectively, a file-sharing space that can contain folders and folders.
For example, we might decide to use Amazon S3 to host memes and register an S3 bucket called ‘watchTowrs-finest-memes’. This would result in the S3 URL (ignoring regions – not our job to teach this!) of `watchtowrs-finest-memes.s3.amazonaws.com` (this doesn’t exist, hijack it you nerds).
We could then upload files to this bucket, and reference said files within this S3 bucket anywhere we felt the need to via a regular HTTPS URL (like `https://watchTowrs-finest-memes.s3.amazonaws.com/our-fav-meme.exe`). It goes without saying that we could store anything here – code, PDFs, images, binaries, etc – and share the link with all and sundry, reaping the benefits of Amazon’s large storage and wide Internet pipes.
### ‘World’s Easiest Bug Bounty Payout’
The problem, from a security standpoint, manifests when these S3 buckets are allowed to decay and subsequently abandoned, allowing bad actors to re-register them for themselves.
This is a known bug class, known as any names, including ‘S3 bucket takeover’ (we know this is not new – bear with us). Second-order Amazon S3 bucket takeovers via broken links are also not new, before you tell us this also.
These issues have been common for a long time. This is partly due to the prevalence of website hijacking and defacement opportunities, stemming from takeovers of abandoned S3 buckets that were previously used to host static websites.
Take this random screenshot that we found on the Internet for example:
Here, the hostname `blog.char49.com` is set to reference an S3 bucket ( `blog.char49.com`) via DNS in order to host a static website.
However, in this example that is not real, the specified S3 bucket no longer exists. Speed running this explanation – if a nefarious actor registers the bucket, they could then host malicious content, which would then be served directly on the legitimate domain `blog.char49.com`.
It’s a simple attack, but extremely common. This is not exciting or new, and it is also not the supply chain attack we promised in the title – we promise we’re getting to that bit.
### You’re Back In The Room
Well, let’s take a step back and look at things from a more generalized viewpoint.
While we’ve focused solely on instances where an S3 bucket is used as a backing store for a website’s static content, S3 is used for far more than just serving websites. A huge swath of the Internet’s resources are stored in S3, and the amount of infrastructure that uses it is considerable.
Looking at the vulnerability class with a wider eye, we can make a key observation – the mere fact that website-based S3 bucket takeovers are so prolific suggests to us that there’s some _inherent challenge_ around S3 bucket ownership. It follows that this inherent challenge – be it technical, political, organizational, or otherwise – is unlikely to be restricted to just static website resources, such as favicons and PDFs.
We asked ourselves, ‘What other things use S3 buckets behind the scenes as a more general storage solution?’ We’re no DevOps gurus, but a few answers slid conveniently off the top of our collective heads and onto our whiteboard.
– Container applications (DevOps people love storing their containers in S3)
– CI/CD tooling infrastructure (Dockerfiles, Jenkinsfiles, built artefacts, etc)
– Source code repositories (yes, GitHub just isn’t enough for some people)
– Mobile applications
– Software documentation
– Deployment instructions
While we could continue our enumeration, we decided to focus mostly on these six areas (you’ll see as we progress that this didn’t go entirely to plan, as other resources popped into our field of vision).
Satisfied with our list, we extracted some of the existing capabilities that we hold in the watchTowr Platform into a standalone binary and modified it to focus on **Internet-wide assets** rather than just our client base.
The result? A curiously modified tool that we apparently decided to call **kidwithafork**.
After a quick ‘does it work’ smoke test, we then deployed our new tooling into a proper pipeline, and got to work.
### What Did We Target?
Now, as you recall, we promised something “supply-chain-esque” against an “Internet-wide” attack surface.
However, we decided it was going to be fairly pointless to set our newly mashed-together tool to work on the _entire_ Internet, so we decided to give ourselves some further parameters to work within.
After a bit of deliberation (random ideas, and seeing what stuck), we decided to focus on the six selected asset types only when potentially previously owned by:
– Governments
– Fortune 500 companies
– Technology companies
– Cyber-security technology companies
– Major open-source projects
The logic behind our decision is simple: These organizations have a large enough user base that an issue would impact numerous people. There’s no real benefit in pointing out that a website with five users has a supply-chain problem—the owners simply don’t have the resources to deal with this kind of detail, and we just don’t care.
Our broader aim, however, remained simple – we wanted to demonstrate an **Internet-wide issue**, grounded in the “abandoned infrastructure” class of weakness, ideally with a supply chain angle (software updates, build pipelines – something like that).
> We’d like to take the opportunity now to make one thing very clear – **we have not targeted any organization in particular, despite the outcomes that we detail below. We will not entertain any conversation or speculation that we targeted any organization.** It is clear that, like expired and abandoned domain names, this issue is prolific and **not representative of any one organization’s approach to infrastructure or cyber security in isolation.**
> Any conclusion that you come to around any individual organization’s security posture as a result of this research would be incorrect, misguided, and likely due to your own bias.
Over the course of two months, our technology ingested a huge amount of data to identify references to abandoned S3 buckets and subsequently alerted us if any were found.
Once we saw S3 bucket names that looked interesting, we registered them and began logging any requests they received.
> Note: We were not ‘sniping’ S3 buckets as they were deleted, nor employing any ‘advanced’ technique to register these S3 buckets. We just.. typed the name into the input box, and used the power of 1 finger to click register.
Our intent was twofold;
– Firstly, to get an idea of just how many people were requesting data from these previously abandoned S3 buckets, and,
– Secondly, what kind of data/files was requested from these previously abandoned S3 buckets.
The below aims to showcase some of our more interesting results. As you can imagine, given that we saw over 8 million inbound requests, there’s a lot we have to leave out simply for brevity’s sake.
Since this is a long article, we’ve labelled each section with a ‘danger level’.
### Poisoned Javascript
_Danger level: North Koreans will use your web browser to mine crypto, it might catch fire_
Being the uninspired primitives that we so keenly embody, our first thought was to see if we could identify any JavaScript files requested from any of the S3 buckets we registered—because we could theoretically return a BEEF Project implant (if it was 2003 and we’d just invented fire) or a crypto miner if we’re pretending it’s 2025.
We took a quick look at our log entries and had our first inkling that we’d stumbled onto something interesting.
Take, for example, this request for a JavaScript file, which came to one of our S3 buckets:
“`
requestParameters.Host: s3.amazonaws.com requestParameters.bucketName: echochamberjs requestParameters.key: dist/main.js
“`
This is a request for a JavaScript file that is involved in the `echo-chamber-js` project (could you guess?).
A quick look at the project’s GitHub page shows the project’s `README` file, revealing that it directed users to install the project into their own websites simply via the magic of an overly engineered web-two-point-zero `