In our quest to find the CVE-2021-3156 vulnerability through fuzzing, we found that
afl was causing our computer CPU and disk resources to get all used up. Using a couple of lines of code, we turned that to our advantage, so
afl can now tell us if a file has been created while it fuzzes
sudo. We also address some
userid issues so that we can fuzz as
root but invoke
sudo as an unprivileged user.
In the last article of the CVE-2021-3156 vulnerability (re)discovery series, we successfully got AFL to fuzz
sudo using the LLVM compiler and modifying a bit of code in
progname.c. We then left AFL to fuzz
So far in this series, what we've found out is that fuzzing
sudo is not as trivial as we anticipated. In this article, we'll pick up where we left off, discussing the result of this bout of fuzzing, and making further progress.
Parallelization and Resource Usage
Running Out of CPU...
After only running our four parallel processes for an hour, we noticed that the execution speed reported by
afl was struggling at around 30 executions per second, which
afl itself told us was
slow!. This wasn't going to get us to the vulnerability very quickly! Time to investigate.
We decided to stop and restart one fuzzing process to see what was going on. Stopping was no issue, but on restart, the program aborted due to our machine running out of CPU cores to allocate for the program. How's that?!
ps aux to pull up the running processes list, we found tons of
/usr/bin/vi processes. It seemed to us like
afl , in its fuzzing frenzy, found arguments that launched the
vi editor... but didn't close it. So we wound up with an accumulation of
vi editor processes that were eating up our available computational resources. We set everything straight by killing the relevant processes with
pkill vi. We started fuzzing again, with no more issues. Great! This is just a band-aid fix though, and we figured that it would be better to permanently sort this.
Let's take a step back, consider the big picture, and think about this for a second. The fuzzing operation started the
vi editor multiple times, and kept the processes running. Who is to say that it wouldn't launch other programs that would also consume precious computational resources? The best way to address this is to prevent any launching of programs. This is not to say that you cannot find a privilege escalation vulnerability through code execution, but we are specifically interested in a memory corruption issue. Therefore, it makes sense for us to prevent the fuzzer from launching other programs.
There are several variations of
exec that are available in Linux, such as
execv... so we searched for these terms in the
sudo source code and commented each relevant line out. This is akin to having the
Good news - the fuzzing speed has now improved! Time to catch a bit of sleep overnight.
... And of Disk Space
In the morning, we checked the
afl fuzzing dashboard, excited to see what we had missed while we were getting some well-deserved rest. Oh... we got an error message:
unable to create /tmp/out/f2/queue/... due to the lack of remaining available space on the storage device.
Interesting. We couldn't even create a new file ourselves in the terminal. Using
df -h, we checked out the disk usage. Curiously enough, the shell output showed only 32% of disk usage. So how could we have run out of space? Thanks to some extensive googling, we found out that we could check the allocated
df -i. Sure enough, as indicated by the shell, all 100% of the
inodes were used. What are
inodes, anyways? As summarized in the corresponding Wiki,
The inode (index node) is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory. Each inode stores the attributes and disk block locations of the object's data.
There is a limited amount that the file system can count to. Effectively, as the article mentions, it is possible to run out of
inodes without actually having filled the disk. In our case, this is a sign that
afl created tons of small, individual files. We tried to track where all of these files actually were. It turns out that many of them were housed in
/var/tmp, and by many, we mean a whopping 2.3 million files (2,328,889 files, to be specific). That's an issue, as the fuzzer should not be creating files via
sudo, as that's not what we are interested in! With all that noted, we noticed that the filenames in this directory were pretty long and random, which indicates that user input can control the file name. Since
sudo runs as
root, maybe there is a path traversal where we can inject
../ into the
var/tmp filepath and write a
root-owned file somewhere.
exec, which we promptly removed on discovery, we want to use the file generation to our advantage.
afl logs the arguments that it used that caused a crash. By forcing a crash to occur when a file is created, we can make
afl tell us what arguments of
sudo create a file. This is actually a fuzzing trick that is not used for memory corruptions, but we can use it so
afl signals to us that it encountered conditions that we are interested in. We use
printf("mkstemps(%s)\n", suff); *(int*)0 = 0; // force crash tfd = mkstemps(*tfile, suff ?, strlen(suff) : 0);
The second line of this code snippet forces a null dereference, causing a segmentation fault that triggers the
afl logging of the crash. The offending file's path is also output, so we can find it at a later time. Now, when
afl finds a crash, we should get an example argument list that causes the crash, as well as the location of the new
tmp file that was created. With the compilation and restart of the fuzzing processes completed,
afl quickly began picking up crashes, meaning that we could start investigating.
Let's consider an example input. We used
hexdump to output the file contents both into the shell as well as a file located in
/tmp/mktemp. We can then use
cat to pipe the file contents as an input to
sudo. Good news: we get the expected crash.
There is a caveat though.
To root Or Not To root...
Throughout this entire procedure, we were running and fuzzing as
root. Can a regular user even reach the arguments that lead to temporary file creation? Time to find out. We switched users to a "normal" one (aptly named
user) and ran the same tests, but we get an error stating that the effective
uid is not zero, indicating that we are not
We copied the binary we had to
/tmp/sudo and set the proper
uid permission bit. Now, we have a
sudo binary and so we can try our payload once again.
We get a password prompt. Yikes! This is in fact a fuzzing issue that we hadn't even considered. Effectively, we are targeting a binary that is set up such that it runs with
setuid, and we are interested in the special case where the binary is invoked by an unprivileged user but runs as
However, we cannot simply fuzz a Turns out
setuid binary as a
afl would not work due to not being able to communicate with the privileged
afl can fuzz a setuid process, but I must have made some other mistake and misinterpreted the error. I concluded it wouldn't work. Eitherway, the solution still shows an important technique when fuzzing - modifying the target to help the fuzzing efforts.
Therefore, we either run completely as
user or as
root, but we cannot do both (one for fuzzing and one for running). Using both is not representative of normal
sudo usage, therefore, we want to avoid this setup entirely. In light of this, we need another plan.
... That Is the Question!
We wound up deciding to fuzz and run as
root, but somewhere in this process,
sudo should get the current user. If we find that location, we can thus modify it and force it to think that an unprivileged user invoked it. The typical function to get the
userid of the user that executed the program is (drum roll, please)
Searching for all the uses of
getuid(), we found the
get_user_info() function in
sudo.c. Instead of letting
0 for the
root user, we simply hardcoded the value of
1000, which is the
userid for a regular user. After compiling, we used our
/tmp/mktemp payload once more. Instead of facing the temporary file creation as we did running as
root, we now were staring straight at a password prompt, just as we would if we were an unprivileged user. Bingo!
This looks like it should work, then. It should also resolve our issue with file creation and running out of
inodes, as these temporary files were only created by running
sudo, already being
root. This means that after all this tinkering, we should finally have a pretty good
sudo-fuzzing setup. This in turn means that we should go for yet another round of fuzzing.
You know the drill: get in your PJs, get comfortable, and just watch the
afl dashboard as the program does its thing. What a pretty sight.
afl now running and most kinks ironed out (that we know of, at least), we want to hear from you. Do any of you have experience with other fuzzers or fuzzing frameworks, such as
honggfuzz? If you can set up a minimal fuzzing environment for
sudo and share it with us, we would be very happy to have a look! We have no experience with these tools and we would love to compare their setup, associated workflows, and performance with our specific use-case against
Also, if you have any recommendations regarding how to optimize the
afl files, please do share your fuzzing setup as well. We know about
afl++ and as mentioned before, we will be switching to it in due course, but we want to hear about the tricks you might have for
In the next video, we'll check out the results from our fuzzing campaign. Spoiler alert: there will be more technical hurdles to overcome. Keep an eye out for the next video and article!