Troubleshooting AFL Fuzzing Problems

In our quest to find the CVE-2021-3156 vulnerability through fuzzing, we found that afl was causing our computer CPU and disk resources to get all used up. Using a couple of lines of code, we turned that to our advantage, so afl can now tell us if a file has been created while it fuzzes sudo. We also address some userid issues so that we can fuzz as root but invoke sudo as an unprivileged user.

The Video

Introduction

In the last article of the CVE-2021-3156 vulnerability (re)discovery series, we successfully got AFL to fuzz sudo using the LLVM compiler and modifying a bit of code in progname.c. We then left AFL to fuzz sudo overnight.

So far in this series, what we've found out is that fuzzing sudo is not as trivial as we anticipated. In this article, we'll pick up where we left off, discussing the result of this bout of fuzzing, and making further progress.

Parallelization and Resource Usage

Running Out of CPU...

After only running our four parallel processes for an hour, we noticed that the execution speed reported by afl was struggling at around 30 executions per second, which afl itself told us was slow!. This wasn't going to get us to the vulnerability very quickly! Time to investigate.

We decided to stop and restart one fuzzing process to see what was going on. Stopping was no issue, but on restart, the program aborted due to our machine running out of CPU cores to allocate for the program. How's that?!

Using ps aux to pull up the running processes list, we found tons of /usr/bin/vi processes. It seemed to us like afl , in its fuzzing frenzy, found arguments that launched the vi editor... but didn't close it. So we wound up with an accumulation of vi editor processes that were eating up our available computational resources. We set everything straight by killing the relevant processes with pkill vi. We started fuzzing again, with no more issues. Great! This is just a band-aid fix though, and we figured that it would be better to permanently sort this.

Let's take a step back, consider the big picture, and think about this for a second. The fuzzing operation started the vi editor multiple times, and kept the processes running. Who is to say that it wouldn't launch other programs that would also consume precious computational resources? The best way to address this is to prevent any launching of programs. This is not to say that you cannot find a privilege escalation vulnerability through code execution, but we are specifically interested in a memory corruption issue. Therefore, it makes sense for us to prevent the fuzzer from launching other programs.

There are several variations of exec that are available in Linux, such as execl, execlp, execle, execv... so we searched for these terms in the sudo source code and commented each relevant line out. This is akin to having the exec failing.

Good news - the fuzzing speed has now improved! Time to catch a bit of sleep overnight.

... And of Disk Space

In the morning, we checked the afl fuzzing dashboard, excited to see what we had missed while we were getting some well-deserved rest. Oh... we got an error message: unable to create /tmp/out/f2/queue/... due to the lack of remaining available space on the storage device.

Interesting. We couldn't even create a new file ourselves in the terminal. Using df -h, we checked out the disk usage. Curiously enough, the shell output showed only 32% of disk usage. So how could we have run out of space? Thanks to some extensive googling, we found out that we could check the allocated inodes using df -i. Sure enough, as indicated by the shell, all 100% of the inodes were used. What are inodes, anyways? As summarized in the corresponding Wiki,

The inode (index node) is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory. Each inode stores the attributes and disk block locations of the object's data.

There is a limited amount that the file system can count to. Effectively, as the article mentions, it is possible to run out of inodes without actually having filled the disk. In our case, this is a sign that afl created tons of small, individual files. We tried to track where all of these files actually were. It turns out that many of them were housed in /var/tmp, and by many, we mean a whopping 2.3 million files (2,328,889 files, to be specific). That's an issue, as the fuzzer should not be creating files via sudo, as that's not what we are interested in! With all that noted, we noticed that the filenames in this directory were pretty long and random, which indicates that user input can control the file name. Since sudo runs as root,  maybe there is a path traversal where we can inject ../ into the var/tmp filepath and write a root-owned file somewhere.

Unlike exec, which we promptly removed on discovery, we want to use the file generation to our advantage. afl logs the arguments that it used that caused a crash. By forcing a crash to occur when a file is created, we can make afl tell us what arguments of sudo create a file. This is actually a fuzzing trick that is not used for memory corruptions, but we can use it so afl signals to us that it encountered conditions that we are interested in. We use

printf("mkstemps(%s)\n", suff);
*(int*)0 = 0; // force crash
tfd = mkstemps(*tfile, suff ?, strlen(suff) : 0);

The second line of this code snippet forces a null dereference, causing a segmentation fault that triggers the afl logging of the crash. The offending file's path is also output, so we can find it at a later time. Now, when afl finds a crash, we should get an example argument list that causes the crash, as well as the location of the new tmp file that was created. With the compilation and restart of the fuzzing processes completed, afl quickly began picking up crashes, meaning that we could start investigating.

Let's consider an example input. We used hexdump to output the file contents both into the shell as well as a file located in /tmp/mktemp. We can then use cat to pipe the file contents as an input to sudo. Good news: we get the expected crash.

There is a caveat though.

To root Or Not To root...

Throughout this entire procedure, we were running and fuzzing as root. Can a regular user even reach the arguments that lead to temporary file creation? Time to find out. We switched users to a "normal" one (aptly named user) and ran the same tests, but we get an error stating that the effective uid is not zero, indicating that we are not root. Is sudo installed setuid root?

We copied the binary we had to /tmp/sudo and set the proper uid permission bit. Now, we have a setuid sudo binary and so we can try our payload once again.

We get a password prompt. Yikes! This is in fact a fuzzing issue that we hadn't even considered. Effectively, we are targeting a binary that is set up such that it runs with setuid, and we are interested in the special case where the binary is invoked by an unprivileged user but runs as root with setuid. However, we cannot simply fuzz a setuid binary as a user, as afl would not work due to not being able to communicate with the privileged setuid process. Turns out afl can fuzz a setuid process, but I must have made some other mistake and misinterpreted the error. I concluded it wouldn't work. Eitherway, the solution still shows an important technique when fuzzing - modifying the target to help the fuzzing efforts.

Therefore, we either run completely as user or as root, but we cannot do both (one for fuzzing and one for running). Using both is not representative of normal sudo usage, therefore, we want to avoid this setup entirely. In light of this, we need another plan.

... That Is the Question!

We wound up deciding to fuzz and run as root, but somewhere in this process, sudo should get the current user. If we find that location, we can thus modify it and force it to think that an unprivileged user invoked it. The typical function to get the userid of the user that executed the program is (drum roll, please) getuid().

Searching for all the uses of getuid(),  we found the get_user_info() function in sudo.c. Instead of letting getuid() return 0 for the root user, we simply hardcoded the value of 1000, which is the userid for a regular user. After compiling, we used our /tmp/mktemp payload once more. Instead of facing the temporary file creation as we did running as root, we now were staring straight at a password prompt, just as we would if we were an unprivileged user. Bingo!

This looks like it should work, then. It should also resolve our issue with file creation and running out of inodes, as these temporary files were only created by running sudo, already being root. This means that after all this tinkering, we should finally have a pretty good sudo-fuzzing setup. This in turn means that we should go for yet another round of fuzzing.

You know the drill: get in your PJs, get comfortable, and just watch the afl dashboard as the program does its thing. What a pretty sight.

Final Words

With afl now running and most kinks ironed out (that we know of, at least), we want to hear from you. Do any of you have experience with other fuzzers or fuzzing frameworks, such as libFuzzer or honggfuzz? If you can set up a minimal fuzzing environment for sudo and share it with us, we would be very happy to have a look! We have no experience with these tools and we would love to compare their setup, associated workflows, and performance with our specific use-case against afl.

Also, if you have any recommendations regarding how to optimize the afl files, please do share your fuzzing setup as well. We know about afl++ and as mentioned before, we will be switching to it in due course, but we want to hear about the tricks you might have for afl itself.

In the next video, we'll check out the results from our fuzzing campaign. Spoiler alert: there will be more technical hurdles to overcome. Keep an eye out for the next video and article!