Found a Crash Through Fuzzing? Minimize AFL Testcases!

Our fuzzer found a case that crashes the sudoedit program (CVE-2021-3156). We conduct an in-depth analysis of the test case that causes the binary to crash. After being sure that it works, we minimize the test case using AFL's own tool.

The Video

Introduction

In the last episode of the sudo vulnerability series (CVE-2021-3156), we ironed out some buffer overflow issues with AFL and set up four fuzzing process. These corresponded to fuzzing sudo and sudoedit using both AFL and AFL++. This is an interesting experiment, as it tests for whether AFL or AFL++ can find the vulnerability through fuzzing, and if so, if it needed a broad or more specific test case input.

In this article, we pick up from there and analyze the results from this overnight fuzzing session. We also attempt to minimize some testcases for further fuzzing.

Fuzzing Results

Since we let these run overnight, the first thing we did when we rolled out of bed was – naturally – go to check the fuzzing results. Here's what we found:

• The first process, running AFL with the sudo test case, had no crashes and just a couple thousand of timeouts to report.
• The second process, with AFL and sudoedit, was running slow and has a whopping 11.6 thousand timeouts, though only 61 were unique. We eventually let that one run for a week and it got some speed going again, but we digress.
• The third, with AFL++ fuzzing just sudo, had no crashes to report, which we found rather interesting. We can however tell you that this process eventually found sudoedit.
• The fourth fuzzing process, running AFL++ and the sudoedit test case, showed 1 crash (and therefore 1 unique crash).

With the last new path time counter starting to tick into the "a few hours" territory, we figured that the fuzzers might have reached the point where it had already found all the easy fuzzing arguments and that further results would take several more hours. It's difficult to decide whether more than an hour is enough or if we should let the fuzzing run for longer; we don't have the experience to judiciously make this decision. Eventually, we decided to call it good and have a look at the unique crash from the fourth fuzzing process.

sudoedit Crash Investigation

Checking the Input File

First, we checked the crash input file itself, looking at it with hexdump. We would've expected it to find a variant of the vulnerability with sudoedit -s or something like that. Apparently, this was not the case. However, this input does have a backslash at the end of the argument, so maybe we might've stumbled on an another input variant that triggers the sudoedit bug? Or, it might be a false positive. Let's run this input file through a series of tests to determine whether it's legitimate, or a false positive.

Testing the Input

We copied the crash to /tmp/crash, then piped it as input for the AFL instrumented version of the sudo binary with

cat /tmp/crash | /pwd/sudo-1.8.31p2/src/sudo

and watched as a malloc(): invalid next size (unsorted) error popped up immediately.

Maybe this is just a result of using our instrumented binary, so we tried it with our neat wrapper that we wrote in the last article.

cat /tmp/crash | ./afl2sudo

We get once again the very same error.

This is a good sign! Let's do one last test, though. You might have noted that we conducted these tests as root, which we decided to do (a while ago) to facilitate reaching the error. With that in mind, it's important to remember that CVE-2021-3156 involves an unauthorized user privilege escalation, so it makes sense to try to encounter the vulnerability as an unprivileged user instead of as root. We switched users using su user, and ran the same command we immediately did... and it works!

This input passes our three tests. It is thus time for us to look at it in much greater detail.

Inner Workings of the Crash

We used gdb to further investigate the cause of the crash. To help us, we used the gdb extension called GEF, or GDB Enhanced Features, which you can find here. You could also use pwndbg, which is available here. We open up our afl2sudo wrapper with gdb, to properly set up the arguments, and run the /tmp/crash input.

Our binary will execute, set up the arguments, and then execute the real sudo binary... and then we crash, this time with a SIGABRT signal, instead of a segmentation fault. We say "crash", but it's more of an "abort", to be technically accurate. So why did that happen?

Apparently malloc detected an inconsistency with the next size, and decided to bail out and stop the program execution before something broke. This however still means that we must've corrupted something on the memory heap.

The backtrace gets us a list of the functions that were executed, and leads us step-by-step to the point where malloc aborted the program. Some of these functions are from malloc, so they belong to libc, whereas the last one was from sudo, namely, sudo_getgrouplist2_v1. When we use up to get into that stack frame, gdb can also show us the source code line that caused the program to abort. In our present case, the function tried to reallocate something, but malloc detected a heap inconsistency and aborted. This in turn means that we did in fact cause some kind of heap corruption. Neat.

This is of course pretty exciting, but we need to better understand this crash and the mechanism underlying it. Is this a variant of the sudoedit vulnerability? Or is this – dare we say – maybe a new and different sudo crash? After all, the test case did not contain sudoedit. Let's not get prematurely excited though. This is probably not a different sudo crash, but we figured that we could at least consider the option if only for a short, fleeting moment.

Keep in mind that you can find the code we've run through, it is available on GitHub. Feel free to play around with it – perhaps you can figure out what happened!

Further Testing

The next test we carry out is to see whether the sudo program in the latest version of Ubuntu, which has been patched to address CVE-2021-3156, also crashes. If that's the case, then we would know that it's a new zero-day. We edit our afl2sudo wrapper code to use it outside of our Docker container and instead execute the actual sudo program that is shipped with Ubuntu instead of our version; we do so simply by changing the execution path in the wrapper. We then compile our wrapper with gcc, and pipe the original input file to the new wrapper. No dice.

We knew that our crash being a new zero-day was a bit of a stretch, but we had to find out for sure. It is possible however that this crash is a variant of the sudoedit CVE-2021-3156 vulnerability... but still a curious variant. Indeed, it seems to not include sudoedit. Remember, sudo shows different behaviors based on whether sudo or sudoedit is invoked (via symlinking). Also, from the console output, you can see that the program name for sudoedit is botched, while the argument lists correspond to sudo. This is quite the curious behavior.

Input Test Case Minimization

At any rate, this test case input file we have is quite complex, and has a significant amount of extraneous information. As a result, we want to minimize this file, by which we mean reduce its contents to the core material. This allows us to have a shorter file to act as a proof-of-concept test case, and it is significantly easier to work with and understand. On the AFL++ GitHub readme, you can find out more about the minimization of test cases. We have no experience with any of these, and we do want to try them all. We'll start with the actively-maintained halfempty project from Google Project Zero.

halfempty

To use this program, we have to create a small wrapper shell script that checks if the input crashed the program or not. The example code provided on the project GitHub page looks like this, for gzip:

#!/bin/sh

gzip -dc 

# Check if we were killed with SIGSEGV

if test $? -eq 139; then
	exit 0 # We want this input
else    
	exit 1 # We don't want this input
fi

The exit value of 139 corresponds to a segmentation fault, but we're interested in the SIGABRT signal, with code 134. We checked this by using echo $? after creating the crash by piping the input test case into the instrumented sudo binary.

We thus modify the script to fit our needs:

#!/bin/sh

/pwd/sudo-1.8.31p2/src/sudo

# Check if we were killed with SIGABRT

if test $? -eq 134; then
	exit 0 # We want this input
else    
	exit 1 # We don't want this input
fi

Now, we call halfempty with the minimizing shell script and our crashing input. To our surprise, it doesn't work – the bisection strategy failed. We tried it ourselves by using the command suggested in our console, and it seemed to work, since the return exit code from the script was indeed 0, just as we wanted. At this point, we just gave up on halfempty. Let's try another method.

afl-pytmin

How about this one? It seems to be a wrapper around AFL, so let's try it! We adjusted the path to our input file and the other parameters, and hit the Return key. The console quickly prints [i] Processing /tmp/in_crash/1 file... before giving us the user input line in the shell again. Is it... done? We check for the file created, and it's empty. That's interesting. Maybe an empty input file also crashes sudo? Unfortunately, that doesn't work either.

afl-tmin

At this point, we were pretty frustrated, so we went with the minimizer that ships directly with afl. We fed it the crashing input, a destination to write the minimized file to, and finally the instrumented binary. Once again, we hit the Return key and the program runs quickly and finishes in a couple of seconds. In fact, we thought that once again we were facing a failure.

We checked the output file with hexdump, and it's very clean now, certainly a lot more than our initial test case file was. That's it, 0edit -s 00000\ is our crashing input! This looks like our sudoedit vulnerability.

Let's try it by first switching to the unprivileged user account using su user, change directory, symlink 0edit to sudo, and then punch it in with our -s 00000\\ flag and arguments (the double-backslash is due to needing to escape the backslash), and then adding a couple more 0s for good measure and... it crashes!

It's interesting that calling just 0edit outputs the sudo help to the console, unlike sudoedit pulling up the sudoedit help, as expected, but the 0edit still goes into the sudoedit argument parsing code and then crashes. Interesting behavior for different symlinks.

We can also play a little with the name and try different variants, and it's really weird why it works. After all, it's not like the name can be just anything, it's weirdly specific and has to follow a <name>edit format, with at least one character before the edit portion.

We incidentally mentioned in a previous video that there is a check that only compares the end of the program name to the edit, but we only found this link from the test cases we just discussed.

Final Words

Now that we have a minimized test case, we can investigate this further. As mentioned, we are crashing when sudo tries to call realloc(). It detects that some metadata value is invalid, which means that the heap metadata was corrupted, too, having been overwritten at some earlier point. So, we now need to figure out where exactly the heap corruption occurs. In fact, you can try it out yourself. How would you find the true location of the overflow? We'll explain our methodology in our next video.