Finding Buffer Overflow with Fuzzing

We're logging crashes with AFL as we try to fuzz our way to the recent critical sudo vulnerability (CVE-2021-3156). The crashes are surprisingly due to buffer overflows in AFL itself, so we set out to fix it. Finally, we set up AFL and AFL++ running in parallel with different test cases to see if we could find the vulnerability.

The Video

Introduction

In our last video and article, we set up afl to fuzz the sudo binary in order to find the sudoedit vulnerability (CVE-2021-3156). We let it run over a day, and then decided it was time to check the results. afl's dashboard indicated that we had some crashes (49 unique ones), so there was definitely a chance that we'd found the sudoedit vulnerability! You know it: we have to investigate.

Checking Crashes

Finding the Fuzzed Argument

If you remember the way afl works and is set up, you'll know that afl stores each input that crashed the fuzzed binary in the out folder, which for our purposes, is located at /tmp/out3/.... Since we ran four parallel fuzzing processes last time, we also have four output folders in the aforementioned directory, aptly-named f1, f2, f3, and f4. Each of these folders contains a crashes subfolder, which itself contains the crashing inputs. Since they are in binary, we use hexdump to read its contents:

hexdump -C /tmp/out3/f2/crashes/id000000,sig:11,src:000641,op:havoc,rep:8

The output needs to be seen as a list of command arguments that have a null-byte separation. Also, the first one should be the program name, in argv[0]. However, our test case shows neither sudo nor sudoedit in the argv[0]. Could afl even have found the sudoedit vulnerability?

To go faster, we used grep to parse through all the output files for the keyword sudoedit using:

grep -R sudoedit .

This lovely command should sort us. It returns... nothing. Absolutely nothing. This is not at all what we were hoping to see. On one hand, we have some crashes, and on the other, none seem to be sudoedit. But, we might have found another zero-day in sudo, and that is exciting! You know what we have to do: keep digging.

Internal Testing

As a first test, we can cat (that is, output) the contents of a crashing input and directly pipe them to our sudo binary using

cat id:000049,* | /pwd/sudo-1.8.31p2/src/sudo

and we can in fact confirm that this input crashes the binary (via a segmentation fault). Excellent!

To better understand how the crash occurs, we use gdb in conjunction with the input, typing in the following into our shell:

gdb /pwd/sudo-1.8.31p2/src/sudo

Just a quick note, we installed the pwndbg plugin into the container to get a nicer gdb output. The installation consists of copy/pasting three commands, which you can find here, on the GitHub page.

We run the binary, piping one of the crashes in. As expected, the crash occurs, and we get the details about the reason. It turns out that... afl_init_argv.in_buf() is to blame? What? But afl_init_argv() is from the header file! The afl_init_argv() function also has an in_buf[MAX_CMDLINE_LEN] static char variable that holds the fake argv arguments. What's going on here?

If we take a look around with gdb, and specifically looking at the step-by-step of what happened before the crash, we find that indeed we crash inside of sudo, and we find a call rax to a function pointer that is in sudo_warn_gettext_v1. We thus have a reason to be fairly suspicious of what is going on here.

Before we continue, let's consider the possible reasons for what we are seeing, and think about how we can sort this out.

Proof of Concept

Before we carry forward with this, we wanted to create a simple proof of concept that crashes regular sudo, not the afl instrumented one, the actual system sudo. If the input crashes the system sudo, then it's a "legitimate" input. If not, we'll have a clear sign that something else is wrong in our methodology. Sounds good? Let's set it up.

The first step is to convert the crash input into an actual set of arguments we can use to execute sudo. We can actually reuse the argv-fuzz-inl.h header file to create a simple wrapper to use the argv macro to pass in the fake arguments and execute the real sudo binary with these fake arguments. Our code looks like this:

#include "argv-fuzz-inl.h"

int main(int argc, char *argv[], char *envp[])
{
	AFL_INIT_ARGV(); // argv is now the fake argv
    execve("/usr/local/bin/sudo", argv, envp);
}

It's test time: running

echo -en "sudo\x00-l\x00" | ./afl2sudo

should execute sudo -l, and in fact it does! So the wrapper works. Cool, so let's try the crash file, now!

cat /tmp/crash | ./afl2sudo

and a lovely Segmentation fault (core dumped) gets output to the console. This is excellent! This could be another zero-day! Let's further investigate with gdb, by debugging our own wrapper (yes, the one we just wrote) and piping in the crash input.

We quickly see that we crash inside of our own program, and once again, it's inside the AFL_INIT_ARGV() function; we didn't even reach the execve function to execute sudo that we called in the next line of our wrapper. Therefore, this leads us to the conclusion that we have a bug in our fuzzing setup. It's not what we wanted to conclude, but it is the reality of the problem. Let's fix it!

Counting High

We decided that the first step in our solution should be the classic "add printf statements to see what the program is doing" approach. It is certainly crude, but it works. In the afl_init_argv() function from the argv-fuzz-inl.h header file, we added the following line,

printf("[*] rc=%d ptr=%p -> %c\n", rc, ptr, *ptr);

which gives us a readout of the values of rc and the pointer. After compiling and running, we immediately see that rc's value is climbing fast, quickly reaching past 1000... even though the [MAX_CMDLINE_PAR]'s value is fixed at 1000. This means that we are in fact causing a buffer overflow, which in this case shows that we are writing pointers from our fake argv to some other memory location. This also explains why we see crashes with afl. Therefore, the crash afl found through fuzzing is a bug in afl's experimental argv wrapper code.

As our fake argv is a static buffer, so basically a global variable, our input must've caused rc to go past 1000, thereby overriding data it shouldn't be touching. This also includes a function pointer used by sudo_warn_gettext_v1, which got overwritten by a pointer from our fake argv.

Though this did not immediately crash sudo, in some instances the sudo_warn_gettext_v1 function was executed using the overwritten pointer, causing the crash that got logged by afl.

The Solution

Fixing Wrappers

With this issue clarified, we fixed the wrapper by ensuring that rc can never exceed the [MAX_CMDLINE_PAR]'s value of 1000.

while (*ptr) {
	if(rc >= MAX_CMDLINE_PAR) {
    	break;
    }
    ret[rc] = ptr;
// a lot more code goes in here, but we like properly closing the while loop for code cleanliness.
}

This should address our false positive crashes. We once again recompiled sudo, and prepared the fuzzing setup.

Running the Fuzzer

This time, we decided to spice things up a little and get some further experimentations going. Let's summarize the setup:

• The first fuzzing process is the same as what we've been using previously. We added a test case with argument flags from the sudo manual page, so that afl can use some correct arguments in its fuzzing strategy.
• The second fuzzing process includes the same flags as the first process, in addition to a sudoedit testcase, to see if the fuzzer can find the vulnerability. Using this approach helps accelerate the fuzzing, as we are leaving less information for the fuzzer to go through. This test case therefore allows us to see if our setup can find the vulnerability altogether.

Both of the previously-described cases use afl. We decided to use afl++ for the remaining two cases, which mirror the first two in their test cases. afl++ is said to be

a superior fork to Google's afl - more speed, more and better mutations, more and better instrumentation, custom module support, etc.

as mentioned on the GitHub page. To facilitate the setup if you want to follow along with us, we added the installation to the Docker file for this episode. Instead of afl, the image installs afl++. The usage is pretty similar, so we didn't need to change anything except for using afl-cc as the compiler:

CC=afl-cc ./configure --disabled-shared && make

Using any other variant such as afl-clang-fast is just a symbolic link (or "symlink") to afl-cc. With the configuration done, we build and make sudo. Now, we use the same input test cases as mentioned above, this time with afl++ instead of afl. Our four fuzzing processes are thus:

• afl fuzzer, flags from the sudo man page
• afl fuzzer, flags from the sudo man page and the sudoedit test case
• afl++ fuzzer, flags from the sudo man page
• afl++ fuzzer, flags from the sudo man page and the sudoedit test case

With those four processes now running, we have a good way of seeing the differences across the test cases and fuzzers.

The Fuzzing Dashboard

This entire time, we haven't actually discussed what the afl (or afl++) dashboard looks like, and what information it contains. We only mentioned that it was pretty to look at; it's time to talk about what we can learn from it.

There are various sections of interest on the dashboard. The process timing section presents how long the fuzzer has been running for (the wall time, to be very specific), as well as the last new path, unique crash, and unique hang. Using this section is a great way to decide early on if your fuzzing approach is working. This brings us to the question of time, and deciding when to call it quits.

Time spent fuzzing is relative. Some projects can involve fuzzing that is completed in a matter of minutes. Some could require multiple days or (dare we say it) weeks. When to decide to cut your (time) losses is entirely up to you.

The stage progress section in particular contains information about the fuzzing speed. This is a great way to see how many executions are occurring every second, but it is also a method of spotting if you're encountering a resource problem, like we have in a previous episode of this series. Look at the number though – something like 200 or 300 executions per second is really nice when you think about it! That's a lot of executing of the sudo binary. Again we need to think about the project scale, though. For us, this is probably enough. For larger fuzzing campaigns, 200 or 300 executions per second might be on the low side. The section also features the total tally of executions. Expect this number to be large.

The findings in depth section gives you information about crashes, including the total number of crashes (including unique ones). The number of timeouts is also provided; timeouts are for when the program runs into an endless loop or a lengthy calculation. After a given amount of time, a timeout is considered to have occurred and the relevant tally increases. If your timeout counter is really large, either you need to increase the timeout cutoff value, or investigate further and find out what is causing the timeout to occur on the fuzzed program's side.

The fuzzing strategies section details some mutations between runs, but we won't pretend that we are fuzzing experts, so we cannot say for sure whether some values showing up in this section of the dashboard is representative of something weird occurring either with the fuzzer or the fuzzed program.

At any rate, let's let the fuzzers run, and we'll see later what it is that they have come up with.

Final Words

In this episode of the CVE-2021-3156 sudo vulnerability series, we found some crashes using afl! Upon further investigation, we found that these crashes were not related to sudoedit. In fact, the crashes were happening instead in afl_init_argv.in_buf(), which is part of the afl_init_argv() function from the header file we used to set up the fuzzing! There, we were causing a buffer overflow, itself causing the crashes afl reported.

After ironing out this issue with a proof of concept and enforcing a limitation on the value of the rc counter, we returned to fuzzing, this time running two cases for both the afl and afl++ fuzzers. This way, we can find out what fuzzer works best, and whether they are even able to find the sudo vulnerability using either the sudo flags by themselves, or with some help by pointing the fuzzer directly at sudoedit in addition to fuzzing with the flags.

In our quest to rediscover the vulnerability, we have set up a fuzzer, crashed it, modified it, run it with test cases, crashed it again, run out of computing resources, gotten false positives... Finding a bug like this sudo vulnerability using fuzzing is not an easy task, so we are happy with the progress we have made! It has been quite the learning experience so far. In the next episode, we will go over the results of this latest fuzzing iteration, and continue on. In the meantime, happy fuzzing!

You can find the files on GitHub here: https://github.com/LiveOverflow/pwnedit/tree/main/episode04