We're logging crashes with AFL as we try to fuzz our way to the recent critical
sudo vulnerability (CVE-2021-3156). The crashes are surprisingly due to buffer overflows in AFL itself, so we set out to fix it. Finally, we set up AFL and AFL++ running in parallel with different test cases to see if we could find the vulnerability.
In our last video and article, we set up
afl to fuzz the
sudo binary in order to find the
sudoedit vulnerability (CVE-2021-3156). We let it run over a day, and then decided it was time to check the results.
afl's dashboard indicated that we had some crashes (49 unique ones), so there was definitely a chance that we'd found the
sudoedit vulnerability! You know it: we have to investigate.
Finding the Fuzzed Argument
If you remember the way
afl works and is set up, you'll know that
afl stores each input that crashed the fuzzed binary in the
out folder, which for our purposes, is located at
/tmp/out3/.... Since we ran four parallel fuzzing processes last time, we also have four output folders in the aforementioned directory, aptly-named
f4. Each of these folders contains a
crashes subfolder, which itself contains the crashing inputs. Since they are in binary, we use
hexdump to read its contents:
The output needs to be seen as a list of command arguments that have a null-byte separation. Also, the first one should be the program name, in
argv. However, our test case shows neither
sudoedit in the
afl even have found the
To go faster, we used
grep to parse through all the output files for the keyword
This lovely command should sort us. It returns... nothing. Absolutely nothing. This is not at all what we were hoping to see. On one hand, we have some crashes, and on the other, none seem to be
sudoedit. But, we might have found another zero-day in
sudo, and that is exciting! You know what we have to do: keep digging.
As a first test, we can
cat (that is, output) the contents of a crashing input and directly pipe them to our
sudo binary using
and we can in fact confirm that this input crashes the binary (via a segmentation fault). Excellent!
To better understand how the crash occurs, we use
gdb in conjunction with the input, typing in the following into our shell:
Just a quick note, we installed the
pwndbg plugin into the container to get a nicer
gdb output. The installation consists of copy/pasting three commands, which you can find here, on the GitHub page.
We run the binary, piping one of the crashes in. As expected, the crash occurs, and we get the details about the reason. It turns out that...
afl_init_argv.in_buf() is to blame? What? But
afl_init_argv() is from the header file! The
afl_init_argv() function also has an
static char variable that holds the fake
argv arguments. What's going on here?
If we take a look around with
gdb, and specifically looking at the step-by-step of what happened before the crash, we find that indeed we crash inside of
sudo, and we find a call
rax to a function pointer that is in
sudo_warn_gettext_v1. We thus have a reason to be fairly suspicious of what is going on here.
Before we continue, let's consider the possible reasons for what we are seeing, and think about how we can sort this out.
Proof of Concept
Before we carry forward with this, we wanted to create a simple proof of concept that crashes regular
sudo, not the
afl instrumented one, the actual system
sudo. If the input crashes the system
sudo, then it's a "legitimate" input. If not, we'll have a clear sign that something else is wrong in our methodology. Sounds good? Let's set it up.
The first step is to convert the crash input into an actual set of arguments we can use to execute
sudo. We can actually reuse the
argv-fuzz-inl.h header file to create a simple wrapper to use the
argv macro to pass in the fake arguments and execute the real
sudo binary with these fake arguments. Our code looks like this:
It's test time: running
sudo -l, and in fact it does! So the wrapper works. Cool, so let's try the crash file, now!
and a lovely
Segmentation fault (core dumped) gets output to the console. This is excellent! This could be another zero-day! Let's further investigate with
gdb, by debugging our own wrapper (yes, the one we just wrote) and piping in the crash input.
We quickly see that we crash inside of our own program, and once again, it's inside the
AFL_INIT_ARGV() function; we didn't even reach the
execve function to execute
sudo that we called in the next line of our wrapper. Therefore, this leads us to the conclusion that we have a bug in our fuzzing setup. It's not what we wanted to conclude, but it is the reality of the problem. Let's fix it!
We decided that the first step in our solution should be the classic "add
printf statements to see what the program is doing" approach. It is certainly crude, but it works. In the
afl_init_argv() function from the
argv-fuzz-inl.h header file, we added the following line,
printf("[*] rc=%d ptr=%p -> %c\n", rc, ptr, *ptr);
which gives us a readout of the values of
rc and the pointer. After compiling and running, we immediately see that
rc's value is climbing fast, quickly reaching past 1000... even though the
[MAX_CMDLINE_PAR]'s value is fixed at 1000. This means that we are in fact causing a buffer overflow, which in this case shows that we are writing pointers from our fake
argv to some other memory location. This also explains why we see crashes with
afl. Therefore, the crash
afl found through fuzzing is a bug in
argv wrapper code.
As our fake
argv is a static buffer, so basically a global variable, our input must've caused
rc to go past 1000, thereby overriding data it shouldn't be touching. This also includes a function pointer used by
sudo_warn_gettext_v1, which got overwritten by a pointer from our fake
Though this did not immediately crash
sudo, in some instances the
sudo_warn_gettext_v1 function was executed using the overwritten pointer, causing the crash that got logged by
With this issue clarified, we fixed the wrapper by ensuring that
rc can never exceed the
[MAX_CMDLINE_PAR]'s value of 1000.
This should address our false positive crashes. We once again recompiled
sudo, and prepared the fuzzing setup.
Running the Fuzzer
This time, we decided to spice things up a little and get some further experimentations going. Let's summarize the setup:
- The first fuzzing process is the same as what we've been using previously. We added a test case with argument flags from the
sudomanual page, so that
aflcan use some correct arguments in its fuzzing strategy.
- The second fuzzing process includes the same flags as the first process, in addition to a
sudoedittestcase, to see if the fuzzer can find the vulnerability. Using this approach helps accelerate the fuzzing, as we are leaving less information for the fuzzer to go through. This test case therefore allows us to see if our setup can find the vulnerability altogether.
Both of the previously-described cases use
afl. We decided to use
afl++ for the remaining two cases, which mirror the first two in their test cases.
afl++ is said to be
a superior fork to Google's afl - more speed, more and better mutations, more and better instrumentation, custom module support, etc.
as mentioned on the GitHub page. To facilitate the setup if you want to follow along with us, we added the installation to the Docker file for this episode. Instead of
afl, the image installs
afl++. The usage is pretty similar, so we didn't need to change anything except for using
afl-cc as the compiler:
Using any other variant such as
afl-clang-fast is just a symbolic link (or "symlink") to
afl-cc. With the configuration done, we build and make
sudo. Now, we use the same input test cases as mentioned above, this time with
afl++ instead of
afl. Our four fuzzing processes are thus:
aflfuzzer, flags from the
aflfuzzer, flags from the
sudo manpage and the
afl++fuzzer, flags from the
afl++fuzzer, flags from the
sudo manpage and the
With those four processes now running, we have a good way of seeing the differences across the test cases and fuzzers.
The Fuzzing Dashboard
This entire time, we haven't actually discussed what the
afl++) dashboard looks like, and what information it contains. We only mentioned that it was pretty to look at; it's time to talk about what we can learn from it.
There are various sections of interest on the dashboard. The
process timing section presents how long the fuzzer has been running for (the wall time, to be very specific), as well as the last new path, unique crash, and unique hang. Using this section is a great way to decide early on if your fuzzing approach is working. This brings us to the question of time, and deciding when to call it quits.
Time spent fuzzing is relative. Some projects can involve fuzzing that is completed in a matter of minutes. Some could require multiple days or (dare we say it) weeks. When to decide to cut your (time) losses is entirely up to you.
stage progress section in particular contains information about the fuzzing speed. This is a great way to see how many executions are occurring every second, but it is also a method of spotting if you're encountering a resource problem, like we have in a previous episode of this series. Look at the number though – something like 200 or 300 executions per second is really nice when you think about it! That's a lot of executing of the
sudo binary. Again we need to think about the project scale, though. For us, this is probably enough. For larger fuzzing campaigns, 200 or 300 executions per second might be on the low side. The section also features the total tally of executions. Expect this number to be large.
findings in depth section gives you information about crashes, including the total number of crashes (including unique ones). The number of timeouts is also provided; timeouts are for when the program runs into an endless loop or a lengthy calculation. After a given amount of time, a timeout is considered to have occurred and the relevant tally increases. If your timeout counter is really large, either you need to increase the timeout cutoff value, or investigate further and find out what is causing the timeout to occur on the fuzzed program's side.
fuzzing strategies section details some mutations between runs, but we won't pretend that we are fuzzing experts, so we cannot say for sure whether some values showing up in this section of the dashboard is representative of something weird occurring either with the fuzzer or the fuzzed program.
At any rate, let's let the fuzzers run, and we'll see later what it is that they have come up with.
In this episode of the CVE-2021-3156
sudo vulnerability series, we found some crashes using
afl! Upon further investigation, we found that these crashes were not related to
sudoedit. In fact, the crashes were happening instead in
afl_init_argv.in_buf(), which is part of the
afl_init_argv() function from the header file we used to set up the fuzzing! There, we were causing a buffer overflow, itself causing the crashes
After ironing out this issue with a proof of concept and enforcing a limitation on the value of the
rc counter, we returned to fuzzing, this time running two cases for both the
afl++ fuzzers. This way, we can find out what fuzzer works best, and whether they are even able to find the
sudo vulnerability using either the
sudo flags by themselves, or with some help by pointing the fuzzer directly at
sudoedit in addition to fuzzing with the flags.
In our quest to rediscover the vulnerability, we have set up a fuzzer, crashed it, modified it, run it with test cases, crashed it again, run out of computing resources, gotten false positives... Finding a bug like this
sudo vulnerability using fuzzing is not an easy task, so we are happy with the progress we have made! It has been quite the learning experience so far. In the next episode, we will go over the results of this latest fuzzing iteration, and continue on. In the meantime, happy fuzzing!
You can find the files on GitHub here: https://github.com/LiveOverflow/pwnedit/tree/main/episode04