In this article we will explore ‘Apport’, the Ubuntu crash handler. When an application crashes Apport is executed by the kernel, reads information about the crashed process, and then creates a crash report that can be sent to Ubuntu developers.
We will show how we were able to bypass several defense mechanisms, manipulate the crash handler, and get local privilege escalation.
When a process in Linux terminates abnormally, a core file is usually created by the kernel. But what exactly is a ‘core’ file?
“In computing, a core dump consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally.” ~ Wikipedia
When a process receives certain signals (for example SIGSEGV or SIGABRT), the default kernel action for these signals terminates the process and creates a coredump file. This file lets us inspect the state of the program at the time of the crash.
By default, a coredump file is named ‘core’ and placed in the current directory of the crashed process, but the path to which the coredump file is written can be configured by writing it to /proc/sys/kernel/core_pattern.
Where is Apport configured to be a crash handler? In Ubuntu the content of the core_pattern file is: “|/usr/share/apport/apport %p %s %c %d %P %E”.
Let’s break this up:
First, the | character at the start of the file. This is a feature introduced in kernel 2.6.19 to let userspace programs handle a crash: the path after the pipe is the path to the userspace program that will be executed to handle the crash (Apport in our case). The coredump content is given to the program as standard input(stdin).
Next, the % specifiers will be formatted and provided as parameters for the program. Each specifier has a different meaning, which can be found here.
%p PID of dumped process, as seen in the PID namespace in
which the process resides.
%s Number of signal causing dump.
%c Core file size soft resource limit of crashing process
(since Linux 2.6.24).
%d Dump mode—same as value returned by prctl(2)
PR_GET_DUMPABLE (since Linux 3.7).
%P PID of dumped process, as seen in the initial PID
namespace (since Linux 3.12).
%E Pathname of executable, with slashes ('/') replaced by
exclamation marks ('!') (since Linux 3.0).
Since a user-space process that creates a coredump will be running as root, we have found Apport an attractive attack surface.
Apport starts on a process crash and has two main responsibilities:
To write the coredump, Apport uses a function named ‘write_user_coredump’ which reads the coredump from the standard input and writes it to a new coredump file in the current directory of the process.
Our goal is to make Apport create a coredump file as root with our controllable user content. That will allow us to write a file as root with our payload (as the file content) in a directory to which our non-privileged user cannot write (more on that later).
You might think the coredump would be created with the Apport process UID as owner, which is root, but that’s incorrect. In fact, just before Apport writes a coredump, or reads the process information, it drops its privileges to those of the process that crashed. It does this to prevent abuse of its root permissions.
Every process in Linux has three main user and group IDs:
Before entering ‘write_user_coredump’ Apport calls the ‘drop_privileges()’ function which permanently drops privileges to those of the crashed process user (using setuid and setgid).
# Totally drop privs before writing out the reportfile.
drop_privileges()
def drop_privileges(real_only=False):
'''Change user and group to real_[ug]id
Normally that irrevocably drops privileges to the real user/group of the
target process. With real_only=True only the real IDs are changed, but
the effective IDs remain.
'''
if real_only:
# Drop any supplemental groups
if os.getuid() == 0:
os.setgroups([])
os.setregid(real_gid, -1)
os.setreuid(real_uid, -1)
else:
os.setgid(real_gid)
os.setuid(real_uid)
assert os.getegid() == real_gid
assert os.geteuid() == real_uid
assert os.getgid() == real_gid
assert os.getuid() == real_uid
‘drop_privileges’ function drops all Apport privileges according to the variables ‘real_uid’ and ‘real_gid’ taken from the crashed process. This will make Apport write a coredump using the crashed process privileges.
Those variables initialize in an early function called get_pid_info.
def get_pid_info(pid):
'''Read /proc information about pid'''
global pidstat, real_uid, real_gid, cwd, proc_pid_fd
proc_pid_fd = os.open('/proc/%s' % pid, os.O_RDONLY | os.O_PATH | os.O_DIRECTORY)
# unhandled exceptions on missing or invalidly formatted files are okay
# here -- we want to know in the log file
pidstat = os.stat('stat', dir_fd=proc_pid_fd)
# determine real UID of the target process; do *not* use the owner of
# /proc/pid/stat, as that will be root for setuid or unreadable programs!
# (this matters when suid_dumpable is enabled)
with open('status', opener=proc_pid_opener) as f:
for line in f:
if line.startswith('Uid:'):
real_uid = int(line.split()[1])
elif line.startswith('Gid:'):
real_gid = int(line.split()[1])
break
assert real_uid is not None, 'failed to parse Uid'
assert real_gid is not None, 'failed to parse Gid'
cwd = os.open('cwd', os.O_RDONLY | os.O_PATH | os.O_DIRECTORY, dir_fd=proc_pid_fd)
‘get_pid_info’ reads the /proc/pid/status file of the crashed process and parse the lines that contain “Uid:” and “Gid:” from it. It takes the values from the first columns which are RUID and RGID. We found that we can inject different UID and GID values with a crafted process name.
/proc/pid/status file contains several fields, one of them is ‘Name’ as seen below:
Name: cat
Umask: 0002
State: R (running)
Tgid: 14496
Ngid: 0
Pid: 14496
PPid: 13369
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
We found that by creating a process named ‘a\rUid: 0\rGid: 0’ and crashing it, we can manipulate ‘get_pid_info’ to iterate through the lines of /proc/pid/status and parse our injected content.
Name: a
Uid: 0
Gid: 0
Umask: 0002
State: R (running)
Tgid: 14497
Ngid: 0
Pid: 14497
PPid: 13369
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
This gave us the ability to nullify every call to drop_privileges and stay in root privileges! CVE-2021-25682
Even though we can now keep our privileges as root, we still can’t write the coredump as root. This is due to the following check in: ‘write_user_coredump’
# don't write a core dump for suid/sgid/unreadable or otherwise
# protected executables, in accordance with core(5)
# (suid_dumpable==2 and core_pattern restrictions); when this happens,
# /proc/pid/stat is owned by root (or the user suid'ed to), but we already
# changed to the crashed process' real uid
assert pidstat, 'pidstat not initialized'
if pidstat.st_uid != os.getuid() or pidstat.st_gid != os.getgid():
error_log('disabling core dump for suid/sgid/unreadable executable')
return
Apport checks our current uid against whoever created the process. Let’s assume that we invoked the process as uid 1000. If we change our process privileges to root(UID 0) Apport will think that we executed a suid executable (pidstat.st_uid == 1000 and os.getuid() == 0).
To bypass this check, we can start a suid program (pidstat.st_uid == 0) and then drop our privileges (with the vulnerability above) to UID 0 (a normal behavior is that suid program has real UID of the user that started the process).
As we don’t have permission to change the filename of the suid program, we are able to control the process name (in /proc/pid/status) by executing a symlink to the suid program whose name we can control.
The last piece of the puzzle before writing the coredump file is the process dump mode setting (%d in the process arguments).
The dump mode of the process indicates whether or not this is a suid process. This is a defense mechanism to prevent unauthorized memory access.
We noticed that the dump mode for a normal process crash is 1, and for a suid process 2.
Apport checks whether the process has a dump mode of 1 or 2, and in the case of 2 (suid binary) sets core_ulimit (size of core file to dump) to 0 and will not write any core file!
dump_mode = options.dump_mode
...
...
if dump_mode == '2':
error_log('not creating core for pid with dump mode of %s' % (dump_mode))
# a report should be created but not a core file
core_ulimit = 0
We had an idea! What if we crash a “regular” process (with a dump mode of 1), and while Apport is running we kill the process and “replace” it with a new suid process which will have the same PID as the old one.
This way we can “trick” Apport into thinking that it is handling a “regular” process whereas in reality it is handling a suid process!
This technique was used in previous research by Kevin from Github Security Lab on Apport and was extremely helpful to us.
Every Linux process has a unique PID that identifies it. PIDs are allocated sequentially from zero to MAX_PID. When MAX_PID is reached the counter starts from zero again, skipping any PID that is currently occupied. This mechanism lets us recycle the PID of a dead process.
The default value of MAX_PID in the file /proc/sys/kernel/pid_max, is 32768 for Ubuntu 18.04 and 4194304 for Ubuntu 20.04.
So how can we create 4 million PIDs before our core file is written? We need to find a way to “stall” the Apport process for enough time to fork that many PIDs, so the “switcheroo” can occur right before ‘get_pid_info’ is called.
When Apport starts it first locks /var/run/apport.lock. If there is already an instance of Apport running, the new Apport will “hang” for 30 seconds, trying to lock the file, and then exit. If we can find a way to run another instance of Apport before our crash, that will give us 30 seconds to switch the PID!
Are 30 seconds enough to generate MAX_PIDs? In Ubuntu 18.04 we can easily do it in a few seconds, but in Ubuntu 20.04, forking four million PIDs will take longer (3-4 minutes on my VM), depending on the CPU, number of cores, system load etc.
What we can do to generate MAX_PIDs in that 30 seconds is to crash the process only after we fork around 4 million processes, then we will have 30 seconds to fork the rest of the PIDs.
To stall the first instance of Apport we create a .crash file in /var/crash, this will be a FIFO file which will freeze the first process until someone writes to the file (because Apport tries to read the FIFO) CVE-2021-25684, in that time /var/run/apport.lock is locked and any additional Apport instance will wait for 30 seconds.
However, we soon found that Apport also has defense mechanisms against process replacing.
# Check if the process was replaced after the crash happened.
# Ideally we'd use the time of dump value provided by the kernel,
# but this would be a backwards-incompatible change that would
# require adding support to apport outside and inside of containers.
apport_start = get_apport_starttime()
process_start = get_process_starttime()
if process_start > apport_start:
error_log('process was replaced after Apport started, ignoring')
sys.exit(0)
The functions will open two files /proc/apport_pid/stat and /proc/crash_pid/stat and read the start time of each process from the 22nd column (divided by space delimiter). Interestingly, the stat file also holds the name of the process, in the 2nd column. Remember that our malicious process name is ‘a\rUid: 0\rGid: 0’ which includes two spaces! So suddenly the 22nd column of the stat file becomes 24th column. And conveniently the new 22nd column now contains a number smaller than apport_start CVE-2021-25683.
As we have now successfully bypassed all defense mechanisms, we can now create a coredump owned by root in a designated directory. Next we need to exploit it to get root permissions.
The coredump contains the crashed process memory. We want to include our content in the core file. To do this, we can create a string variable in the program that includes our content. When the coredump is generated, the memory will contain the variable value.
We tried to find a mechanism we could abuse to “execute” our coredump. Luckily for us, we found a great blog post written by Shiga from Flatt Security which enabled us to select our target. The target was Logrotate.
Logrotate is a program that by default runs daily on Ubuntu to rotate, compress, remove and mail log files. Logrotate keeps configurations in /etc/logrotate.d which we can abuse.
Because of the non-strict configuration parsing that logrotate uses, we are able to place our coredump in that directory as a valid configuration file.
In the coredump file, we will include a valid logrotate configuration which will get executed by logrotate.
Our final exploit strategy is:
The second Apport instance will continue execution, now get_pid function will be under the new suid process but Apport thinks that the dump_mode variable is 1 and will write our coredump in /etc/logrotate.d/.
We included a reverse shell payload in the coredump configuration, which connects to our netcat, and we should see a root shell on the next execution of logrotate.
Working exploit can be found here.
This was really exciting vulnerability research, during which we explored ‘Apport’, the Ubuntu crash handler, and showed how we achieved LPE with the vulnerabilities we discovered. Kudos to the Ubuntu security team for their prompt response! They have issued a patch that fixes these vulnerabilities in the latest version of Apport.
Many thanks to Jonathan Afek and Gal Zror for their help in this research.