CVE-2022-31813: Forwarding addresses is hard

Written by Gaetan Ferry - 26/07/2022 - in Exploit , Pentest - Download

A few weeks ago, version 2.4.54 of Apache HTTPD server was released. It includes a fix for CVE-2022-31813, a vulnerability we identified in mod_proxy that could affect unsuspecting applications served by an Apache reverse proxy.
Let's see why it is rated as low in the software changelog and why it still matters.
TL;DR: when in doubt, patch!

Underlying concepts

If you are already familiar with HTTP proxy chains, and particularly the transmission of requests' information from a client to an app, you may skip to So, what is wrong with mod_proxy ?. If you only want the digest of the story, skip to Wrapping up.

The good application, the bad IP address and the ugly proxy

The time when all web applications were exposed directly on the internet is long gone. We now hardly see web infrastructures that do not contain a reverse proxy, load balancer, or both, that fronts the application(s). The reasons for that are numerous, performance, costs, IPv4 exhaustion, etc. This evolution in the standard architecture brought a new set of issues and technical challenges.

One of these problematics is source IP management. When using a reverse proxy (or other fronting component), HTTP requests sent to the application containers are issued from the IP address of this equipment and not the real client one. This is a problem when applications need this information, which is often the case, be it for logging or filtering purposes.

To solve this challenge, reverse proxies started injecting information about the original HTTP request in headers that are forwarded to the application. Those include entries for the client's IP address, the originally requested host, the protocol, etc. Up until (not that) recently, no standard field existed in the HTTP specification to store this information. Therefore, non-standard headers have been created:

X-Forwarded-{Host,For,Proto}
X-Real-IP
Client-IP

X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto have then become the de facto standard. This still holds true, despite the publication of RFC 7239, in 2014, which standardized a Forwarded header field.

The operating of this headers is pretty straightforward. The values in them represent a piece of information relating to the original client's request:

X-Forwarded-Proto: contains the original request protocol (i.e. HTTP/HTTPS).
X-Forwarded-Host: contains the original value of the Host header as received by the proxy.
X-Forwarded-For: contains the original client's IP address, along with the IP addresses of all the previous intermediate proxies.

In essence, that's all for the forwarded headers.

Hop hop hop-by-hop

Hop-by-hop headers is an HTTP mechanism that allows instructing proxies about which headers should be forwarded along a request (end-to-end) and which ones should be immediately treated and dropped (hop-by-hop).

There are a few headers that are always considered as hop-by-hop. They are:

Connection
Keep-Alive
Proxy-Authenticate
Proxy-Authorization
TE
Trailers
Transfer-Encoding
Upgrade

While all other headers are considered end-to-end, it is possible to add some of them to the preceding set, by listing them in the Connection header, right after the classical Close or Keep-Alive keyword. Doing so, all listed headers should be treated and dropped by the first proxy that receives them.

This mechanism exposes an interesting attack surface, as explained by Nathan Davison in his 2019 article [HBH].

So, what is wrong with mod_proxy ?

As all reverse proxies, Apache HTTPD with mod_proxy tries its best to follow both the real and defacto standards. This means the proxy component will add the X-Forwarded headers, as stated in the documentation.

When acting in a reverse-proxy mode (using the ProxyPass directive, for example), mod_proxy_http adds several request headers in order to pass information to the destination server. These headers are:

X-Forwarded-For: The IP address of the client.

X-Forwarded-Host: The original host requested by the client in the Host HTTP request header.

X-Forwarded-Server: The hostname of the proxy server.

In the meantime, the proxy also respects the hop-by-hop headers indication and removes them from the forwarded request.

The problem is, if the X-Forwarded headers are listed as hop-by-hop, Apache fails at forwarding them to the upstream infrastructure. In a normal situation, when receiving a legitimate request, for example from this curl call:

$ curl myhost.local/test

an application behind an Apache mod_proxy reverse proxy receives the following HTTP request:

GET / HTTP/1.1
Host: localhost:5000
User-Agent: curl/7.74.0
Accept: */*
X-Forwarded-For: 192.168.42.42
X-Forwarded-Host: myhost.local
X-Forwarded-Server: 127.0.1.1
Connection: Keep-Alive

But if the same request is sent, including the X-Forwarded headers in the hop-by-hop list:

$ curl -H "Connection: close, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Server" myhost.local/test

the received request is different:

GET / HTTP/1.1
Host: localhost:5000
User-Agent: curl/7.74.0
Accept: */*
Connection: Keep-Alive

What is interesting here, is that the received request is absolutely no different from one that would have been sent from the loopback interface directly on the application server.

The cart before the horse

This weird behavior can be quickly tracked down in the source code of the mod_proxy HTTPD module. Everything interesting is packaged in the proxy_util.c file (modules/proxy/proxy_util.c).

The function responsible for handling hop-by-hop headers is ap_proxy_clear_connection. It is even nicely indicated in the code documentation.

/**
 * Remove all headers referred to by the Connection header.
 * Returns -1 on error. Otherwise, returns 1 if 'Close' was seen in
 * the Connection header tokens, and 0 if not.
 */
static int ap_proxy_clear_connection(request_rec *r, apr_table_t *headers)
{
    int closed = 0;
    header_connection x;

    x.pool = r->pool;
    x.array = NULL;
    x.error = NULL;
    x.is_req = (headers == r->headers_in);

    apr_table_unset(headers, "Proxy-Connection");

    apr_table_do(find_conn_headers, &x, headers, "Connection", NULL);
    apr_table_unset(headers, "Connection");

    //Skipped

    if (x.array) {
        int i;
        for (i = 0; i < x.array->nelts; i++) {
            const char *name = APR_ARRAY_IDX(x.array, i, const char *);
            ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r, APLOGNO(02807)
                          "Removing header '%s' listed in Connection header",
                          name);
            if (!ap_cstr_casecmp(name, "close")) {
                closed = 1;
            }
            apr_table_unset(headers, name);
        }
    }

    return closed;
}

The code snippet responsible for adding the X-Forwarded headers is located in the ap_proxy_create_hdrbrgd function. Once again, a nice code doc indicates the feature.

PROXY_DECLARE(int) ap_proxy_create_hdrbrgd(apr_pool_t *p,
                                        apr_bucket_brigade *header_brigade,
                                        request_rec *r,
                                        proxy_conn_rec *p_conn,
                                        proxy_worker *worker,
                                        proxy_server_conf *conf,
                                        apr_uri_t *uri,
                                        char *url, char *server_portstr,
                                        char **old_cl_val,
                                        char **old_te_val)
{
//Skipped
/* X-Forwarded-*: handling
 *
 * SKIPPED
 */
if (dconf->add_forwarded_headers) {
    if (PROXYREQ_REVERSE == r->proxyreq) {
        const char *buf;

        /* Add X-Forwarded-For: so that the upstream has a chance to
         * determine, where the original request came from.
         */
        apr_table_mergen(request_headers, "X-Forwarded-For",
                         r->useragent_ip);

        /* Add X-Forwarded-Host: so that upstream knows what the
         * original request hostname was.
         */
        if ((buf = apr_table_get(r->headers_in, "Host"))) {
            apr_table_mergen(request_headers, "X-Forwarded-Host", buf);
        }

        /* Add X-Forwarded-Server: so that upstream knows what the
         * name of this proxy server is (if there are more than one)
         * XXX: This duplicates Via: - do we strictly need it?
         */
        apr_table_mergen(request_headers, "X-Forwarded-Server",
                         r->server->server_hostname);
    }
}

What is important is that, while the X-Forwarded addition code is called at line 4050 (or so), the ap_proxy_clear_connection function is called AFTER that, around line 4083.

You got it now, mod_proxy, as bundled Apache HTTPD version 2.4.53, first fills the X-Forwarded headers before removing them right away. This is a fail.

This issue was introduced in version 2.2.1, more precisely in revision 377053 on the 1st of April 2006 (a bad taste April fool). Which makes this bug a 16 years old one.

How bad is that?

Well, you are probably not going to compromise an Apache server with that issue. The fact is, this issue falls in the "application impacting" category. Its consequences will depend on the application and infrastructure setup.

There are (nearly) no generic, evergreen, attack scenario. Instead of that, multiple use cases can be found, depending on the context.

A look into popular web frameworks

Depending on the frameworks used by applications, this issue might be exploitable or not. Indeed, all exploitation scenarios depend on the willingness of frameworks to use and trust the X-Forwarded headers. Because the X-Forwarded headers can sometimes be spoofed and used to attack applications, web frameworks developed defensive measures to protect their users. They are mostly of two categories:

Proxy trust settings: when users need to define a setting telling the application is behind proxies and which addresses should be considered as a trusted forwarder.
Do nothing: when applications provide no mean to exploit the X-Forwarded headers. In that case, plugins or other middleware often exist to handle the case.

Let's see common examples.

ExpressJS

When using ExpressJS, a setting exists to consider the X-Forwarded headers. Its name is trust proxy, and it accepts multiple parameter types depending on the goal to achieve.

Express documentation[EXPRIP] indicates that the proxy_addr package is used to manage the headers' treatment. What is interesting is that, when the received request does not include X-Forwarded headers, the request's values are kept as the legitimate ones.

That means the client address will be set to the remote address (the proxy's one), the virtual host name will be set to the Host header value (which might differ from the client's requested one), etc.

Flask

Flask documentation[FLASKIP] declares that the framework itself does not trust or use the X-Forwarded header, but it proposes to use a Werkzeug middleware to handle that task.

The X-Forwarded-For Proxy Fix component is indeed intended to receive and treat incoming X-Forwarded headers. It accepts a configuration telling the number of values to trust from each header type (corresponding to those added by trusted proxies, the next one being the actual value to use).

The behavior is very much the same as for Express. When no X-Forwarded headers are set, the actual requests' values are kept.

Django

Since version 1.1 in 2009, Django has removed the support for the X-Forwarded-For header [DJANIP].

Removed SetRemoteAddrFromForwardedFor middleware¶

[...] It has been demonstrated that this mechanism cannot be made reliable enough for general-purpose use, and that (despite documentation to the contrary) its inclusion in Django may lead application developers to assume that the value of REMOTE_ADDR is “safe” or in some way reliable as a source of authentication.

The statement in the changelog is absolutely true. But this leaves users of the framework with no easy solution to handle incoming addresses, and they might go wrong implementing a custom workaround.

The wild internet (stackoverflow) tends to recommend the use of django-ipware. This one is highly configurable and, unlike Express and Flask, accepts to return no address if a trusted proxy is configured but not found in the chain. However, being so configurable, it is also possible to completely mess up. The consequences would be out of scope of this post.

Tomcat

The behavior of Tomcat regarding X-Forwarded headers is somewhat a mix between django-ipware and Flask or Express. Tomcat himself does not take the headers into account. However, it provides a number of Valves that can be configured to do that.

The RemoteIPValve is as configurable as django-ipware. But while it is clearly possible to misconfigure it, even when properly configured, it will keep the requests values when no X-Forwarded header is provided.

For example, the valve can be configured to consider a single IP as a trusted proxy.

<Valve className="org.apache.catalina.valves.RemoteIpValve" hostHeader="x-forwarded-host"  trustedProxies="192.168.122.1" />

In that case, forwarding a request containing an X-Forwarded-For header gives the expected, legitimate, result.

$ curl http://srv.local:8080/ipinfo.jsp -H "X-Forwarded-For: 1.0.0.0" # request sent from the proxy

<html>
<body>
    <h2>Client IP is: 1.0.0.0</h2>
</body>
</html>

When no such header is provided, the trusted proxy address is kept instead.

$ curl http://srv.local:8080/ipinfo.jsp # request sent from the proxy

<html>
<body>
    <h2>Client IP is: 192.168.122.1</h2>
</body>
</html>

WEN ETA attack scenario

Now that we know that most application framework will just accept request coming directly from a reverse proxy, let's see what we can actually achieve.

Proxy IP address spoofing

This is pretty obvious. As long as you can send requests on behalf of the Apache reverse proxy, you are automatically spoofing its IP address. This is particularly interesting when both the application and Apache reverse proxy are hosted on the same machine. In that case, the request to the application will appear to come from localhost.

Performing IP based access control, while whitelisting localhost as a trusted address, is not uncommon. It is even what allowed us to identify the mod_proxy issue in the first place during an intrusion test.

A quick dumb search through open source projects can already lead to some applications that would be affected when set behind an Apache reverse-proxy.

https://grep.app/search?q=%5C.remote_addr%20%3F%3D%3D%20%3F%5B%27%22%5D127&regexp=true&filter[lang][0]=Python

The typical code snippet that you would look for is, for instance, demonstrated in the Ziconius/FudgeC2 project returned by previous search.

def shutdown_listener():
    if request.remote_addr == "127.0.0.1":
        shutdown_hook = request.environ.get('werkzeug.server.shutdown')
        if shutdown_hook is not None:
            shutdown_hook()

In that case, having an Apache reverse proxy in front of the app would lead to an authentication bypass.

Arbitrary address spoofing

In some situations, it might be possible to extend the previous attack a bit. Particularly, if the application or framework accepts other sources for IP addresses as backup for X-Forwarded-For. In that case, forcing the X-Forwarded-For header to be removed while adding an alternative could allow spoofing an arbitrary IP address.

This is the case with a default configuration of django-ipware when the IPWARE_META_PRECEDENCE_ORDER setting is left untouched. Depending on the other settings, removing the X-Forwarded-For header might lead either to no IP address or the proxy one being returned by the component.

$ curl -i 192.168.122.225/test/ip/

HTTP/1.1 200 OK
[SKIPPED]

Hello 192.168.122.1

$ curl -H "Connection: close, X-Forwarded-For" -i 192.168.122.225/test/ip/

HTTP/1.1 200 OK
[SKIPPED]

Hello 127.0.0.1

But in that case, spoofing an arbitrary IP becomes possible by adding a Client-IP header to the request.

$ curl -H "Client-IP: 10.10.10.10" -H "Connection: close, X-Forwarded-For" -i 192.168.122.225/test/ip/

HTTP/1.1 200 OK
Date: Thu, 07 Jul 2022 14:34:50 GMT
Server: WSGIServer/0.2 CPython/3.9.2
Content-Type: text/html; charset=utf-8
X-Frame-Options: DENY
Content-Length: 17
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
Connection: close

Hello 10.10.10.10

Host injection / host filter bypass

So far we only worked with the X-Forwarded-For header to achieve some kind of IP address spoofing. X-Forwarded-Host can also be a good attack option. Particularly in complex architectures, it might be possible to access the default VirtualHosts of backend servers or applications while those are not intended to be accessed from public access.

In fact, what is pretty funny is that this exploitation scenario has already been used in the wild, as part of a bigger exploit chain. Related vulnerability is CVE-2022-1388, an authentication bypass to remote code execution in F5 BIG-IP, which you have probably heard about.

If you read the detailed explanation about this issue [F5VULN], you might notice the following statement:

Connection: X-F5-Auth-Token, X-Forwarded-Host

This prohibits Jetty from knowing that the request was provided by Apache and treats the request as if it were done locally.

And this is actually the issue we are discussing. By removing the X-Forwarded-Host header, the jetty server is tricked into considering the request as local, allowing access to the administration application.

Denial of service

Because we can send requests on behalf of the reverse proxy, we could try to abuse rate limiting or anti-bruteforce mechanisms. For example, imagine a fail2ban protection set on failed authentications from an IP address. Such a protection could be abused to force the application server to blacklist the reverse proxy's IP address.

Wrapping up

Apache HTTPD mod_proxy between versions 2.2.1 and 2.4.53, does not fill the X-Forwarded headers when those are listed as hop-by-hop. Applications hosted behind it can misunderstand the real client's IP address or requested hostname.

Depending on the applications and architectures, this can lead to an authentication or filtering bypass, an IP address spoofing or a denial of service. This is illustrated by vulnerability CVE-2022-1388, which exploit uses mod_proxy's issue to access a private application.

The issue is fixed in Apache HTTPD version 2.4.54.

Timeline

Time	Event
05/10	Issue reported to HTTPD security team.
05/18	Issue is refused by HTTPD security team.
05/18	Further details added to the case.
05/30	Issue accepted.
06/08	Patch released.
06/09	CVE published.

References

[HBH]	Abusing HTTP hop-by-hop request headers https://nathandavison.com/blog/abusing-http-hop-by-hop-request-headers
[EXPRIP]	Express behind proxies https://expressjs.com/en/guide/behind-proxies.html
[FLASKIP]	Tell Flask it is Behind a Proxy https://flask.palletsprojects.com/en/2.1.x/deploying/proxy_fix/
[DJANIP]	Django 1.1 release notes https://docs.djangoproject.com/en/4.0/releases/1.1/
[F5VULN]	F5 BIG-IP Remote Code Execution Vulnerability CVE-2022-1388 https://blog.cyble.com/2022/05/12/f5-big-ip-remote-code-execution-vulnerability-cve-2022-1388/

CVE-2022-31813: Forwarding addresses is hard

Underlying concepts

The good application, the bad IP address and the ugly proxy

Hop hop hop-by-hop

So, what is wrong with mod_proxy ?

The cart before the horse

How bad is that?

A look into popular web frameworks

ExpressJS

Flask

Django

Tomcat

WEN ETA attack scenario

Proxy IP address spoofing

Arbitrary address spoofing

Host injection / host filter bypass

Denial of service

Wrapping up

Timeline

References

Other publications

Relaying Kerberos over SMB using krbrelayx

Exploiting a Blind Format String Vulnerability in Modern Binaries: A Case Study from Pwn2Own Ireland 2024

Forensic analysis of bitwarden self-hosted server

Contact us

PARIS

TOULOUSE

LYON

RENNES

LILLE