Diving into ADB protocol internals (2/2)

Dealing with emulators: emu requests

Emulators prove very useful for managing multiple versions of the same device or fleets of devices. In the adb ecosystem, they are treated like physical devices but also offer additional features like sending text messages, changing location… Emulators are also essential for automated testing, whether on APKs or low-level native tools.

Android provides a comprehensive suite of tools called Command Line Tools to manage them and available for download here. These tools include apkanalyzer to gather information about APKs, lint to analyze source code but also a particularly interesting tool for our use case, sdkmanager. This tool allows you to download SDKs from different versions of Android but also system_images, referring to specific versions of Android operating system. These images can then be executed in the Android emulator, and are then called as AVD (Android Virtual Device). These AVDs are managed by another tool called avdmanager (and also available in the command line tools), allowing for creation, deletion, and listing.

To download and create a brand-new virtual device, it’s straightforward:

# First download and install command line tools...

# Download desired system image (here API 35 +> Android 15)
sdkmanager "platform-tools" "platforms;android-35" "system-images;android-35;google_apis;x86_64"

# Create an AVD from this system image
# DEVICE_ID refers to the device id retrieved via `avdmanager list avd` command
avdmanager create avd -n AVD_NAME -k "system-images;android-34;google_apis;x86_64" -d DEVICE_ID

# Launch AVD
emulator -avd AVD_NAME

After being launched, emulators appear as standard Android devices when running adb devices command, distinguished by their names (starting with emulator-NNNN). These devices automatically register themselves with the adb server. The NNNN value represents the port number on which the emulator listens for incoming connections. This port can be used to directly interact with it.

From a protocol perspective, clients must authenticate themselves to the emulator device using a token, latter being located by default in $HOME/.emulator_console_auth_token file. This token is created when the emulator starts if it doesn’t already exist. It is then loaded by the emulator and must be sent at every TCP connection initiated on its port. From a security perspective, this token only prevent attackers from accessing emulators when coming from outside the system actually emulating them. An attacker with access to the system can easily load this token and authenticate to the emulator.

Once again, we’re working with an unencrypted, text-based protocol that’s easy to manipulate directly from the command line:

1. The client opens a TCP connection to the emulator’s port
2. The emulator responds with authentication instructions
3. The client sends the “auth TOKEN” message, where the TOKEN is retrieved from $HOME/.emulator_console_auth_token file
4. The emulator either acknowledges with an OK or rejects it with a KO
5. The client can now send commands to the emulator (a list is available when sending help or help-verbose commands)
6. The connection is closed when the underlying TCP connection is closed

After having explored the interactions between an adb client and an emulator device, we can now focus on the most interesting part: how an adb server communicates with an end device.

Host-to-device protocol

Android introduces a new protocol, distinct from the two we previously studied, between adb server and end devices. This protocol relies on “messages” and unlike the previous ones has not been designed to be human-readable. It is transport-agnostic, and can be carried over either TCP/IP or USB. TCP/IP is typically used for network connections (e.g., Smart TVs, Android TVs, or smartphones), while USB requires the device to be physically connected.

Studying this protocol involved using the powerful USB sniffing capabilities of wireshark to observe traffic and analyze how device (or clients) react to incoming messages. We also examined ADB source code (and the associated documentation) to understand the purpose of each field are how they interconnect.

In the following sections, we will explain how the authentication process works, how commands are executed on the device, and how synchronization tasks, such as file pushing, are handled.

Protocol overview

As mentioned earlier, this protocol is based on “messages” that consist of a 24-byte header followed by the actual data. This structure is taken from adb source code1

struct message {
    unsigned command;
    unsigned arg0;
    unsigned arg1;
    unsigned data_length;
    unsigned data_crc32;
    unsigned magic;
};

A message is composed of 6 unsigned 32-bit words, each representing a distinct protocol field:

• command: An unique identifier for the command (more details later)
• arg0: The first argument, specific to each command’s semantics.
• arg1: The second argument, specific to each command’s semantics.
• data_length: The length of data that follows
• data_crc32: The checksum of the data (or 0 if no data is present)
• magic: A magic value, calculated as xor(command, 0xFFFFFFFF)

When starting a connection, both sides generate an identifier, referred to as the Local identifier. This identifier helps recipients to identify the connection. It is then sent in nearly every message, along with the Remote identifier, which corresponds to the recipient’s identifier (the fields are inverted when receiving a message, so the remote identifier corresponds to the local identifier).

Below is a list of available commands, their purpose and the meaning of the arg0 and arg1 fields:

Authentication process

When a device is connected over USB to a computer, the ADB server detects the connection and attempts to authenticate with the device, initially without any user interaction. If the client has never been authenticated before, a prompt is displayed asking the user to accept a key fingerprint, which corresponds to the fingerprint of the ADB public key (usually stored in $HOME/.android/adbkey.pub)

Under the hood, a mechanism is set up to allow either already registered keys to be accepted or new keys to be added:

1. A CNXN command is sent to device to initiate the connection process
2. The server responds with an AUTH (TOKEN (1)) message, providing random data to sign with the client’s private key
3. The client signs this data using his private key, and sends it back to the server with another AUTH (SIGNATURE (2)) message
   1. If the signature can be verified with a known private key, the server accepts the connection with a CNXN message
   2. If the signature cannot be verified, the client can send his public key using a AUTH (RSAPUBLICKEY (3)) message, and the server will prompt the device to accept this new public key (via its fingerprint)

One issue encountered during the development of this authentication process is that the public key is not sent using standard PEM or DER formats, but rather a custom one. This can be found by inspecting the ADB source code, specifically under libcrypto_utils/android_pubkey.cpp. Once filled, the following structure is encoded in base64 and sent to the server.

// Public keys are stored as a sequence of little-endian 32 bit words. Note that Android only supports little-endian
// processors, so we don't do any byte order conversions when parsing the binary struct.
struct RSAPublicKey {
  // Modulus length. This must be ANDROID_PUBKEY_MODULUS_SIZE.
  uint32_t modulus_size_words;
  // Precomputed montgomery parameter: -1 / n[0] mod 2^32
  uint32_t n0inv;
  // RSA modulus as a little-endian array.
  uint8_t modulus[ANDROID_PUBKEY_MODULUS_SIZE];
  // Montgomery parameter R^2 as a little-endian array.
  uint8_t rr[ANDROID_PUBKEY_MODULUS_SIZE];
  // RSA modulus: 3 or 65537
  uint32_t exponent;
};

When interacting with an end-device directly over TCP/IP, the process differs significantly. The public key is transmitted as part of the client certificate within a TLS handshake. The device verifies whether the certificate corresponds to a known public key. If it does, the connection is accepted seamlessly. Otherwise, the device prompts the user to approve the new public key.

Run a command on the device

After completing the authentication process, the protocol splits into two main categories: standard commands and sync commands, similar to what we’ve already discovered in the adb -> adb-server protocol. Standard commands cover general operations such as executing shell commands, dump device framebuffer.., while sync commands are related to file transfer and similar operations.

To run a command on device, client must send an initial OPEN command. This command initiates a new connection between the host and the device, allowing the desired operation to be carried out. The following figure illustrates the process in detail.

Push a file on device

Pushing a file to a device belongs to the sync command category. This operation uses a dedicated “sync” mode, initiated during the OPEN command by specifying the mode as “sync:”. Once the session is established, a sequence of WRTE commands is used to transfer the file’s content, adhering to a structured protocol that employs specific subcommands.

Rust implementation specifics

This section outlines some of the architectural decisions made during the development of the library, heavily influenced by the Rust programming language’s features and idioms.

Sharing common behaviors between “server” and “direct” devices

From a developer’s perspective, ADBServerDevice (for server-mediated connections),ADBUSBDevice (for direct USB connections) and ADBTCPDevice (for direct TCP connections) share many common behaviors. All three types of devices need to implement methods such as push(), pull(), execute(), among others. This functional overlap can be elegantly handled in Rust using traits.

A trait in Rust acts as a blueprint for shared functionality, allowing developers to define a set of required methods that device types must implement. By applying a trait, the compiler ensures that any future methods added to the trait will be consistently implemented across all relevant structures. This enforces consistency and reduces duplication in code, making it easier to extend and maintain.

/// Trait representing all features available on devices.
pub trait ADBDeviceExt {
    /// Run command in a shell on the device, and write its output and error streams into output.
    fn shell_command(&mut self, command: &[&str], output: &mut dyn Write) -> Result<()>;

    /// Pull the remote file pointed to by `source` and write its contents into `output`
    fn pull(&mut self, source: &dyn AsRef<str>, output: &mut dyn Write) -> Result<()>;

    /// Push `stream` to `path` on the device.
    fn push(&mut self, stream: &mut dyn Read, path: &dyn AsRef<str>) -> Result<()>;

    /// Reboot the device using given reboot type
    fn reboot(&mut self, reboot_type: RebootType) -> Result<()>;

    ...
}

Sharing common behaviors between USB and TCP “direct” devices

To share behavior between USB and TCP “direct” devices, we identified that the underlying ADB protocol is identical for both; the only difference lies in the transport method for sending packets (either USB or TCP). To handle this, we introduced the ADBMessageTransport trait, which abstracts away the transport-specific details for reading and writing message structures. This trait requires implementations to also implement the ADBTransport trait, which ensures basic connection management (with methods like connect() and disconnect()).

/// Trait representing a transport able to read and write messages.
pub trait ADBMessageTransport: ADBTransport + Clone + Send + 'static {
    /// Read a message using given timeout on the underlying transport
    fn read_message_with_timeout(&mut self, read_timeout: Duration) -> Result<ADBTransportMessage>;
    
    /// Write a message using given timeout on the underlying transport
    fn write_message_with_timeout(
        &mut self,
        message: ADBTransportMessage,
        write_timeout: Duration,
    ) -> Result<()>;
}

This abstraction allows us to create device structures that are agnostic to the transport:

// At this point, types look like:
struct ADBMessageDevice<T: ADBMessageTransport> {}
// Providing these concrete types:
ADBMessageDevice<USBTransport> /* For USB "direct" devices */
// and
ADBMessageDevice<TCPStream> /* For TCP "direct" devices */

Given that we need to store attributes depending on the concrete type, type aliases were not feasible. We had to consider three different options:

1. Trait-based solution: We could create and implement a new ADBMessageDevice trait that would handle all command implementations for both USB and TCP devices. Since all methods could have default implementations, it would result in unnecessary boilerplate. Moreover, trait would just store all concrete implementation, which we did not find elegant.
2. Using Deref and DerefMut: By creating separate structures for ADBUSBDevice and ADBTCPDevice, we could store the corresponding transport type (ADBMessageDevice<USBTransport> or ADBMessageDevice<TCPTransport>) as an attribute. While this works for methods that take &self, it doesn’t address methods that take ownership of self. This is currently not a limitation in current version of the API, but may be an issue for the future. Additionally, documentation also becomes unclear as the compiler automatically dereferences the type.
3. Wrapping with inner attributes: This solution stores the concrete ADBMessageDevice<USBTransport> or ADBMessageDevice<TCPTransport> in an inner attribute field, and we implement the necessary traits for each device structure. Although it requires some boilerplate code to redirect all trait calls to self.inner, it results in a more understandable and less generic public API. This approach was chosen for its clarity and simplicity.

Benchmarks

The final phase of this project focused on benchmarking the performance of our crate against the official ADB implementation. We used Rust’s cargo bench in combination with the criterion library to set up benchmark scenarios.

These benchmarks were run using given parameters:

• File size: To test performance differences between small and large files
• Sample size: To increase our benchmark results
• Devices: Samsung S10 SM-G973F (locked) and a laptop with an Intel i7-1265U CPU

We benchmarked file pushes with random data to assess the efficiency of the implementation.

The benchmark results indicate that the adb_client crate performs comparably to the official ADB command line implementation:

• For 10MB, 500MB, and 1GB file transfers, the average time difference between the crate and the official ADB is minimal (ranging from 1.57% to 0.10% faster).
• The performance gap for the small file arises from the need to spawn a new shell command when running every adb push. This overhead becomes negligible as the file size increases.
• The Rust overhead is imperceptible, suggesting that the custom implementation is highly efficient and almost on par with ADB’s performance, making it a viable alternative in most scenarios for pushing files to devices.

Conclusion

In conclusion, the development of the adb_client library and the adb_cli command line tool has proven effective as an alternative to the official adb tool, especially by incorporating direct USB protocol support. It reduces the need for additional tools and allows ADB features to be used as a library, thus improving performance. Future development could still bring more functionality to this library, such as interacting with Android’s Binder service (via abb protocol commands) and debugging Java programs (via jdwp commands), further enhancing the library’s capabilities.

Library source code can be found on github, and all PRs are welcome !

• 1. ADB message structure https://android.googlesource.com/platform/packages/modules/adb/+/refs/h…