iOS: a journey in the USB networking stack
In this article, we give a small journey inside the implementation of networking interfaces exposed by iOS when connected via USB. These are used for sharing a computer's connection with iPhone (tethering), sharing an iPhone's connection with a computer (reverse tethering) and since the latest versions of iOS, USB networking even carries RemoteXPC packets which seems to be the future common ground for all Apple based cross-device communications.
There is an ongoing hot topic in the Synacktiv development team between a handful of iOS developers that are willing to work on a full macOS environment and people that'd rather spend an insane amount of time to make it work on their Linux development setup.
This Linux-extremist gang was recently interested in making reverse tethering on iOS devices work on Linux hosts. This is particularly useful to avoid creating Wi-Fi networks to connect iPhones to a controlled network.
This process somehow led us to explore the USB networking devices exposed by iOS in recent versions.
Very few literature exists on that matter, hence we'll try to aggregate the knowledge we gathered in this article.
A foreword on tethering
Tethering is the process of sharing your phone's internet connection with a computer. The idea is that the phone acts as a router, routing packets sent by the computer to whichever air interface is connected. As a user, it's basically like doing a Wi-Fi hotspot but over USB.
On iOS, this feature has been around for a very long time: it was introduced in iOS 3.
At that time, Apple developed a very thin protocol layer over USB to encapsulate Ethernet frames (the associated driver in iOS is named AppleUSBEthernetDevice). The protocol consists in encapsulating a single Ethernet frame in each USB request block and sending them over two dedicated USB bulk endpoints: one for each direction (RX/TX).
$ lsusb -d 05ac:12a8 -vvv
[...]
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 2
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 253
bInterfaceProtocol 1
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x86 EP 6 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x05 EP 5 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 0
[...]
This protocol has a dedicated Linux kernel driver, mainlined many years ago, called ipheth. This driver has seen very minimal changes over the years1 and thus the tethering protocol is compatible very broadly with all iOS versions even the recent ones.
The world of reverse tethering
Reverse tethering is the process of sharing your computer's internet connection with the phone. The idea being that the computer receives packets from the phone and routes them to the internet. On macOS system its only a matter of ticking a box in the network preferences.
While tethering and reverse tethering could have been implemented with a single protocol, it would have implied some modifications to allow for exchanging the traffic direction and gateway. As there is no code in ipheth implying a possibility to reverse the packet flow, we decided to take a look at USB exchanges while using the reverse tethering feature on a macOS system.
We quickly noticed that it uses the CDC-NCM protocol (Communications Device Class - Network Control Model, one of the few protocols used by the industry to perform Ethernet over USB2) on a USB interface that we did not see on Linux.
As you can see in the two screenshots below, there is one additional configuration advertised by iPhone when hooked to a macOS host compared to when hooked to a Linux machine. This is rather unusual.
A quick detour on iOS USB internals
This weird behavior made us wonder how these USB interfaces are exposed and which were the kernel space and user space actors behind the USB networking functionalities, so that we could ultimately figure out how to reproduce the macOS behavior.
While searching keywords linked to our objective, we figured that USB devices configurations used by iOS are listed in a file stored at /System/Library/AppleUSBDevice/USBDeviceConfiguration.plist holding a configurations dictionary containing things like:
<key>stdMuxPTPEthValIDA</key>
<array>
[...]
<dict>
<key>Description</key>
<string>PTP + Apple Mobile Device</string>
<key>Interfaces</key>
<array>
<string>PTP</string>
<string>AppleUSBMux</string>
</array>
<key>DefaultConfiguration</key>
<true/>
</dict>
<dict>
<key>Interfaces</key>
<array>
<string>PTP</string>
<string>AppleUSBMux</string>
<string>AppleUSBEthernet</string>
</array>
<key>Description</key>
<string>PTP + Apple Mobile Device + Apple USB Ethernet</string>
</dict>
<dict>
<key>Description</key>
<string>PTP + Apple Mobile Device + NCM</string>
<key>Interfaces</key>
<array>
<string>PTP</string>
<string>AppleUSBMux</string>
<string>AppleUSBNCMControl</string>
<string>AppleUSBNCMData</string>
<string>AppleUSBNCMControlAux</string>
<string>AppleUSBNCMDataAux</string>
</array>
<key>ExtendedConfiguration</key>
<true/>
</dict>
<dict>
<key>Description</key>
<string>PTP + Apple Mobile Device + Valeria</string>
<key>Interfaces</key>
<array>
<string>PTP</string>
<string>AppleUSBMux</string>
<string>Valeria</string>
<string>AppleUSBNCMControlAux</string>
<string>AppleUSBNCMDataAux</string>
</array>
<key>ExtendedConfiguration</key>
<true/>
</dict>
[...]
<dict>
<key>Description</key>
<string>PTP + Apple Mobile Device + Apple USB Ethernet + NCM</string>
<key>Interfaces</key>
<array>
<string>PTP</string>
<string>AppleUSBMux</string>
<string>AppleUSBEthernet</string>
<string>AppleUSBNCMControl</string>
<string>AppleUSBNCMData</string>
<string>AppleUSBNCMControlAux</string>
<string>AppleUSBNCMDataAux</string>
</array>
<key>ExtendedConfiguration</key>
<true/>
</dict>
</array>
In fact, a lot of configurations are available including Carplay related ones, UVC Camera, audio, as well as some testing configurations.
This file is used by a very central user space daemon called configd. This daemon is in charge of keeping state for a lot of network related tasks (link/IP/PPP state for example) as well as creating interfaces such as BSD's. It has a modular structure composed of binary bundles3 that gets loaded and executed.
Among these executables, a very small bundle named USBDeviceConfiguration.bundle is in charge of loading and parsing the USBDeviceConfiguration.plist file (via the IOKit API IOUSBDeviceDescriptionCreateFromDefaultsAndController). The specific configuration which will be selected will by default is based on the configuration-string property hardcoded in the iOS device-tree.
In current versions of iOS, the default configuration loaded is named stdMuxPTPEthValIDA.
This selected configuration is forwarded by the userland to the kernel via the IOUSBDeviceControllerSendCommand API. Ultimately, this configuration is used by the kernel initialization function IOUSBDeviceController::createUSBDevice which will match each interface of the selected configuration (e.g. AppleUSBMux, AppleUSBNCMControl, shown in the property list above) to an (IOKit) driver that will provide the functionality.
Back to these CDC-NCM interfaces
If you look carefully at the default loaded configuration specified in USBDeviceConfiguration.plist (in the previous part), you can see that there is some mention of CDC-NCM interfaces. These are located inside dictionaries containing a special ExtendedConfiguration property.
After some research, it turns out that these Extended Configurations can be switched to and from using a proprietary USB control transfer packet with bRequest = 0x52. Indeed, this is confirmed by a USB capture showing that it is performed by macOS to enable the CDC-NCM interface. This control transfer is called kVendorRequestSelectExtendedFunction inside the iOS kernel. The Extended Configurations that are allowed to be switched to are hardcoded in the kernel and are limited to the possibilities below.
This proprietary modeswitch4 was quite recently implemented in usbmuxd5 by exposing an environment variable named USBMUXD_DEFAULT_DEVICE_MODE allowing to switch between the different configurations on Linux.
| Values | Description |
|---|---|
| 1 | Switch to mode with the initial descriptors (with UsbMux descriptors only) |
| 2 | Switch to mode with Valeria descriptors (proprietary protocol used by QuickTime to gather a video/audio stream of the device) |
| 3 | Switch to mode with CDC-NCM descriptors (used for internet sharing) |
Unfortunately, this means that some configurations are not made to coexist: e.g. it seems you can't stream your device screen with reverse tethering at the same time.
After relaunching usbmuxd with the environment variable USBMUXD_DEFAULT_DEVICE_MODE=3, a new USB interface descriptor appears and Linux recognizes it properly as a generic CDC-NCM device as shown in the lsusb dump below. This is now the default mode that is requested by usbmuxd in the latest release.
$ lsusb -d 05ac:12a8 -vvv
[...]
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 2
[...]
iInterface 16 NCM Control
CDC Union:
bMasterInterface 2
bSlaveInterface 3
CDC Header:
bcdCDC 1.10
CDC Ethernet:
iMacAddress 15 XXXXXXXXXXXX
bmEthernetStatistics 0x00000000
wMaxSegmentSize 1514
wNumberMCFilters 0x0000
bNumberPowerFilters 0
CDC NCM:
bcdNcmVersion 1.00
bmNetworkCapabilities 0x3b
8-byte ntb input size
crc mode
max datagram size
net address
packet filter
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x86 EP 6 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0010 1x 16 bytes
bInterval 11
[...]
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 3
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 10 CDC Data
bInterfaceSubClass 0
bInterfaceProtocol 1
iInterface 21 NCM Data
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x87 EP 7 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x05 EP 5 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
[173575.291851] usb 3-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[173575.291857] usb 3-4.1: Product: iPhone
[173575.291862] usb 3-4.1: Manufacturer: Apple Inc.
[173575.291866] usb 3-4.1: SerialNumber: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[173575.407645] cdc_ncm 3-4.1:5.2: MAC-Address: xx:xx:xx:xx:xx:xx
[173575.407868] cdc_ncm 3-4.1:5.2 usb0: register 'cdc_ncm' at usb-0000:08:01.0-4.1, CDC NCM (NO ZLP), xx:xx:xx:xx:xx:xx
[173575.418259] cdc_ncm 3-4.1:5.2 enxxxxxxxxxxxxx: renamed from usb0
On iOS, the drivers associated to this CDC-NCM interface are AppleUSBDeviceNCMControl and AppleUSBDeviceNCMData.
After that, it's only a matter of bringing up this interface, giving it an address, making a DHCP server listen and iPhone will promptly send a DHCP request and start throwing packets at your Linux box.
sudo ip addr add dev enxxxxxxxxxxxxx 192.168.1.1/24
sudo ip link set dev enxxxxxxxxxxxxx up
sudo dnsmasq -i enxxxxxxxxxxxxx -K -F 192.168.1.2,192.168.1.3
So we achieved our objective of making iOS reverse tethering work on Linux, right ?
iOS 16 broke my workflow
Turns out that things are not so easy. Starting with iPhones running iOS 16, after switching modes with usbmuxd, we see not one but two CDC-NCM interfaces6.
Our first assumption was that Apple began to unify their tethering and reverse tethering networking stacks by using CDC-NCM for both. This later turned out to be completely false.
Furthermore, depending on the specific version of iOS and the SoC, one or both interfaces made the Linux kernel unhappy in these terms:
[93636.679508] cdc_ncm 3-2:5.2: setting rx_max = 16384
[93636.679961] cdc_ncm 3-2:5.2: setting tx_max = 16384
[93636.681232] cdc_ncm 3-2:5.2 usb0: register 'cdc_ncm' at usb-0000:08:01.0-2, CDC NCM (NO ZLP), xxxxxxxxxxxxxxxxx
[93636.712089] cdc_ncm 3-2:5.4: bind() failure
[93636.714769] cdc_ncm 3-2:5.2 enxxxxxxxxxxxxx: renamed from usb0
In this case (iOS 16), one interface (5.2) was successfully recognized but the second (5.4) made the kernel error out bind() failure.
This error message happens because the interrupt endpoint used in CDC-NCM to pass link status messages is missing. This endpoint, although mandatory in the spec, does not do much, so it's easy to work around the problem by patching the driver7 in Linux to comment out all occurrences of the interrupt endpoint (dev->status).
Fortunately, I won't share this horrible solution with you because someone8 wrote a much cleaner (and smaller) patch for this.
However, after fixing the kernel driver, reverse-tethering only works on one of the two exposed endpoints and this other one does not respond to anything.
Watching the IORegistry on an iPhone running iOS 16.7.2 gives some insight of what's happening:
| | +-o usb-complex@30000000 <class AppleARMIODevice, id 0x100000178, registered, matched, active, busy 0 (330 ms), retain 11>
| | | +-o AppleT8011USBArbitrator <class AppleT8011USBArbitrator, id 0x100000245, registered, matched, active, busy 0 (165 ms), retain 12>
| | | +-o usb-device@100000 <class AppleEmbeddedUSBNub, id 0x100000179, registered, matched, active, busy 0 (88 ms), retain 13>
| | | | {
[...]
| | | | "product-string" = <"iPhone">
| | | | "ncm-interrupt-ep-disabled" = <01000000>
| | | | "device-mac-address" = <XXXXXXXXXXX>
[...]
| | | +-o AppleUSBNCMControl@6 <class IOUSBDeviceInterface, id 0x100000690, registered, matched, active, busy 0 (38 ms), retain 11>
| | | | +-o AppleUSBDeviceNCMControl@6 <class AppleUSBDeviceNCMControl, id 0x1000006a3, registered, matched, active, busy 0 (9 ms), retain 7>
| | | | {
| | | | "CFBundleIdentifierKernel" = "com.apple.driver.AppleUSBDeviceNCM"
| | | | }
| | | |
| | | +-o AppleUSBNCMData@7 <class IOUSBDeviceInterface, id 0x100000691, registered, matched, active, busy 0 (44 ms), retain 11>
| | | | +-o AppleUSBDeviceNCMData <class AppleUSBDeviceNCMData, id 0x1000006aa, registered, matched, active, busy 0 (5 ms), retain 8>
| | | | +-o en2 <class IOEthernetInterface, id 0x1000006b0, registered, matched, active, busy 0 (4 ms), retain 10>
| | | | | {
| | | | | "BSD Name" = "en2
| | | | | }
| | | | |
| | | | +-o IONetworkStack <class IONetworkStack, id 0x100000393, registered, matched, active, busy 0 (0 ms), retain 14>
| | | | +-o IONetworkStackUserClient <class IONetworkStackUserClient, id 0x10000057d, !registered, !matched, active, busy 0, retain 5>
| | | | {
| | | | "IOUserClientCreator" = "pid 43, configd"
| | | | "IOUserClientDefaultLocking" = Yes
| | | | }
| | | |
| | | +-o AppleUSBNCMControlAux@8 <class IOUSBDeviceInterface, id 0x100000692, registered, matched, active, busy 0 (37 ms), retain 13>
| | | | +-o AppleUSBDeviceNCMControl@8 <class AppleUSBDeviceNCMControl, id 0x1000006a1, registered, matched, active, busy 0 (7 ms), retain 7>
| | | | {
| | | | "ncm-control-use-aux" = <01000000>
| | | | "CFBundleIdentifierKernel" = "com.apple.driver.AppleUSBDeviceNCM"
| | | | }
| | | |
| | | +-o AppleUSBNCMDataAux@9 <class IOUSBDeviceInterface, id 0x100000693, registered, matched, active, busy 0 (43 ms), retain 13>
| | | | +-o AppleUSBDeviceNCMData <class AppleUSBDeviceNCMData, id 0x10000069e, registered, matched, active, busy 0 (5 ms), retain 8>
| | | | | {
| | | | | "IOClass" = "AppleUSBDeviceNCMData"
| | | | | "IOActiveMedium" = "00100026"
| | | | | "waitControlStart" = 45591
| | | | | "IOMatchedAtBoot" = Yes
| | | | | "IOMinPacketSize" = 64
| | | | | "IOProviderClass" = "IOUSBDeviceInterface"
| | | | | "IOLinkStatus" = 1
| | | | | "IOProbeScore" = 0
| | | | | "HiddenInterface" = Yes
| | | | | "HostAttached" = 0
| | | | | "CFBundleIdentifierKernel" = "com.apple.driver.AppleUSBDeviceNCM"
| | | | | "IOMACAddress" = <XXXXXXXXXXX>
| | | | | "CFBundleIdentifier" = "com.apple.driver.AppleUSBDeviceNCM"
| | | | | }
| | | | |
| | | | +-o anpi0 <class AppleUSBDeviceNCMPrivateEthernetInterface, id 0x1000006ae, registered, matched, active, busy 0 (4 ms), retain 10>
| | | | +-o IONetworkStack <class IONetworkStack, id 0x100000393, registered, matched, active, busy 0 (0 ms), retain 14>
| | | | +-o IONetworkStackUserClient <class IONetworkStackUserClient, id 0x10000057d, !registered, !matched, active, busy 0, retain 5>
| | | | {
| | | | "IOUserClientCreator" = "pid 43, configd"
| | | | "IOUserClientDefaultLocking" = Yes
| | | | }
| | | |
First of all, by watching the usb-device node inside this output, we can confirm that the absence of the interrupt descriptor is intentional because of the existence of a key named ncm-interrupt-ep-disabled. This key originates from the device-tree and is not consistent across devices and versions of iOS.
We also notice that two interfaces are bound to this usb-device: the first one is a standard Ethernet interface (BSD interface prefixed en) that can be used for reverse tethering. The second one however is handled by a new driver AppleUSBDeviceNCMPrivateEthernetInterface (BSD interface prefixed anpi). It is also designated by the Aux suffix (and the ncm-control-use-aux property).
This driver has been in the kernel cache at least since 15.0.1, but doesn't seem to have really been used until iOS 17, even though it was already loaded (with HiddenInterface=1) on iOS 16.
Until iOS 17, there were very few explanations on what was the purpose of this endpoint and driver.
iOS 17 broke my mind
This endpoint is a transport medium for the RemoteXPC protocol which is (notably) used by XCode since iOS 17.
It turns out that, with iOS 17, Apple introduced a new way of communicating with a device using this special CDC-NCM interface. The RemoteXPC protocol allows establishing an encrypted tunnel between the phone and a (trusted) host transporting all host-phone exchanges including to services that, until then, used to be handled exclusively via the usbmux protocol and lockdownd.
Apple pushed a lot of recent protocols in this new communication stack: the RemoteXPC protocol itself is based on HTTP/2, all communications are happening over IPv6 and the tunnel is based on QUIC.
If you are looking into that topic, I suggest you have a look at the reverse-engineering efforts by the Duo Team9 (made as part of a research on the T2 coprocessor), and the documentation efforts by people from pymobiledevice10.
It turns out that the only obstacle preventing Linux users from using the RemoteXPC interfaces is the CDC-NCM Linux driver not supporting the missing interrupt endpoint. This problem is known since 202311, but the proposed patch hasn't been mainlined in the kernel, yet.
Conclusion
In this article we summarized the knowledge we gathered on USB network interfaces exposed by iOS while doing research to make reverse tethering work on Linux.
Coincidentally, we stumbled upon the new RemoteXPC-based communication stack relying on more standard and modern transport protocols over an (almost) standard tethering interface. This new stack will presumably, at some point, replace the usbmux protocol.
Those changes will bring some integration challenges to the Linux community: not only does the CDC-NCM driver modifications need to be mainlined at some point, but the question of the integration of the USB modeswitching and new pairing/tunneling protocol introduced by iOS 17 will need to be addressed either in libimobiledevice, pymobiledevice or somewhere.
- 1. https://github.com/torvalds/linux/commits/master/drivers/net/usb/ipheth…
- 2. https://en.wikipedia.org/wiki/Ethernet_over_USB
- 3. https://real-world-systems.com/docs/configd.1.html
- 4. Intended reference to the usb_modeswitch era that existed on some WAN USB sticks. Apple kind of recycled their techniques.
- 5. https://github.com/libimobiledevice/usbmuxd/issues/205
- 6. This is seemingly not the case for iPads which still have a unique interface.
- 7. https://github.com/torvalds/linux/blob/master/drivers/net/usb/cdc_ncm.c…
- 8. https://marc.info/?l=linux-usb&m=170138175510294&w=4
- 9. https://duo.com/labs/research/apple-t2-xpc
- 10. https://github.com/doronz88/pymobiledevice3/blob/master/misc/RemoteXPC…
- 11. https://github.com/doronz88/pymobiledevice3/issues/566