We collect data for the following attributes (to be extended):
- Average patch frequency [days]: The average duration between receiving security updates is an indication for how long a device may run with an open security issue in standard cases. Note that often the release of security updates is coordinated with the public release of details on discovered security issues, but that this measure does not correlate the public disclosure of a vulnerability with the release of the related security fix.*
- Guaranteed maximum patch delay [days]: The maximum duration between receiving security updates promised by the device manufacturer through device update policies published at (or after) product release is an indication for how long a device may run with open security issues in the worst case.
- Samsung publishes the list of devices with defined updated schedules distinguished as “monthly” (we assume the maximum of 31 days for comparability), “quarterly” (we assume the maximum of 92 days), and “other”.
- Guaranteed patch availability [years]: The minimum duration for which security updates are made available after initial product launch promised by the device manufacturer is an indication for how long a device can be used securely after it has been bought. Note that most manufacturers guarantee availability of security patches for a duration starting with the initial launch of the product, and not necessarily from the date it has been purchased.
- Samsung publishes the list of devices with a 3-year update guarantee.
- Google publishes the list of devices with security update guarantees.
- Fairphone publishes quarterly security updates with a 5 year software support promise.
- Published end of security patches [date]: The date at which security patches end for a specific device as published by the device manufacturer can be used for planning appropriate actions after that device is no longer supported. Note that this date is typically equal to or after the date of initial product launch + guaranteed patch availability (which is normally a generic guarantee, while this metric may be published for specific devices).
- Latest security patch level [date]: The date of the latest security patch level seen in the test lab (or the field) for a specific device. This indicates if the device has, in fact, recently received a security update. If this date is too far in the past, the device may be at risk.
- Latest Android release [API number]: The version number of the latest Android release. Newer Android releases generally add security mitigations and are typically preferrable for better security. orthogonally of the latest security patch level.
- Multi-user support (MUS) [boolean]: True if a device supports switching between multiple Android users. This can be used to separate data of multiple people using the same shared (e.g. family of organizational) device or to separate between different tasks/contexts of a single user, and is therefore beneficial for securing data.
- Seamless updates (SUS) [boolean]: True if system updates can be downloaded and applied in the background, to be activated automatically upon next reboot. This decreases the user-visible time to apply an update during which a device is non-functional, and therefore increases the probaility that users will quickly apply available system updates. While it does not make a difference to users who always apply updates explicitly, on average and over a large population, this typically increases apply rate of updates.
- Device encryption type [“file” or “block”]: Indicates if the device uses file based encryption (FBE) or block-based full disk encryption (FDE). File based encryption has the advantage of using separate encryption keys for different Android users or profiles (such as a work profile) and can support “device encrypted” files that are available after a reboot but before the lockscreen factor has been entered.
- Strongbox support (SBA) [boolean]: True if the device support Keymaster Strongbox to hold cryptographic keys in dedicated secure, tamper resistant hardware (such as a secure element chip). This increases security of cryptographic key material stored within Strongbox by reducing attack surface of the Keymaster implementation.
- Secure bootloader re-locking [boolean]: True if a device supports securely unlocking and re-locking its bootloader. While this does not increase security of the out-of-the-box system, it supports special use cases with custom system images used in a secure way, e.g. in large organizations, highly exposed users (journalists, dissidents, etc.), or to use alternative Android derivates (e.g. LineageOS) after the OEM security support period ends.
- Preloaded apps with system privileges [count]: The number of apps (APK files) pre-loaded in the system image signed with the platform signing key and therefore with privileges/permissions on the level of the main Android system. Generally, lower numbers indicate less attack surface.
- Preloaded apps with pre-granted privileged permissions [count]: The number of non-platform-signed apps (APK files) in the system image that have pre-granted permissions considered privileged (above what would be normally expected for user-installed apps). This is orthogonal to the count above, as it explicitly excludes apps signed with the platform signing key (which have the highest level of privileges possible for apps). Generally, lower numbers indicate less attack surface.
When possible, we try to map those attributes to related definitions such as those in ETSI TS 103 645 or by the Internet of Secure Things Alliance (both of which are focused on IoT devices, but the the definitions can apply to mobile phone security as well).
The average patch frequency and guaranteed patch availability / published end of security patches can be seen as specific metrics e.g. for ETSI TS 103 645 Provision 4.3-3 and 4.3-4, respectively. Future collected attributes may also map to other such standard provisions. However, this project also aims to collect and analyze metrics that do not yet map to a specific security target, but support explorative analysis of device security postures.