BLaDE: Brave’s Performance Evaluation Testbed

This post describes work done by Kleomenis Katevas, Stefanos Laskaridis, Aravindh Raman, Mihai Plesa, and Hamed Haddadi. This post was written by Senior Machine Learning Researcher Kleomenis Katevas.

In today’s fast-paced market, continuous evaluation of product releases is essential. Companies must regularly assess their products to stay competitive and meet evolving customer expectations. This process also provides crucial insights when comparing a product with competitors. However, there are significant challenges.

Automating cross-device mobile app activities—including shipping new features, simulating Web browsing user activity across different browsers and evaluating on-device machine learning models—is challenging, primarily due to inherent operating system (OS) restrictions. Moreover, accurately profiling the power consumption of these activities is difficult because of unreliable OS estimates and often lower sampling rates (number of samples per second).

To address the above challenges, we introduced BLaDE (BatteryLab Device Evaluations) as a successor to BatteryLab ^{1 2} (originally released in 2019), which serves as Brave’s open-source solution for automated performance evaluations on mobile devices. Supporting both Android and iOS, BLaDE is capable of recording power consumption, CPU utilization, device temperature, and network traffic (bandwidth) in a highly precise and configurable frequency, while simulating user actions using different types of automation methods. By open sourcing the system design and code base, any developer or researcher can replicate this setup to conduct comprehensive performance evaluations on their own mobile devices.

BLaDE System Design

Our system is designed with the following key hardware components:

• Main controller (Raspberry Pi 4, 8GB): The central device coordinator (internally known as BLaDE Runner) that’s responsible for managing and orchestrating all system activities.

• Mac controller (Mac Mini, M2, 8-core): A Mac-based machine to support iOS-based automation through Appium, and to build iOS apps through Xcode.

• Power monitor (Monsoon HV): A critical component for measuring the power consumption of the connected devices at a maximum frequency of 5KHz.

• Programmable AC switch (Keene KPS1): Utilized to control the power supply to the power monitor, enabling automated power cycling to the device.

• Programmable relay board (Yizhet 5V, 8 Ch.): Utilized for controlling the power distribution from the power monitor to the devices. This is to programmatically select which device is powered on.

• Programmable USB hub (YKUSH 3): These hubs allow selective powering on/off of USB connections, providing isolated power control for each port.

• IR-thermometer (MLX90614): Deployed for continuous monitoring of device temperature.

• LCD display (Waveshare LCD1602): A notification unit that reports the system state (device being monitored, real-time power consumption, etc).

• Mobile devices: A variety of iOS (iPhone 14 Pro, SE 2022) and Android (Galaxy S23, Pixel 6a) devices, configured to be compatible with our BLaDE infrastructure.

• Surveillance camera (Anker PowerConf C200): Employed to remotely monitor the status of evaluations. Although remote screencasting options are available via the OS, they are avoided as they could interfere with performance metrics.

• Router (UniFi Dream Router): Connected to a 70MBit dedicated VLAN network, it provides Ethernet network access to the controller and Mac Mini, and WiFi 6 network access to all mobile devices. Additionally, it is capable of forwarding Internet traffic through a VPN service, if required, allowing simulation of Internet access from different geolocations.

Device configuration

To facilitate accurate power measurements, we stepped away from the available software-based options and utilized a battery bypass technique. This involves disassembling each device to remove its battery, extracting the internal battery controller, and exposing the power terminals via cables. This method allows us to monitor the devices’ power consumption directly from their power terminals, achieving precise measurements at a maximum frequency of 5 kHz using the power monitor.

Further configurations are made in the device’s OS to minimize the influence of extraneous factors. These include disabling automated OS and app updates, turning off adaptive brightness/charging/battery features, enabling dark mode, and standardizing the brightness level to 25% across devices.

Device testing automation

Automating testing on mobile devices, especially in this custom setup, can be a complex task. BLaDE provides a comprehensive set of APIs that facilitate various aspects of device management, such as reading device status, controlling power states, and managing measurement processes.

Consider the following example of the “switch on” API call for switching a mobile device:

1. Initialize and connect: Begin by switching on the power monitor. BLaDE waits until the monitor is available and establishes a connection.

2. Enable USB power: BLaDE then enables the USB power to the device, ensuring it is ready for further operations.

3. Relay activation: The appropriate relay on the relay board is activated to manage the power flow.

4. Voltage setting: Set the required voltage for the device to switch on.

5. Device synchronization: BLaDE waits for the device to become available and synchronizes its clock with the controller’s clock. This ensures time measurements are accurate.

6. Unlocking the device: For iOS devices, BLaDE connects using the Bluetooth HID service and enters the PIN to unlock the device. This step is required to allow the device to communicate via USB after boot.

When measuring the power performance of a phone, it is crucial to disable the USB power lanes. This step ensures that the USB current does not interfere with the power monitor readings. All communication with the device must be wireless, using either Wi-Fi or Bluetooth, before or after the measurement has happened.

Currently, BLaDE supports automation via Android Debug Bridge (ADB), Bluetooth HID, and Appium-based frameworks:

Android Debug Bridge (ADB) automations (Android only)

The Android Debug Bridge (ADB) is a versatile command-line tool that allows developers to communicate with an Android device. It supports a wide range of device operations, such as installing and debugging applications, transferring files, and interacting with the device’s screen and keyboard. ADB commands can be executed via USB, Wi-Fi, or Bluetooth. BLaDE builds on top of this tool to support actions that are required for device automations, such as “type text”, “tap”, “scroll down”, “close foreground app”, etc.

Bluetooth HID automations (iOS, Android)

Automating third-party apps on iOS presents challenges due to the absence of a publicly available ADB-like API that operates over Wi-Fi or Bluetooth. To address this, we have virtualized a Bluetooth mouse and keyboard by creating a virtual Human Interface Device (HID) service at the controller. We also created an automation library that translates keyboard keystrokes, mouse clicks, and gestures into HID actions, effectively simulating user interactions with the controlled device. This approach allows for the automation of tasks—such as locating and launching apps and interacting with them—by mimicking user actions. 

Appium-based automations (iOS, Android)

As an alternative approach, we experimentally support an Appium-based framework to automate actions, minimizing the overhead of the above automation methods when aiming multiple platforms. Appium uses ADB for Android and Apple’s XCTest for iOS automations. Although Appium is generally used for testing features of custom application builds, we currently experiment with the framework to predominantly measure the performance of Android and iOS apps.

BLaDE for product evaluations

The use of BLaDE is crucial to our product pipeline, ensuring the performance, robustness and reliability of various features, including on-device performance of LLMs, cost efficiency of modules, and browsing capabilities. When a new development build is ready and a specific feature needs to be tested, it is dispatched on-demand to BLaDE for an in-depth performance assessment across devices. This process helps identify new regressions and bottlenecks, enabling us to address issues swiftly and maintain the high standards our users expect. With BLaDE’s continuous monitoring, we can confidently release updates, assured that any potential performance issues have been proactively managed.

For more details about how BLaDE is used at Brave, please refer to the following blog posts:

• MELTing Point: Mobile Evaluation of Language Transformers (July, 2024) ³

• Brave 1.0 Performance: Methodology and Results (January, 2020)

• Significant Battery Savings with Brave on Mobile: Brave Consumes 40% Less Battery than Other Leading Browsers (February, 2019)

The complete source code and system design of BLaDE are available at https://github.com/brave/blade. We welcome contributors willing to extend the system capabilities.

References