Jetpack Benchmark : Jetpack Component which help you Benchmark your Android App

Table of Contents

Introduction
— What is benchmark?
— Why do we need to benchmark?

Benchmarking Kotlin’s code
— measureTimeMillis

Jetpack Benchmark
— Quick Start
— Project Setup
— Examples Benchmark

Additional Configuration
— Enable Json Output
— Clock Stability
— Suppress Errors

What is Benchmark?

If you google the keyword “What is Benchmark?” google translate will answer this to you

Evaluate or check (something) by comparison with a standard.

Why do we need to benchmark our code?

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
- Donald Knuth

Donald Knuth said that speed efficiency have strong negative impact to debugging and maintainability and most of the time it is not needed.

So I prefer to alias maintainability vs performance as
UX (User Xperience) vs (Developer Xperience)

Good UX is where your app is optimised whereas Good DX means your app is maintainable

Let’s have an example, we want to see which is faster between these 3 loop approach in kotlin

So we can see that forEach took 1,720,664ns which is ~20% slower than forSize which took 1,421,716ns

But from the developer side forEach should be more readable than forSize, is it worth to sacrifice readability for that performance efficiency?

Let’s do another example

For this case, we can see kotlin’s inlined function is ~760 times faster than nonInlined function

By looking at those numbers, I think you must be able to decide which optimisation should be done, and which shouldn’t be.

so let me question this again. Why do we need to benchmark?

To determine whether the result of optimisation is worth the cost of maintainability

Benchmarking Kotlin’s Code

To benchmark kotlin’s code you can use measureTimeMillis or MeasureNanoTime. To know more about this you can read this article here

Jetpack Benchmark

Benchmark | Android Developers

Most of the time when we want to benchmark some approach on our Android App, we will put some logger or tracer in our production code. But in reality the user doesn’t need that code right?

So how do we handle warmup and measure repeated benchmark on Android without changing the implementation code?

Let me introduce you to Jetpack benchmark

• Jetpack Benchmark is a Jetpack Library that measures the performance of Android app code.
• It helps reduce measurement errors that are easy to do in general, and is integrated into Android Studio.
• It is standard JUnit instrumentation tests that run on an Android device, and use a rule provided by the library to perform the measuring and reporting.

and it has stable release!

Jetpack Benchmark Version 1.0.0

Quick Start

If you open up https://developer.android.com/studio/profile/benchmark there will be a quick start if you want to try benchmarking your app without moving it into modules.

Add the library to your module’s build.gradle file:
project_root/module_dir/build.gradle

set jvmTarget to 1.8 in your module’s build.gradle file:
project_root/module_dir/build.gradle

Then you need to force-disable debugging temporary by setting android:debuggable="false" in your AndroidManifest.xml

To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Write a benchmark.

The following code snippet shows how to add a benchmark to a JUnit test:

BUT Please note this!

Project Setup

Setup jetpack-benchmark on your project:

• Create new module for benchmark
  We need to force the application debuggable to false, by creating a new module we can isolate the custom configuration just to be used on benchmark module
• Separate feature module from app
  We need our benchmark module to depend on the feature we want to benchmark. And it cannot depend on app/base module.

I assume we already have this kind of structure where all the features are not on app module anymore

Let’s start by creating a new benchmark module. This module template is already available on Android Studio 3.6

Now we have 3 modules

*If you want to run benchmark on emulator
Open up your :benchmark module build.gradle and put this code to supress error when doing benchmark on emulator

So let’s say we want to benchmark feature_example, so we need it as a dependency on our benchmark module

Benchmark Example

Let’s say we want to benchmark the time needed to inflate each of this 3 xml layout

and we have MainActivity on feature_example

We add the benchmark into androidTest in benchmark module like when doing Instrumentation Test

To measure the code, you need to create a BenchmarkRule() and then call the measureRepeated { /*your code goes here*/ }

Then you can run the test as how you run instrumented test

The result will be shown at test output

Notice there’re warning because you run the tests on emulator. Google suggest us to run it on real device to have better accuracy

The benchmark result can be seen here

For another example such as

• Linear vs Constraint on complex nested layout
• RecyclerView vs ListView

you can try to look at this repo

Additional Configurations

Enable Json Output

on your benchmark module’s build.gradle
add androidx.benchmark plugin

androidx.benchmark plugin on build.gradle

on your gradle.properties
add android.enableAdditionalTestOutput=true so we can run the test from terminal to have .json output as a report for the benchmark

when you run

./gradlew moduleName:connectedCheck

for example on our module it will be

./gradlew benchmark:connectedCheck

the results will be generated at

$modulePath/build/outputs/connected_android_test_additional_output

This is the example result for running renderConstraintLayout

Clock Stability

Clocks on mobile devices dynamically change from high state (for performance) to low state (to save power, or when the device gets hot). These varying clocks can make your benchmark numbers vary widely, so the library provides ways to deal with this issue.

Configuration Errors

The library detects the following conditions to ensure your project and environment are set up for release-accurate performance:

• Debuggable is set to false.
• A physical device, not an emulator, is being used.
• Clocks are locked if the device is rooted.
• Sufficient battery level on device.

If any of the above checks fail, the benchmark will throw an error to discourage inaccurate measurements.

Here are some additional learning resources

Benchmark | Android Developers

Benchmark app code | Android Developers

Run benchmarks in Continuous Integration | Android Developers

Improving Android app performance with Benchmarking