Comparing the Stock Android ROM with Rooted Custom Android ROMs

Jayanthi Balasubramanian Jay Inamdar Taeyeon Ki Vinay Menon Department of Computer Science and Engineering, SUNY at Buffalo, Buffalo, NY 14260 {jayanthi,jayinamd,tki,vinaymen}@buffalo.edu

Abstract
Rooted Custom Android ROMs are available in the Android community and come in many avors suited for various purposes. These Custom ROMs are used widely and are preferred over the Stock Android ROM, mainly due to their observed performance gains over the Stock ROM. However, it is unclear as to where exactly these performance gains emerge from. Is it just CPU overclocking that is responsible for the observed gains? If not, are the performance gains trivial or signicant? Do they arise from the core system or is this not the case? This paper outlines a study undertaken which includes careful comparison and performance evaluation of the Stock Android ROM with CyanogenMod which is one of the most prominent Custom Android ROMs. We evaluate CyanogenMod against the Stock ROM to gain an insight into the core systems of both and target specic areas where observed performance gains are signicant. Further, we try to delve deeper into such areas and perform targeted experiments which would enable us to proceed and suggest design changes to the Stock Android ROM in order to improve it further.

rooted ROMs like CyanogenMod, users have reported tremendous performance boosts mainly due to tweaks like CPU overclocking. Handy applications: Rooting the phone unleashes access to a pile of very useful applications that otherwise need superuser access. Some examples are SetCPU [11].(Overclocking, Underclocking), ROM Manager [10] (Managing and ashing ROMs), ConnectBot [5] (SSH Client), Titanium Backup [14] (A backup utility), Wireless Tether [15] (Using your device as a Wireless and Bluetooth hotspot), and many more. Numerous other features: A lot of features like the ability to delete stock apps, alter system les (adding custom binaries etc.), change boot images etc. The observed performance improvements in Custom Android ROMs encourage us to compare their performance with the Stock Android ROM in order to target specic areas that need further attention. We use Google Nexus S phones for the purpose of our experiments . We select the CyanogenMod Custom ROM [6] because it is the most popular ROM which supports Nexus S phones. In order to do performance evaluation, we used four benchmarking applications that are available for Android and which test the Android System based on certain predetermined parameters. Initially, we started off by evaluating four congurations of the ROMs and compared benchmark parameter values for each conguration pertaining to all the benchmark applications used. The results of our initial benchmark tests revealed IO to be an interesting area to look into as it showed significant performance improvement in case of CyanogenmMod. The CyanogenMod system uses Compcache [4], following an approach similar to [16]. We were interested in comparing Android System performance when either Compcache or Swap were used. This was in pur-

Introduction

Rooting is the process of modifying the operating system that is shipped with the Android [1] device in order to grant the user complete control over it. Rooting the phone extends system functionality and helps in overcoming limitations imposed on users phones by carriers and manufacturers. Most of the Custom Android ROMs available are rooted. Some of the major advantages of rooting the phone are: Ability to do OS level tweaks: As numerous users/developers have observed, the stock Android ROM is not capable of exploiting the potential of the underlying hardware. With custom

suit of understanding if the use of Compcache was causing improvements in performance. We carried out a string of tests in order to compare the Android System performance with Compcache and Swap enabled. In this process, we ran a few tests where we articially limited the main memory available to Android and created memory pressure using a program that we created. We have tested the performance of the System under congurations wherein Swap is enabled. The rest of the paper is organized as follows. In Section 2 we briey describe the background. Section 3 explains the approach used to do benchmarking, along with some results and analysis regarding the same. Section 4 outlines the method we followed to perform tests in order to evaluate Swap behavior of the system under memory pressure, along with results and analysis. In Section 5, we draw conclusions and discuss future work. Section 6 is acknowledgements and Section 7 is resources for detailed information.

2.2

Compcache

Compcache (Compressed Caching) is basically virtual Swap. It sets aside a certain percentage of the RAM as compressed Swap. It compresses the data that would otherwise go to Swap. It creates a RAM based block device called as ramzswap which acts as a Swap disk. Hence, the pages that are swapped to the disk are compressed and stored in memory itself. Compressing pages and keeping them in RAM virtually increases its capacity and thus allows to t more applications in the given amount of memory.

Benchmarking

Background

In this section, we briey explain the relevant concept of the Compcache system which we use in later sections for our experiments. In order to compare and contrast, we also give a brief overview of traditional Swap system and the methods used in the Stock Android System.

Here, we will discuss the benchmarking approach used along with the results and their respective analysis. In order to compare the performance of the Stock and Custom ROMs, we chose to run benchmarks as they provide a reliable means by which the various Android System characteristics can be measured on common ground. Moreover, benchmarking lets us do comparison from a larger perspective and depending on the results obtained, we can further narrow down into a desired area.

3.1

Benchmarking approach and setup

2.1

Swap and Androids lowmemorykiller

In the traditional Swap system, when the RAM is used up, the computer moves inactive blocks of memory i.e. pages from the main memory, to the secondary memory. This is done in order to free up RAM which can be utilized by active processes. When a page needs to be accessed again, it is swapped back into the RAM and an inactive page in the RAM will be swapped out to the secondary memory. However, disks and SD cards are slower than physical RAM, so there is performance loss when something needs to be swapped. As opposed to the traditional Swap System, Androids Memory Manager kills inactive processes to free up memory. The processes get a signal from the Android System, after which they write out specic information about their state before exiting. When the application restarts, it is loaded back from storage. It then retrieves the most recent state information that it saved while closing. The Android System sets up six different levels of open applications in the init.rc le. These levels are FOREGROUND APP, VISIBLE APP, SECONDARY SERVER, HIDDEN APP, CONTENT PROVIDER and EMPTY APP. We can tweak these parameters which are present in /sys/module/lowmemorykiller/parameters/minfree 2

There are many benchmarks available in the Android market to measure various system characteristics of Android phone. We chose the following four Android benchmarking applications in order to compare the important metrics like CPU, I/O, 2D Graphics, 3D Graphics, RAM etc. of Stock ROM versus those of the Custom CyanogenMod ROM: 1. Quadrant Professional [9] v1.1.2 2. Antutu Benchmark [2] v2.3.1 3. Smartbench [12] 2011 4. Linpack [8] v1.2.8 We collected the results of ten runs of each of the above applications on the following combination of ROMs: 1. The Stock Android ROM with Samsung Kernel (Gingerbread 2.3.5 with kernel v2.6.35.7) 2. CyanogenMod 4G Stable Version (Android v2.3.7 with kernel v2.6.35.14) 3. CyanogenMod 4G Stable Version with Samsung Kernel (Android v2.3.7 with kernel v2.6.35.7)

Figure 1: Benchmarking using Quadrant 4. CyanogenMod 4G Stable Version with Overclocking enabled (Android v2.3.7 with kernel v2.6.35.14. CPU MAX set to 1440 Mhz as opposed to the default 1000Mhz) The use of a mixed conguration like CyanogenMod with Samsung Kernel was basically done to compare and thus isolate the performance contribution from the kernel itself. We also wanted to see the impact of CPU overclocking on performance of the system because of which we used the fourth conguration.

Figure 2: Benchmarking using Antutu

3.2

Benchmarking using Quadrant

Figure 3: Benchmarking using Smartbench ory Performance, CPU (giving the CPU Floating point performance and CPU Integer Performance), Graphics (giving 2D and 3D Graphics performance), SD card (giving writing and reading speed) and IO performance testing on the mobile phone to compute the scores. Figure 2. shows the results that we obtained on using Antutu

This benchmark does performance evaluation with respect to the following categories: 1. CPU : The benchmark pushes the device through checks like compression, checksum, arithmetics,oating point, XML parsing, and video / audio decoding. 2. Memory : Indication of throughput 3. I/O : This looks over le system access and database operations 4. 2D and 3D graphics tests : This test, consisting of several graphics simulations, evaluates the devices OpenGL capabilities, analyzing single-pass and multi-pass rendering with stencil buffers. Figure 1. shows the results that we obtained on using Quadrant

3.4

Benchmarking using Smartbench

Smartbench 2011 is a multi-core friendly benchmark application that measures the overall performance of the smartphone. The overall performance is indicated by two metrics: 1. Productivity index 2. Games Index. Figure 3. shows the results that we obtained on using Smartbench 3

3.3

Benchmarking using Antutu

Antutu is yet another Android benchmarking tool which can run a full test of a key project, through the Mem-

CyanogenMod with overclocked kernel, as expected, outperformed all other congurations as shown in the benchmark results. This could be attributed to the CPU being overclocked to 1440 Mhz. In the process of evaluating the benchmarking results, we observed that there is signicant difference in I/O scores of Stock Android and CyanogenMod.

Additional experiments

Figure 4: Benchmarking using Linpack

After the results of benchmarking experiments, we decided to look into the IO system of Android. We wanted to run some experiments in this area and try to nd out which factors in the IO System inuenced signicant improvement, in case of CyanogenMod. So, in this process we followed two approaches in order to test IO.

3.5

Benchmarking using Linpack 4.1 First Approach

We evaluated the impact of alternate memory management techniques in custom ROM like Swap and Compcache on the overall performance of the Android System by running the same set of benchmarks. We ran the benchmarking applications on CyanogenMod with Compcache enabled for different ratios of Compcache initially and then by disabling Compcache and enabling swap for various Swappniess values. We again observed striking differences in I/O scores for few tests. Swap enabled CyanogenMod had comparatively better scores than Compcache enabled CyanogenMod, in many cases. We were also interested in measuring performance by using some prominent IO benchmarks for Linux. IOzone [7] is one such benchmark. We ported IOzone on the Android platform and used it to perform read/write tests again with Compcache enabled. Later, we paused with proceeding in the direction of the First Approach when we discovered that there were some Compcache driver issues with CyanogenMod, on Nexus S specically. However, detailed results of this experiment can be found on our website mentioned in Section 7.

Linpack Benchmark is a measure of a systems oating point computing power. Linpack instructs the device to solve a linear equation to measure the systems oating point computing power in Millions of Floating-point Operations Per Second (MFLOP/s). The value of linpack scores also reects the state of the Android Dalvik Virtual Machine along with the oating point performance of the underlying processor. Figure 4. shows the results that we obtained on using Linpack

3.6

Observations and Analysis of benchmarking results

In Quadrant, there is no signicant difference in CPU, 2D, 3D scores for stock Samsung kernel, CyanogenMod kernel and CyanogenMod with stock kernel. However, CyanogenMod and CyanogenMod with Stock kernel outperformed Stock kernel in I/O as reected in I/O score of Quadrant. Figure 4 shows the results of Linpack benchmark. The overall performance of the system is indicated in terms of Millions of Floating-point Operations Per Second (MFLOP/s).In case of Linpack, CyanogenMod and CyanogenMod with Samsung kernel performs relatively better than stock kernel. In Smartbench 2011, there is no noticeable difference when it came to productivity index but in games index, which is an indication of how well processor can control graphics and display on screen is signicantly higher for CyanogenMod compared to stock kernel. In Antutu, CyanogenMod performed much better than Stock kernel in areas of Database I/O and SD card write. The scores are almost consistent for other metrics like RAM, CPU integer, CPU oating-point, 2D Graphics, 3D Graphics. 4

4.2

Second Approach

A set of experiments were created to test the effectiveness of the possible memory hierarchy options available in CyanogenMod. This involved creating articial memory pressure in two ways to test out the use and efcacy of Swap space and Compcache. The rst method involved changing the kernel boot parameters and compiling the OS image so as to make it just utilize specic portions of the physical memory, thus decreasing the effective available memory. Multiple OS images with different effective memory limits were

Figure 5: Snapshot of a portion of Androids System memory map, depicting System RAM created, in the process also helping us identify the bare minimum memory required by Android to bootup. Figure 5. shows a snapshot of a portion of the current map of systems memory for each physical device for Android. We can modify the kernel boot parameters by adding the mem parameter in order to limit the System RAM from the second block which exists from 40000000 to 47ffffff. An example can be as follows: mem=80M@0x30000000 mem=128M@0x40000000 mem=127M@0x50000000. These are passed into the kernel command line in the following make le while compiling Cyanogenmod/Stock from source: /device/samsung/crespo/BoardCongCommon.mk. The size of the resulting overall memory after articially limiting it can be veried after compilation by looking at /proc/meminfo or the output of dmesg which outputs memory in blocks of System RAM. The second way of creating an articial memory pressure was implemented by writing native C code which dynamically allocated different amounts of memory. These levels of memory ranged from small values easily allocated by the system, to values higher than the available physical memory, forcing the Android System to make use of Swap/Compcache for the completion of the task. The allocated dynamic memory was randomly accessed instead of a linear access to alleviate the chances of swapped out pages not being reused and thus not being a true reection for their performance. The running time, available free memory and the runtime usage of Swap were simultaneously recorded for each of the test runs, using the free and vmstat commands respectively. For the code, please visit our website mentioned in Section 7. The memory management scheme in the stock Android takes away much of these gains obtained by Swap/Compcache since it has a per application memory limit of (16/24MB) for the Android Applications and the Android low memory killer starts killing applications when faced with memory pressure. One of the major visible advantage of using Swap/compcache is the ability to run more number of applications before the out of memory killer kicks in. 5

Figure 6: Program execution on limited memory (Without using Swap). Original memory available is 512MB RAM. Here, memory size is limited by 128MB. Program gets killed at malloc size 140. As a part of the second approach of limiting Android System memory, we got a chance to test the performance using our program, by enabling Swap on the phone. We used the Swapper 2 [13] application to enable Swap space on the phone. We set the Swap value to 32MB and Swappniess value to 100. We can see the results in Figure 6. and Figure 7. As expected, when we use Swap, our program gets killed at a later stage than without using Swap. However, there is a tradeoff as the executing time taken is less in case of not using Swap, thus resulting in a loss in performance due to the obvious processing and IO overheads. The use of Swap enables applications with memory requirements higher than available free memory to run to completion instead of getting killed.

Conclusions and Future work

As we saw in the previous section, the behavior of enabling Swap is as expected. It will be interesting to evaluate the performance of Compcache under similar circumstances. We also plan to resolve some Compcache driver issues that exist with CyanogenMod, specically on Nexus S. We would also like to enable Swap in the Stock Android ROM and compare and contrast its performance with Swap enabled CyanogenMod to check if we get similar results. It might be interesting to explore the possibility of making an intelligent decision, wherein the use of either Swap or Compcache can be triggered dynamically, to achieve maximum performance. We would also like to experiment with using different IO task schedulers for Android. One of such schedulers

[11] Setcpu. http://www.setcpu.com/ . [12] Smartbench 2011. http://smartphonebenchmarks.com/ . [13] Swapper 2. http://android.n3o.lv/ . [14] Titanium backup. http://matrixrewriter.com/android/ . [15] Wireless tether. http://code.google.com/p/android-wi-tether/ . [16] T UDUCE , I. C., AND G ROSS , T. Adaptive Main Memory Compression, Dec. 2006.

Figure 7: Program execution on limited memory (Using Swap). Original memory available is 512MB RAM. Here, memory size is limited by 128MB. Program gets killed at malloc size 190. that can be potentially experimented with is the Brain Fuck Scheduler(BFS) [3].

Acknowledgments

We would like to thank Geoffrey Challen and Steven Ko, who provided us with invaluable guidance during the course of this project. We also appreciate the comments and feedback that we received from our fellow classmates which helped us a lot.

Resources

For detailed benchmarking data and results, graphs, code and other resources, please visit our website. Following is the link: https://sites.google.com/site/622prj/

References
[1] Android. http://www.android.com/ . [2] Antutu. http://www.antutu.com/ . [3] Brain fuck scheduler. faq.txt. http://ck.kolivas.org/patches/bfs/bfs-

[4] Compcache. http://code.google.com/p/compcache/ . [5] Connectbot. http://code.google.com/p/connectbot/ . [6] Cyanogenmod. http://www.cyanogenmod.com/ . [7] Iozone lesystem benchmark. http://www.iozone.org/ . [8] Linpack. http://www.greenecomputing.com/ . [9] Quadrant standard edition. http://www.aurorasoftworks.com/ . [10] Rom manager. http://www.koushikdutta.com/2010/02/clockworkrecovery-image.html.