Fighting the Bugs: An Android developer’s adventure.

Fighting the Bugs: An Android developer’s adventure.

No two bugs are alike. Our Android engineer Nikita Belokopytov writes about his personal view on the different types of bugs, and how he almost lost patience with a particular nasty one.

The Good

During my time as an Android developer, I came to appreciate hard crashes. I love seeing a regular, run-of-the-mill null pointer exception in our logs. What it usually means is that the code works more or less, the users enjoy our app and just found some particularity not working as expected. With every such crash fixed, our app becomes more predictable, more polished and more stable. Not all crashes are like that - there are also the nightmarish ones that will return to haunt you during a stormy night: No stacktrace, obscure devices, no way to contact the user, popping up once in awhile with no obvious pattern. Long after you have first caught them on your bugtracker, you keep wondering whether you should spend more time on them, or just accept them as a part of your life and move on. But hey, at least you are aware of their existence.

The Bad

Then, there are non-crashes: problems that you only notice when customers start to complain about them or during a random analytics check-up. In that case, they are hardly distinguishable from a regular irregularity; in fact, a health-metric monitor might miss those. Imagine a problem with password-recovery: The business metrics are only affected in a slight way, not significant enough to warrant an audit, but noticeable if you look close enough at the amount of logged in users and then check the number of successful logins after a password recovery attempt. Such errors are usually semantic - it might be that some exotic loophole made it through the approval process or an edge case turned out to be not so edge in the end such that a lot of users have found a way to trigger it. Just perform a regular analysis of your funnels and keep an eye out for random peaks and valleys in the distribution of your events. Thank God for the analytics! Even if we are slower to react to such problems than to hard crashes, it’s still quite easy to keep these problems in check. Of course, once you detect a semantic problem, it could be still challenging to understand what exactly is happening on the code level. At this point, it might be worthwhile to browse through your app’s reviews or feedback emails. There are usually a couple of tech-savvy people who would send you a hint on steps to reproduce.

And the Nightmare

Finally, there are special cases like the one I’m writing about: an installation error. There is no way to track it because no tracking has been setup yet, and no structured information can be extracted from the users. No crashes, no events, no device information. Unless you have a wide audience, you might not even get an email. Even if you get one, you might just discard it as a user problem.

Time for Joy

Not if you are working at GetYourGuide. Positivity is a core value, and positivity includes that we enjoy overcoming challenges.

We started receiving odd reports of users being unable to install our app once in awhile. Out of tens of good and bad reviews, there would be one that mentions installation problems. The problem would usually pop up on an old device, pre-KitKat – the market segment that every Android developer wishes to magically move to a more modern device overnight. The pain of solving these problems already starts with finding an old enough device or setting up an emulator to reproduce the problem. It does not become better when investigating on the problem’s source.

Please Sign Here: The Technical Details

After grabbing a Nexus 4 and hoping it still had an outdated Android on it (which it had), we tried to install the app through Play Store and managed to reproduce the problem. An obscure error message appeared: “Unknown error code during installation: -103”.

After some web research, we quickly found reports about some obscure problems with the signing algorithm. Somewhere around Java 1.7, the default signing algorithm had switched from SHA1 to SHA256. Coincidentally, Android versions before 4.4 do not have Java 1.7 support thus they had no means to verify the signature upon downloading the APK from Play Store. However, neither Android Studio nor Gradle (in its “signingConfigs”) allow for configuring the signing algorithm.

I exactly knew what to do next: Get the unsigned APK, manually sign it with the same keystore and (less secure) SHA1, and upload an update to the store. That should be easy. I remembered seeing all the intermediary APKs in the our project’s folder “build/output/apk”. One of them should be the unaligned one. I just had to manually zipalign it, sign it and release it.

Unfortunately, this didn’t work as expected. I learnt that there are no more intermediary APKs in the folder for Gradle versions being newer than 2.2. Only the final signed and aligned APK is stored there. My next idea was getting an unsigned APK by removing the signing config from the Gradle file, but the Android Studio build stalled. Alright, I had to do everything myself!

I started by generating an unsigned APK via

          > gradlew assembleRelease

According to Android Studio’s user guide, it’s also unaligned, but unaligned binaries don’t get accepted by the Play Store. So, I had to zip align it manually:

          > zipalign -v -p 4 file.apk aligned-file.apk

That seemed to work. I went further and tried to sign the file:

          > apksigner aligned-file.apk key -keystore keystore.keystore

           -sigalg SHA1withRSA -digestalg SHA1

This failed since the syntax was wrong. I gave it another try, this time with jarsigner which provides much more flexibility than the Android SDK’s own apksigner:

          > jarsigner aligned-file.apk key -keystore keystore.keystore

           -sigalg SHA1withRSA -digestalg SHA1

The result was another setback: “jarsigner error: private key algorithm is not compatible with signature algorithm”. GetYourGuide has been on the market for a long time, so it turned out that our keystore was too old for RSA. Luckily, my fix worked:

          > jarsigner aligned-file.apk key -keystore keystore.keystore

           -sigalg SHA1withTSA -digestalg SHA1

Lo and behold, the long awaited upload to the Play Store:

Wait, what? I had aligned my upload just a minute ago! The answer to this problem was that jarsigner breaks the alignment. I aligned it again:

          > zipalign -v -p 4 aligned-file.apk


This worked. Android 4.1 to 4.3 users could finally enjoy their tours and activities with our app and I slept like a baby that night.

PS. We are going to drop the support of earlier API versions than 19 in the app’s next release. Then, we can sign the app with the more secure SHA256 algorithm.

If you like the way we do our operations, get in touch - we're hiring!

Position Spotlight - Office Manager.

Position Spotlight - Office Manager.

Position Spotlight - Director of Engineering, Marketplace.

Position Spotlight - Director of Engineering, Marketplace.