1039810 - Firefox Beta for Android July 15th 2014 build has an "invalid" APK

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Will Pittenger]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:24.0) Gecko/20140610 Firefox/24.0 PaleMoon/24.6.2 (Nightly/Aurora)
Build ID: 20140610142618

Steps to reproduce:

Google Play tells me there's an update for Firefox Beta available.  But when I have Google Play install it, I'm told the APK is invalid.  This has happened repeatedly.


Actual results:

Google Play shouldn't have had any issues.


Expected results:

The APK was reported to be invalid.

Comment 1

[Will Pittenger]

Please note that I'm running on a Intel-based tablet (ASUS ME302).

Comment 2

[Aaron Train [:aaronmt]]

I believe there is a build-date difference throwing off the update precedence order on Google Play.

2014071417 (31.0) → (ARMv7)
2014071416 (31.0) → (x86)
2014071414 (31.0) → (ARMv6)

So your device is downloading an ARMv7 build by accident.

Comment 3

[Aaron Train [:aaronmt]]

As I understand it (x86) typically has the highest build date-date version.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

We do three separate jobs (armv6, armv7, x86), and internally they have a version (in Google Play speak) which should be 2014071414, 2014071415, 2014071416 respectively. So the armv7 being 2014071417 is definitely incorrect.

It turns out the build for armv7 failed, due to a source error for the ms locale. We reran the job when that was resolved, and buildbot was supposed to use the same MOZ_BUILD_ID when calling make. That would have ensured the version in the apk was correct. In fact a newer MOZ_BUILD_ID was used, resulting in a version of 2014071417. This is a newly discovered buildbot bug.

The problem was missed when the apk's were uploaded to Google Play. The wiki notes do mention checking the version, and Google Play has this warning:

Some devices with Native platforms containing any of [x86] are eligible to receive version 2014071416, which is better optimized for the device's Native Platform, but actually receive version 2014071417 because it has a higher version code and the device supports Native platforms containing any of [armeabi-v7a] either directly (e.g. ARMv7 devices support a superset of ARMv5TE instructions) or indirectly (e.g. some x86 devices support ARMv7 or ARMv5TE via native code translation). This would occur when
API levels in range 8+ and
Screen layouts containing any of [small, normal, large, xlarge] and
OpenGL ES versions in range 2.0+ and
Features containing all of [android.hardware.TOUCHSCREEN].

Unfortunately it's routine for Play to warn about something, so it's easy to miss this.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I don't think we can pull the x86 build, as the version for 31.0b8 is even lower. We'd have to recompile to fix the existing apks. Maybe we should roll back to 31.0b8 to prevent more users updating on x86.

Comment 6

[Nick Alexander :nalexander [he/him]]

I typed this, mid-aired with nthomas, and thought I'd include it for general education.  The relevant version is defined in the manifest here:

http://hg.mozilla.org/mozilla-central/file/f6e46d1fc903/mobile/android/base/AndroidManifest.xml.in#l7

and produced here:

http://hg.mozilla.org/mozilla-central/file/f6e46d1fc903/mobile/android/base/Makefile.in#l27

My reading of the logic, and the helpful comments, gives:

N+1 -> x86
N   -> ARMv7
N-1 -> ARMv6

Comment 7

[Richard Newman [:rnewman]]

The Play console for Beta currently has a big yellow warning, if we'd only care to look:

---
Some devices with Native platforms containing any of [x86] are eligible to receive version 2014071416, which is better optimized for the device's Native Platform, but actually receive version 2014071417 because it has a higher version code and the device supports Native platforms containing any of [armeabi-v7a] either directly (e.g. ARMv7 devices support a superset of ARMv5TE instructions) or indirectly (e.g. some x86 devices support ARMv7 or ARMv5TE via native code translation). This would occur when 
---

Comment 8

[Richard Newman [:rnewman]]

Apologies, I mid-aired with nthomas.

Comment 9

[Aaron Train [:aaronmt]]

Target 31.0b11-build1 sign-off tomorrow (07/17); email sent to team.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

re the buildbot issue ....

Inspecting the db to see if the buildid property was carried forward in the rebuild. tl;dr - it wasn't, bu we don't know why.

mysql> select id, buildsetid, buildername, from_unixtime(submitted_at) as submitted_at, complete, results from buildrequests where buildername="release-mozilla-beta-android_build" order by id desc limit 5;
+----------+------------+------------------------------------+---------------------+----------+---------+
| id       | buildsetid | buildername                        | submitted_at        | complete | results |
+----------+------------+------------------------------------+---------------------+----------+---------+
| 46026661 |   10142621 | release-mozilla-beta-android_build | 2014-07-14 17:47:39 |        1 |       0 |
| 46016527 |   10141394 | release-mozilla-beta-android_build | 2014-07-14 15:43:13 |        1 |       2 |
| 45505051 |   10075448 | release-mozilla-beta-android_build | 2014-07-07 15:55:04 |        1 |       0 |
| 44991678 |   10008069 | release-mozilla-beta-android_build | 2014-06-30 18:37:34 |        1 |       0 |
| 44459231 |    9937681 | release-mozilla-beta-android_build | 2014-06-23 17:26:15 |        1 |       0 |
+----------+------------+------------------------------------+---------------------+----------+---------+
5 rows in set (0.01 sec)

mysql> select * from buildset_properties where buildsetid in (select buildsetid from buildrequests where buildername="release-mozilla-beta-android_build" and submitted_at >=1403569575) and property_name in ('buildid','builduid') order by buildsetid desc, property_name;
+------------+---------------+-----------------------------------------------------------+
| buildsetid | property_name | property_value                                            |
+------------+---------------+-----------------------------------------------------------+
|   10142621 | builduid      | ["337d75a8c845406eb25bc1a1cf7dd171", "buildUIDSchedFunc"] |
|   10141394 | buildid       | ["20140714154313", "buildIDSchedFunc"]                    |
|   10141394 | builduid      | ["337d75a8c845406eb25bc1a1cf7dd171", "buildUIDSchedFunc"] |
|   10075448 | buildid       | ["20140707155504", "buildIDSchedFunc"]                    |
|   10075448 | builduid      | ["7b8a4bcb45f34133b7c81935705aba9a", "buildUIDSchedFunc"] |
|   10008069 | buildid       | ["20140630183734", "buildIDSchedFunc"]                    |
|   10008069 | builduid      | ["6842f5ad23a94f68a6543b8a17a8962b", "buildUIDSchedFunc"] |
|    9937681 | buildid       | ["20140623172615", "buildIDSchedFunc"]                    |
|    9937681 | builduid      | ["07ec4eadd7554f588c657bab9cdb5581", "buildUIDSchedFunc"] |
+------------+---------------+-----------------------------------------------------------+
9 rows in set (0.01 sec)

Comment 11

[Aaron Train [:aaronmt]]

Hey Will,

An update was posted to Google Play today for Firefox Beta that fixes this. Are you able to install that update now?

Comment 12

[Will Pittenger]

No.

Comment 13

[Will Pittenger]

Now it works.  Aaron seemed to think it might have been a cache issue on Google's server.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the report Will.