1194741 - Updater fails for Android Nightly: Package hash does not match

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[u421692]

Nightly 43.0a1 (2015-08-13) installed on the device
1. Go to about:firefox and check for updates
2. After build is downloaded tap on "Install update"

Expected result:
Should update to Nightly 43.0a1 (2015-08-14)

Actual result:
Nothing happens

Errors in logcat:
E/UpdateService(32643): Package hash does not match
E/UpdateService(32643): Not installing update, failed verification

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

Michael, this seems to be biting us again.

Comment 2

[Michael Shal [:mshal]]

:(. I suspect this is fallout from bug 1118778. Unless I get a better idea or solution by the end of the day I'll try backing that out.

Comment 3

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1194741-Display-upload-output.patch

The problem is by redirecting the post_upload.py stderr to stdout, the "http://" lines that mozharness was looking for were no longer there. As a result of the weird android property setting, we were getting the en-US binary uploaded to balrog but with the multi sha/size parameters. Printing the output seems to fix it in staging. I imagine this would be much cleaner if we moved multi-l10n in the build system instead of mozharness.

Comment 4

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8648253 [details] [diff] [review]
0001-Bug-1194741-Display-upload-output.patch

14:11 nalexander: mshal: sure.
14:11 nalexander: mshal: I'd make the comment clearer:
14:12 nalexander: mshal: "We print since mozharness may parse URLs from the output stream."
14:12 nalexander: mshal: as formatted, it looks like two statements.
14:12 nalexander: mshal: anyway, r+.

Comment 5

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/04ad18f8377c

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've locked nightly updates to the 20150813030208 build, the last good data in the update server. This is to prevent users from downloading the build from tinderbox-builds and it failing to install, and the loop repeating frequently.

Once we have the fix in m-c, andh ave nightlies based on that, we should check the updates server (ie that the urls in the Fennec-mozilla-central-api-9-nightly-latest and Fennec-mozilla-central-nightly-latest releases are .../mobile/nightly/....) then unlock the updates.

Comment 7

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #6)
> I've locked nightly updates to the 20150813030208 build, the last good data
> in the update server. This is to prevent users from downloading the build
> from tinderbox-builds and it failing to install, and the loop repeating
> frequently.
> 
> Once we have the fix in m-c, andh ave nightlies based on that, we should
> check the updates server (ie that the urls in the
> Fennec-mozilla-central-api-9-nightly-latest and
> Fennec-mozilla-central-nightly-latest releases are .../mobile/nightly/....)
> then unlock the updates.

Well if that is what needs to be done this change seems to be incomplete.  I just went to nightly.mozilla.org and downloaded the build offered and it is today's nightly.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

nightly.mozilla.org is completely separate from the update server, it just points at the latest-mozilla-central directories on the ftp server. 

The issue here is that the nightly runs and uploads to ftp as normal, then the location it ended up is scraped out of upload output, and sent to the update server. The theory is that the scraping is going wrong because the output is different since bug 1118778.

Comment 9

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #8)
> nightly.mozilla.org is completely separate from the update server, it just
> points at the latest-mozilla-central directories on the ftp server. 
> 
> The issue here is that the nightly runs and uploads to ftp as normal, then
> the location it ended up is scraped out of upload output, and sent to the
> update server. The theory is that the scraping is going wrong because the
> output is different since bug 1118778.

But my question is if we don;t want people going to tinderbox-builds to download things newer than 20150813030208, then perhaps 20150813030208 should be the build provided on the nightly.mozilla.org page as well.

Comment 10

[Bill Gianopoulos [:WG9s]]

The reason I mention this is that the current situation is that it currently downloads the en-US instead of the multi version but does not install it, so you can go to an android file manager and install it or go to nightly.mozilla.org and just install the latest multi build form there, so many nightly users might be doing that.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

The build which is on nightly.mozilla.org is fine. The updater offers the wrong build but it doesn't install.

Comment 12

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #11)
> The build which is on nightly.mozilla.org is fine. The updater offers the
> wrong build but it doesn't install.

No, I ma trying to say if we don't want people installing newer builds than 20150813030208 because they might not update properly even after we fix this, then we should not be offering those builds on nightly.mozila.org.

Comment 13

[Bill Gianopoulos [:WG9s]]

Plus it would be nice if the build we offer on nightly.mozilla.org works with gmail.  But this part of the issue is short term as tomorrows nightly will fix that.  I am more concerned with why it was deemed appropriate to  lock nightly updates to the 20150813030208 build. and why having non fixed version past that build on nightly.mozilla.org is deemed to be OK.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Bill Gianopoulos [:WG9s] from comment #12)
> No, I ma trying to say if we don't want people installing newer builds than
> 20150813030208 because they might not update properly even after we fix
> this, then we should not be offering those builds on nightly.mozila.org.

This is not true. Use the builds from nightly.m.o as you will, they are still the multilocale builds.

(In reply to Bill Gianopoulos [:WG9s] from comment #13)
> I am more concerned with why it was deemed appropriate to 
> lock nightly updates to the 20150813030208 build. and why having non fixed
> version past that build on nightly.mozilla.org is deemed to be OK.

There is no known issue with the updater in the build. However the update server was sending the wrong information, which will be resolved once the patch from this bug makes to a nightly build. Anyone with a newer build is not being offered an update in the meantime (ie we don't downgrade).

Locking is a short term fix to prevent repeated failing updates, which is bad UX. This trumps things which would be 'nice' to have. Updates will be unlocked as soon as they are correct again. You are over-rotating on this.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/04ad18f8377c

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

I unlocked updates at mshal's request, and successfully updated an api-11+ build.