1217431 - Updater fails for Android Nightly and Aurora: Package hash does not match

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Eugen Sawin [:esawin]]

Attachment: UpdateService log

UpdateService fails to verify downloaded update package on Fennec 44.0a1 2015-10-21.

Same issue as in bug 1194741, bug 1179382, etc. Logs attached.

Comment 1

[Michael Shal [:mshal]]

I suspect that the aws post-upload changes subtly messed with the ordering here, since we are somehow getting "latest-mozilla-central-android" in the fileUrl instead of the unique date one (ie: 2015/10/2015-10-22-03-05-46-mozilla-central-android-api-9):

"fileUrl": "http://download.cdn.mozilla.net/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk

So we probably just need to fix the crappy mozharness logic that tries to pull out the right fileUrl from the upload output again. However, I'm having trouble testing this in staging since I can't upload to the new stage host:

$ ssh -o IdentityFile=/home/cltbld/.ssh/ffxbld_rsa [e-mail]
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

(ffxbld_rsa is the staging ffxbld key).

:nthomas - what key & user do I need to upload in staging nowadays?

Comment 2

[Bill Gianopoulos [:WG9s]]

BTW this appears to be an issue on Aurora as well.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The staging server has been setup to accept the stage ffxbld key.

I've disabled updates for Fennec nightly and aurora channels, to avoid repeated downloads which don't apply. The desktop nightlies are not affected because funsize is doing the right thing there.

We can use the nightlytest and/or auroratest channels to verify a fix. Visit about:config, and search for app.update.url.android, then replace %CHANNEL% with nightlytest or auroratest. Then restart Fennec, visit about:firefox and 'Check for Updates'. 'adb logcat' is useful if app.update.log is true.

To restore updates, use 'https://aus4-admin.mozilla.org/rules#bug%201217431' to find the rules which will need to be reverted afterwards. Please note that we have 4 rules, and the correct mappings to restore are:
  Fennec : aurora   -  os 2.3, 2.3.1, 2.3.2  -  Fennec-mozilla-aurora-api-9-nightly-latest
  Fennec : nightly  -  os 2.3, 2.3.1, 2.3.2  -  Fennec-mozilla-central-api-9-nightly-latest
  Fennec : aurora   -  no OS                 -  Fennec-mozilla-aurora-nightly-latest
  Fennec : nightly  -  no OS                 -  Fennec-mozilla-central-nightly-latest

Comment 4

[Chris AtLee [:catlee]]

I think perhaps the old post_upload didn't print out the "latest" URL?

Comment 5

[Bill Gianopoulos [:WG9s]]

(In reply to Chris AtLee [:catlee] from comment #4)
> I think perhaps the old post_upload didn't print out the "latest" URL?

That makes since as the file that is downloaded is NOT the latest one.

Comment 6

[Michael Shal [:mshal]]

I think they were displayed, but buildbase.py relies on the YYYY/MM one being displayed first, and now the latest-mozilla-central ones are first. I'm trying to test it out in staging.

Comment 7

[Michael Shal [:mshal]]

(In reply to Bill Gianopoulos [:WG9s] from comment #5)
> (In reply to Chris AtLee [:catlee] from comment #4)
> > I think perhaps the old post_upload didn't print out the "latest" URL?
> 
> That makes since as the file that is downloaded is NOT the latest one.

Hmm, what do you mean here? I would think the updater is now always pulling 'http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk' since that is what is being (incorrectly) submitted to balrog.

Comment 8

[Michael Shal [:mshal]]

Actually catlee is right, the latest- ones weren't displayed before. I was thinking of the tinderbox urls (eg: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-9/1445508346/fennec-44.0a1.multi.android-arm.apk).

Comment 9

[Michael Shal [:mshal]]

Attachment: 0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch

The URL fun continues!

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8678320 [details] [diff] [review]
0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch

Review of attachment 8678320 [details] [diff] [review]:
-----------------------------------------------------------------

I *think* this will work though it's fragility and complexity is getting harder to grep. e.g. I'm still a little confused about the  `self.matches['packageUrl'] = m` part. I think it will get executed more than once with different values for `m`.

your patch makes sure we only execute `self.matches['completeMarUrl'] = m` once but I'm less confident about packageUrl which gets used here: https://dxr.mozilla.org/mozilla-central/rev/0625c68c0abcfe4d10880d15d8fe7d06df3369c9/testing/mozharness/mozharness/mozilla/building/buildbase.py#1779

I wonder if there are other side effects we are missing or if things are broken for other MakeUploadOutputParser calls, particularly ones that do not call with `use_package_as_marfile=True` :\

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/9999357e9d94

Comment 12

[Michael Shal [:mshal]]

Well I'll try it out and see what happens...

:nthomas, can you take a look & see if we're hosing anything up with wrong packageUrls (either before or after this patch)?

Comment 13

[Jordan Lund (:jlund)]

(In reply to Michael Shal [:mshal] from comment #12)
> Well I'll try it out and see what happens...
> 
> :nthomas, can you take a look & see if we're hosing anything up with wrong
> packageUrls (either before or after this patch)?

fwiw - I poked a few jobs and I think we might be okay. I'm speculating now that my speculation was off.

Comment 14

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/d53a52b39a95

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Justification for landing on m-c & m-a:
* merging appears to take a break over the weekend (fair enough)
* the code change is in a nightly only path (only multi-l10n sets use_package_as_marfile=True)
* nightly updates are disabled (comment #3), so the only risk is that nightlies burn in multi-l10n
* it's getting near merge time, and we need the feedback and crash data

https://hg.mozilla.org/mozilla-central/rev/d53a52b39a95
https://hg.mozilla.org/releases/mozilla-aurora/rev/577654e32aa8

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

In an aurora nightly:

02:43:12     INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - Using package as mar file: http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-aurora-android-api-9/1445761500/fennec-43.0a2.multi.android-arm.apk
....
02:43:13     INFO - Setting buildbot property completeMarUrl to http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk

ie, it's using an url like nightly/2015/10/ insted of nightly/latest-mozilla-aurora/, which is all good. Verified updates work on-device for an api-11+ build on the auroratest channel, then re-enabled aurora updates. 

Nightlies for mozilla-central (Nightly) are running now, but I won't be able to verify the fix and re-enable updates for another half-day or so.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Data in the update server for Nightly looks good, so updates have been re-enabled there. Verified that the api-11+ build updates without error. 

Thanks for the patch Mike, was there anything else you were planning for this bug ?

Comment 18

[Michael Shal [:mshal]]

(In reply to Nick Thomas [:nthomas] from comment #17)
> Thanks for the patch Mike, was there anything else you were planning for
> this bug ?

Thanks for uplifting & verifying! I was not planning to do anything else here unless something is still broken.