Updater fails for Android Nightly and Aurora: Package hash does not match

RESOLVED FIXED

Status

()

Firefox for Android
General
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: esawin, Assigned: mshal)

Tracking

43 Branch
All
Android
Points:
---

Firefox Tracking Flags

(firefox43 fixed, firefox44 fixed)

Details

Attachments

(2 attachments)

(Reporter)

Description

2 years ago
Created attachment 8677493 [details]
UpdateService log

UpdateService fails to verify downloaded update package on Fennec 44.0a1 2015-10-21.

Same issue as in bug 1194741, bug 1179382, etc. Logs attached.
(Assignee)

Comment 1

2 years ago
I suspect that the aws post-upload changes subtly messed with the ordering here, since we are somehow getting "latest-mozilla-central-android" in the fileUrl instead of the unique date one (ie: 2015/10/2015-10-22-03-05-46-mozilla-central-android-api-9):

"fileUrl": "http://download.cdn.mozilla.net/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk

So we probably just need to fix the crappy mozharness logic that tries to pull out the right fileUrl from the upload output again. However, I'm having trouble testing this in staging since I can't upload to the new stage host:

$ ssh -o IdentityFile=/home/cltbld/.ssh/ffxbld_rsa ffxbld@upload.ffxbld.productdelivery.stage.mozaws.net
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

(ffxbld_rsa is the staging ffxbld key).

:nthomas - what key & user do I need to upload in staging nowadays?
Flags: needinfo?(nthomas)
BTW this appears to be an issue on Aurora as well.
The staging server has been setup to accept the stage ffxbld key.

I've disabled updates for Fennec nightly and aurora channels, to avoid repeated downloads which don't apply. The desktop nightlies are not affected because funsize is doing the right thing there.

We can use the nightlytest and/or auroratest channels to verify a fix. Visit about:config, and search for app.update.url.android, then replace %CHANNEL% with nightlytest or auroratest. Then restart Fennec, visit about:firefox and 'Check for Updates'. 'adb logcat' is useful if app.update.log is true.

To restore updates, use 'https://aus4-admin.mozilla.org/rules#bug%201217431' to find the rules which will need to be reverted afterwards. Please note that we have 4 rules, and the correct mappings to restore are:
  Fennec : aurora   -  os 2.3, 2.3.1, 2.3.2  -  Fennec-mozilla-aurora-api-9-nightly-latest
  Fennec : nightly  -  os 2.3, 2.3.1, 2.3.2  -  Fennec-mozilla-central-api-9-nightly-latest
  Fennec : aurora   -  no OS                 -  Fennec-mozilla-aurora-nightly-latest
  Fennec : nightly  -  no OS                 -  Fennec-mozilla-central-nightly-latest
Flags: needinfo?(nthomas)
I think perhaps the old post_upload didn't print out the "latest" URL?
(In reply to Chris AtLee [:catlee] from comment #4)
> I think perhaps the old post_upload didn't print out the "latest" URL?

That makes since as the file that is downloaded is NOT the latest one.
(Assignee)

Comment 6

2 years ago
I think they were displayed, but buildbase.py relies on the YYYY/MM one being displayed first, and now the latest-mozilla-central ones are first. I'm trying to test it out in staging.
(Assignee)

Comment 7

2 years ago
(In reply to Bill Gianopoulos [:WG9s] from comment #5)
> (In reply to Chris AtLee [:catlee] from comment #4)
> > I think perhaps the old post_upload didn't print out the "latest" URL?
> 
> That makes since as the file that is downloaded is NOT the latest one.

Hmm, what do you mean here? I would think the updater is now always pulling 'http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk' since that is what is being (incorrectly) submitted to balrog.
(Assignee)

Comment 8

2 years ago
Actually catlee is right, the latest- ones weren't displayed before. I was thinking of the tinderbox urls (eg: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-9/1445508346/fennec-44.0a1.multi.android-arm.apk).
(Assignee)

Comment 9

2 years ago
Created attachment 8678320 [details] [diff] [review]
0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch

The URL fun continues!
Assignee: nobody → mshal
Attachment #8678320 - Flags: review?(jlund)
Comment on attachment 8678320 [details] [diff] [review]
0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch

Review of attachment 8678320 [details] [diff] [review]:
-----------------------------------------------------------------

I *think* this will work though it's fragility and complexity is getting harder to grep. e.g. I'm still a little confused about the  `self.matches['packageUrl'] = m` part. I think it will get executed more than once with different values for `m`.

your patch makes sure we only execute `self.matches['completeMarUrl'] = m` once but I'm less confident about packageUrl which gets used here: https://dxr.mozilla.org/mozilla-central/rev/0625c68c0abcfe4d10880d15d8fe7d06df3369c9/testing/mozharness/mozharness/mozilla/building/buildbase.py#1779

I wonder if there are other side effects we are missing or if things are broken for other MakeUploadOutputParser calls, particularly ones that do not call with `use_package_as_marfile=True` :\
Attachment #8678320 - Flags: review?(jlund) → review+

Comment 11

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/9999357e9d94
(Assignee)

Comment 12

2 years ago
Well I'll try it out and see what happens...

:nthomas, can you take a look & see if we're hosing anything up with wrong packageUrls (either before or after this patch)?
(In reply to Michael Shal [:mshal] from comment #12)
> Well I'll try it out and see what happens...
> 
> :nthomas, can you take a look & see if we're hosing anything up with wrong
> packageUrls (either before or after this patch)?

fwiw - I poked a few jobs and I think we might be okay. I'm speculating now that my speculation was off.

Comment 14

2 years ago
https://hg.mozilla.org/mozilla-central/rev/d53a52b39a95
Justification for landing on m-c & m-a:
* merging appears to take a break over the weekend (fair enough)
* the code change is in a nightly only path (only multi-l10n sets use_package_as_marfile=True)
* nightly updates are disabled (comment #3), so the only risk is that nightlies burn in multi-l10n
* it's getting near merge time, and we need the feedback and crash data

https://hg.mozilla.org/mozilla-central/rev/d53a52b39a95
https://hg.mozilla.org/releases/mozilla-aurora/rev/577654e32aa8
In an aurora nightly:

02:43:12     INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - Using package as mar file: http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk
02:43:12     INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-aurora-android-api-9/1445761500/fennec-43.0a2.multi.android-arm.apk
....
02:43:13     INFO - Setting buildbot property completeMarUrl to http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk

ie, it's using an url like nightly/2015/10/ insted of nightly/latest-mozilla-aurora/, which is all good. Verified updates work on-device for an api-11+ build on the auroratest channel, then re-enabled aurora updates. 

Nightlies for mozilla-central (Nightly) are running now, but I won't be able to verify the fix and re-enable updates for another half-day or so.
Data in the update server for Nightly looks good, so updates have been re-enabled there. Verified that the api-11+ build updates without error. 

Thanks for the patch Mike, was there anything else you were planning for this bug ?
status-firefox43: --- → fixed
status-firefox44: affected → fixed
Summary: Updater fails for Android Nightly: Package hash does not match → Updater fails for Android Nightly and Aurora: Package hash does not match
Version: Trunk → Firefox 43
(Assignee)

Comment 18

2 years ago
(In reply to Nick Thomas [:nthomas] from comment #17)
> Thanks for the patch Mike, was there anything else you were planning for
> this bug ?

Thanks for uplifting & verifying! I was not planning to do anything else here unless something is still broken.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.