Created attachment 8677493 [details] UpdateService log UpdateService fails to verify downloaded update package on Fennec 44.0a1 2015-10-21. Same issue as in bug 1194741, bug 1179382, etc. Logs attached.
I suspect that the aws post-upload changes subtly messed with the ordering here, since we are somehow getting "latest-mozilla-central-android" in the fileUrl instead of the unique date one (ie: 2015/10/2015-10-22-03-05-46-mozilla-central-android-api-9): "fileUrl": "http://download.cdn.mozilla.net/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk So we probably just need to fix the crappy mozharness logic that tries to pull out the right fileUrl from the upload output again. However, I'm having trouble testing this in staging since I can't upload to the new stage host: $ ssh -o IdentityFile=/home/cltbld/.ssh/ffxbld_rsa firstname.lastname@example.org Permission denied (publickey,gssapi-keyex,gssapi-with-mic). (ffxbld_rsa is the staging ffxbld key). :nthomas - what key & user do I need to upload in staging nowadays?
BTW this appears to be an issue on Aurora as well.
The staging server has been setup to accept the stage ffxbld key. I've disabled updates for Fennec nightly and aurora channels, to avoid repeated downloads which don't apply. The desktop nightlies are not affected because funsize is doing the right thing there. We can use the nightlytest and/or auroratest channels to verify a fix. Visit about:config, and search for app.update.url.android, then replace %CHANNEL% with nightlytest or auroratest. Then restart Fennec, visit about:firefox and 'Check for Updates'. 'adb logcat' is useful if app.update.log is true. To restore updates, use 'https://aus4-admin.mozilla.org/rules#bug%201217431' to find the rules which will need to be reverted afterwards. Please note that we have 4 rules, and the correct mappings to restore are: Fennec : aurora - os 2.3, 2.3.1, 2.3.2 - Fennec-mozilla-aurora-api-9-nightly-latest Fennec : nightly - os 2.3, 2.3.1, 2.3.2 - Fennec-mozilla-central-api-9-nightly-latest Fennec : aurora - no OS - Fennec-mozilla-aurora-nightly-latest Fennec : nightly - no OS - Fennec-mozilla-central-nightly-latest
I think perhaps the old post_upload didn't print out the "latest" URL?
(In reply to Chris AtLee [:catlee] from comment #4) > I think perhaps the old post_upload didn't print out the "latest" URL? That makes since as the file that is downloaded is NOT the latest one.
I think they were displayed, but buildbase.py relies on the YYYY/MM one being displayed first, and now the latest-mozilla-central ones are first. I'm trying to test it out in staging.
(In reply to Bill Gianopoulos [:WG9s] from comment #5) > (In reply to Chris AtLee [:catlee] from comment #4) > > I think perhaps the old post_upload didn't print out the "latest" URL? > > That makes since as the file that is downloaded is NOT the latest one. Hmm, what do you mean here? I would think the updater is now always pulling 'http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-9/fennec-44.0a1.multi.android-arm.apk' since that is what is being (incorrectly) submitted to balrog.
Actually catlee is right, the latest- ones weren't displayed before. I was thinking of the tinderbox urls (eg: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-central-android-api-9/1445508346/fennec-44.0a1.multi.android-arm.apk).
Created attachment 8678320 [details] [diff] [review] 0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch The URL fun continues!
Comment on attachment 8678320 [details] [diff] [review] 0001-Bug-1217431-Don-t-get-latest-URLs-for-android-update.patch Review of attachment 8678320 [details] [diff] [review]: ----------------------------------------------------------------- I *think* this will work though it's fragility and complexity is getting harder to grep. e.g. I'm still a little confused about the `self.matches['packageUrl'] = m` part. I think it will get executed more than once with different values for `m`. your patch makes sure we only execute `self.matches['completeMarUrl'] = m` once but I'm less confident about packageUrl which gets used here: https://dxr.mozilla.org/mozilla-central/rev/0625c68c0abcfe4d10880d15d8fe7d06df3369c9/testing/mozharness/mozharness/mozilla/building/buildbase.py#1779 I wonder if there are other side effects we are missing or if things are broken for other MakeUploadOutputParser calls, particularly ones that do not call with `use_package_as_marfile=True` :\
Well I'll try it out and see what happens... :nthomas, can you take a look & see if we're hosing anything up with wrong packageUrls (either before or after this patch)?
(In reply to Michael Shal [:mshal] from comment #12) > Well I'll try it out and see what happens... > > :nthomas, can you take a look & see if we're hosing anything up with wrong > packageUrls (either before or after this patch)? fwiw - I poked a few jobs and I think we might be okay. I'm speculating now that my speculation was off.
Justification for landing on m-c & m-a: * merging appears to take a break over the weekend (fair enough) * the code change is in a nightly only path (only multi-l10n sets use_package_as_marfile=True) * nightly updates are disabled (comment #3), so the only risk is that nightlies burn in multi-l10n * it's getting near merge time, and we need the feedback and crash data https://hg.mozilla.org/mozilla-central/rev/d53a52b39a95 https://hg.mozilla.org/releases/mozilla-aurora/rev/577654e32aa8
In an aurora nightly: 02:43:12 INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk 02:43:12 INFO - http://archive.mozilla.org/pub/mobile/nightly/latest-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk 02:43:12 INFO - Using package as mar file: http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk 02:43:12 INFO - http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk 02:43:12 INFO - Skipping wrong packageUrl: http://archive.mozilla.org/pub/mobile/tinderbox-builds/mozilla-aurora-android-api-9/1445761500/fennec-43.0a2.multi.android-arm.apk .... 02:43:13 INFO - Setting buildbot property completeMarUrl to http://archive.mozilla.org/pub/mobile/nightly/2015/10/2015-10-25-01-25-00-mozilla-aurora-android-api-9/fennec-43.0a2.multi.android-arm.apk ie, it's using an url like nightly/2015/10/ insted of nightly/latest-mozilla-aurora/, which is all good. Verified updates work on-device for an api-11+ build on the auroratest channel, then re-enabled aurora updates. Nightlies for mozilla-central (Nightly) are running now, but I won't be able to verify the fix and re-enable updates for another half-day or so.
Data in the update server for Nightly looks good, so updates have been re-enabled there. Verified that the api-11+ build updates without error. Thanks for the patch Mike, was there anything else you were planning for this bug ?
(In reply to Nick Thomas [:nthomas] from comment #17) > Thanks for the patch Mike, was there anything else you were planning for > this bug ? Thanks for uplifting & verifying! I was not planning to do anything else here unless something is still broken.