Updater fails for Android Nightly: Package hash does not match

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: snorp, Assigned: mshal)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Logcat states that the hash does not match. Where there changes around this either in updater or releng recently?
I have the 6/12 nightly installed.
Chris, did anything change recently with AUS that could cause this?
Flags: needinfo?(catlee)
<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="41.0a1" appVersion="41.0a1" platformVersion="41.0a1" buildID="20150614030204">
        <patch type="complete" URL="http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk" hashFunction="sha512" hashValue="0c5f10e8968c3f58f36bea38d80214bc774b8fe75f4eea5bc5d757c81be1ca23092eb7193a693cc6fbaf5840c5960c98d9c364a2feb5eea49ca785c2b54f1a82" size="39458831"/>
    </update>
</updates>


The SHA512 of that APK is indeed not 0c5...:

ef74c5e9cd62cbfcb0250a2cc8b25d3b27c5bc0cfbc24644d8fbe2b8ca52001d842108c94d0bd87d53496c25faffe885fe68ce62a103fd0a944244a9487e5638

And that SHA512 matches the file alongside the build:

http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.checksums


If I had to guess wildly, I'd speculate that the workaround in Bug 1168311 Comment 26 is doing something wrong with the hash. mshal?
Flags: needinfo?(mshal)
Component: General → Mozharness
Product: Firefox for Android → Release Engineering
QA Contact: jlund
Summary: Updater fails on latest nightly → Updater fails for Android Nightly: Package hash does not match
Version: Firefox 34 → unspecified
That's right. Here's the relevant bits of the latest nightly log [1]:

04:23:54     INFO - Using package as mar file: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk
...
04:23:58     INFO - Setting mar properties to match package: /builds/slave/m-cen-and-api-11-ntly-00000000/build/src/obj-firefox/embedding/android/geckoview_example/geckoview_example.apk
...
04:23:58     INFO - Setting buildbot property completeMarSize to 39458831
...
04:23:58     INFO - Setting buildbot property completeMarHash to 0c5f10e8968c3f58f36bea38d80214bc774b8fe75f4eea5bc5d757c81be1ca23092eb7193a693cc6fbaf5840c5960c98d9c364a2feb5eea49ca785c2b54f1a82
...
04:23:58     INFO - Setting buildbot property completeMarUrl to http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk

So it has the right completeMarUrl, but reads geckoview_example.apk to get the size and hash  (those should be 43290910, and start ef74c5e9cd62c).


[1] http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/mozilla-central-android-api-11-nightly-bm71-build1-build0.txt.gz
Duplicate of this bug: 1174720
I've disabled fennec updates on nightly for now.
Flags: needinfo?(catlee)
(Assignee)

Comment 8

3 years ago
Created attachment 8622719 [details] [diff] [review]
android-mar-checksum

I think this fixes it - I've at least manually verified that the url, size, and hash seem to lineup for the 3 android varieties (api-9, api-11, and x86). Without any kind of downstream verification that the update actually works though I'm a bit in the dark.
Assignee: nobody → mshal
Flags: needinfo?(mshal)
Attachment #8622719 - Flags: review?(jlund)
FWIW, this didn't affect api-9 builds because geckoview isn't built there. At comment #7 Chris disabled updates for api-11 and x86 builds.
Comment on attachment 8622719 [details] [diff] [review]
android-mar-checksum

Review of attachment 8622719 [details] [diff] [review]:
-----------------------------------------------------------------

darn.

It's hard to see exactly how this patch will all play out without an actual nightly tested against it. Hot fixing this again seems wrong. Judging by your 'in the dark' comment, I wonder if we should be thinking about rolling back mh android builds and using MozillaBuildFactory until we have the kinks worked out? That way m-c nightly would be fixed again while we have proper time to deal with this in either staging or on a twig that does android nightlies. Would that be possible to do?

I won't block since you have done the work for this one, so if you feel confident about landing, ship it. Saying that, I think we should at the very least lock android nightly in balrog, let the new nightles run against this patch, test, verify, and if all looks well, point balrog back to nightly-latest.
Attachment #8622719 - Flags: review?(jlund) → review+
(Assignee)

Comment 11

3 years ago
(In reply to Jordan Lund (:jlund) from comment #10)
> It's hard to see exactly how this patch will all play out without an actual
> nightly tested against it. Hot fixing this again seems wrong. Judging by
> your 'in the dark' comment, I wonder if we should be thinking about rolling
> back mh android builds and using MozillaBuildFactory until we have the kinks
> worked out? That way m-c nightly would be fixed again while we have proper
> time to deal with this in either staging or on a twig that does android
> nightlies. Would that be possible to do?

This is something I'll be bringing up tomorrow at our group meeting. I'm not sure how much we'd gain with using a twig vs. what I've been able to do so far in staging. My concern is largely that if something gets left out (like a balrog update) or is wrong (like a bad checksum in this case), there isn't an automated way to detect that and fail the build as far as I know. Ideally I'd like to be able to push to try and have a test fail telling me that updates were generated incorrectly or whatever.
Given that this is checked-in, when do we expect updates to be functional again?
(Assignee)

Comment 14

3 years ago
(In reply to Mark Finkle (:mfinkle) from comment #13)
> Given that this is checked-in, when do we expect updates to be functional
> again?

mozharness was just bumped on m-c today and we triggered some android nightly builds that just finished. Can anyone confirm that updates are working? Or if not, what the error seems to be?
I have the 2015-06-11 version and when I click "Check for updates" from about:firefox I see this in logcat:

> I/UpdateService( 7302): next update will be at: Thu Jun 18 20:58:23 CDT 2015
> I/UpdateService( 7302): opening connection with URI: https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150611030208/Android_arm-eabi-gcc3/en-US/nightly/5.0.2/default/default/41.0a1/update.xml
> I/UpdateService( 7302): no update available
Updates are still disabled on nightly. I'm attempting to verify the fix on nightlytest first, but running into some data sync issues in the update server.
nightlytest worked for me, and the app seems fine. Updates have been re-enabled on the nightly channel.
I got an update as well. After updating about:buildconfig points to cset c9a79ec16280 from today, so things are looking good.
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.