1174581 - Updater fails for Android Nightly: Package hash does not match

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

Logcat states that the hash does not match. Where there changes around this either in updater or releng recently?

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

I have the 6/12 nightly installed.

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

Chris, did anything change recently with AUS that could cause this?

Comment 3

[Richard Newman [:rnewman]]

Last commit in Bug 1168311 landed on June 11.

Downloaded URL:

http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk

Update XML:

https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150612030205/Android_arm-eabi-gcc3/en-US/nightly/5.0.2/default/default/41.0a1/update.xml

Comment 4

[Richard Newman [:rnewman]]

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="41.0a1" appVersion="41.0a1" platformVersion="41.0a1" buildID="20150614030204">
        <patch type="complete" URL="http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk" hashFunction="sha512" hashValue="0c5f10e8968c3f58f36bea38d80214bc774b8fe75f4eea5bc5d757c81be1ca23092eb7193a693cc6fbaf5840c5960c98d9c364a2feb5eea49ca785c2b54f1a82" size="39458831"/>
    </update>
</updates>


The SHA512 of that APK is indeed not 0c5...:

ef74c5e9cd62cbfcb0250a2cc8b25d3b27c5bc0cfbc24644d8fbe2b8ca52001d842108c94d0bd87d53496c25faffe885fe68ce62a103fd0a944244a9487e5638

And that SHA512 matches the file alongside the build:

http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.checksums


If I had to guess wildly, I'd speculate that the workaround in Bug 1168311 Comment 26 is doing something wrong with the hash. mshal?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

That's right. Here's the relevant bits of the latest nightly log [1]:

04:23:54     INFO - Using package as mar file: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk
...
04:23:58     INFO - Setting mar properties to match package: /builds/slave/m-cen-and-api-11-ntly-00000000/build/src/obj-firefox/embedding/android/geckoview_example/geckoview_example.apk
...
04:23:58     INFO - Setting buildbot property completeMarSize to 39458831
...
04:23:58     INFO - Setting buildbot property completeMarHash to 0c5f10e8968c3f58f36bea38d80214bc774b8fe75f4eea5bc5d757c81be1ca23092eb7193a693cc6fbaf5840c5960c98d9c364a2feb5eea49ca785c2b54f1a82
...
04:23:58     INFO - Setting buildbot property completeMarUrl to http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/fennec-41.0a1.multi.android-arm.apk

So it has the right completeMarUrl, but reads geckoview_example.apk to get the size and hash  (those should be 43290910, and start ef74c5e9cd62c).


[1] http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/06/2015-06-14-03-02-04-mozilla-central-android-api-11/mozilla-central-android-api-11-nightly-bm71-build1-build0.txt.gz

Comment 7

[Chris AtLee [:catlee]]

I've disabled fennec updates on nightly for now.

Comment 8

[Michael Shal [:mshal]]

Attachment: android-mar-checksum

I think this fixes it - I've at least manually verified that the url, size, and hash seem to lineup for the 3 android varieties (api-9, api-11, and x86). Without any kind of downstream verification that the update actually works though I'm a bit in the dark.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, this didn't affect api-9 builds because geckoview isn't built there. At comment #7 Chris disabled updates for api-11 and x86 builds.

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8622719 [details] [diff] [review]
android-mar-checksum

Review of attachment 8622719 [details] [diff] [review]:
-----------------------------------------------------------------

darn.

It's hard to see exactly how this patch will all play out without an actual nightly tested against it. Hot fixing this again seems wrong. Judging by your 'in the dark' comment, I wonder if we should be thinking about rolling back mh android builds and using MozillaBuildFactory until we have the kinks worked out? That way m-c nightly would be fixed again while we have proper time to deal with this in either staging or on a twig that does android nightlies. Would that be possible to do?

I won't block since you have done the work for this one, so if you feel confident about landing, ship it. Saying that, I think we should at the very least lock android nightly in balrog, let the new nightles run against this patch, test, verify, and if all looks well, point balrog back to nightly-latest.

Comment 11

[Michael Shal [:mshal]]

(In reply to Jordan Lund (:jlund) from comment #10)
> It's hard to see exactly how this patch will all play out without an actual
> nightly tested against it. Hot fixing this again seems wrong. Judging by
> your 'in the dark' comment, I wonder if we should be thinking about rolling
> back mh android builds and using MozillaBuildFactory until we have the kinks
> worked out? That way m-c nightly would be fixed again while we have proper
> time to deal with this in either staging or on a twig that does android
> nightlies. Would that be possible to do?

This is something I'll be bringing up tomorrow at our group meeting. I'm not sure how much we'd gain with using a twig vs. what I've been able to do so far in staging. My concern is largely that if something gets left out (like a balrog update) or is wrong (like a bad checksum in this case), there isn't an automated way to detect that and fail the build as far as I know. Ideally I'd like to be able to push to try and have a test fail telling me that updates were generated incorrectly or whatever.

Comment 12

[Michael Shal [:mshal]]

Comment on attachment 8622719 [details] [diff] [review]
android-mar-checksum

https://hg.mozilla.org/build/mozharness/rev/254c9420f7bb

Comment 13

[Mark Finkle (:mfinkle) (use needinfo?)]

Given that this is checked-in, when do we expect updates to be functional again?

Comment 14

[Michael Shal [:mshal]]

(In reply to Mark Finkle (:mfinkle) from comment #13)
> Given that this is checked-in, when do we expect updates to be functional
> again?

mozharness was just bumped on m-c today and we triggered some android nightly builds that just finished. Can anyone confirm that updates are working? Or if not, what the error seems to be?

Comment 15

[Dave Townsend [:mossop]]

Not working for me, AUS isn't offering any update: https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150612030205/Android_arm-eabi-gcc3/en-US/nightly/5.1.1/default/default/41.0a1/update.xml

Comment 16

[Mike Taylor [:miketaylr]]

I have the 2015-06-11 version and when I click "Check for updates" from about:firefox I see this in logcat:

> I/UpdateService( 7302): next update will be at: Thu Jun 18 20:58:23 CDT 2015
> I/UpdateService( 7302): opening connection with URI: https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150611030208/Android_arm-eabi-gcc3/en-US/nightly/5.0.2/default/default/41.0a1/update.xml
> I/UpdateService( 7302): no update available

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Updates are still disabled on nightly. I'm attempting to verify the fix on nightlytest first, but running into some data sync issues in the update server.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

nightlytest worked for me, and the app seems fine. Updates have been re-enabled on the nightly channel.

Comment 19

[Kartikaya Gupta (email:[e-mail])]

I got an update as well. After updating about:buildconfig points to cset c9a79ec16280 from today, so things are looking good.