1256765 - fennec 46.0b2 build1 repacks are failing

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Kim Moir [:kmoir] ET]

11:15:41    ERROR -  make[1]: *** [repackage-zip] Error 1
11:15:41     INFO -  make[1]: Leaving directory `/builds/slave/rel-m-beta-and-api-15_rpk_7-00/build/mozilla-beta/obj-l10n/mobile/android/locales'
11:15:41     INFO -  make: *** [repackage-zip-ml] Error 2
11:15:41     INFO -  WARNING: Command failed. See logs for output.
11:15:41     INFO -   # umount -n /builds/mock_mozilla/mozilla-centos6-x86_64-android/root/builds/slave
11:15:41     INFO -  WARNING: Forcibly unmounting '/builds/mock_mozilla/mozilla-centos6-x86_64-android/root/builds/slave' from chroot.
11:15:41    ERROR - Return code: 2
11:15:41     INFO - Setting buildbot property ml_failure to ml failed in make installers-ml!
11:15:41     INFO - Writing buildbot properties [u'ml_failure'] to /builds/slave/rel-m-beta-and-api-15_rpk_7-00/properties/ml_failure
11:15:41     INFO - Writing to file /builds/slave/rel-m-beta-and-api-15_rpk_7-00/properties/ml_failure
11:15:41     INFO - Contents:
11:15:41     INFO -  ml_failure:ml failed in make installers-ml!
11:15:41    ERROR - ml failed in make installers-ml!
11:15:41  WARNING - setting return code to 1

looking at the machine
[[e-mail] rel-m-beta-and-api-15_rpk_7-00]$ ls /builds/slave/rel-m-beta-and-api-15_rpk_7-00/build/mozilla-beta/obj-l10n/dist
bin  exploded-aar  fennec-46.0b2.en-US.android-arm.apk  gecko.ap_  gecko.apk  gecko-unsigned-unaligned.apk  l10n-stage  system_wrappers  xpi-stage

Comment 1

[Kim Moir [:kmoir] ET]

should have included the former above

scripts/scripts/mobile_l10n.py --cfg single_locale/release_mozilla-beta_android_api_15.py --tag-override FENNEC_46_0b2_RELEASE --cfg single_locale/production.py --no-taskcluster-upload --cfg balrog/production.py --total-chunks 10 --this-chunk 7

11:19:26    FATAL - Found no apks files in /builds/slave/rel-m-beta-and-api-15_rpk_7-00/build/mozilla-beta/obj-l10n/dist, don't know what to do:

Comment 2

[Kim Moir [:kmoir] ET]

nalexander: What apks should be in the directory?

Comment 3

[Kim Moir [:kmoir] ET]

It looks like all the specified files are being downloaded from tooltool so not sure what is missing here.

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to Kim Moir [:kmoir] from comment #2)
> nalexander: What apks should be in the directory?

From http://archive.mozilla.org/pub/mobile/candidates/46.0b2-candidates/build1/logs/release-mozilla-beta-android-api-15_repack_1-bm72-build1-build0.txt.gz I see

11:22:13     INFO -  cp: cannot stat `/builds/slave/rel-m-beta-and-api-15_rpk_1-00/build/mozilla-beta/obj-l10n/mobile/android/bouncer/bouncer-unsigned-unaligned.apk': No such file or directory

so it looks like I have made an error somewhere in the repackaging.  This is very hard to deal with in beta, and this bouncer stuff can't even be used because I now think the build job that uses this is TaskCluster only and thus only runs up to Aurora.  Therefore, let's just disable the bouncer and try again.  Patch incoming.

Comment 5

[Nick Alexander :nalexander [he/him]]

jlund: can you confirm that the "android-partner-sample1" is TC-only, and thus can't be run against Beta?  Thanks!

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1256765 - Disable building Fennec bouncer APK. r=bustage

The bouncer is breaking Beta l10n repacks (and presumably would also
break Release l10n repacks).

This patch disables the bouncer globally, since it's broken past Beta
and the bouncer is not useful on Nightly and Aurora, since it
redirects to the current package in the Play Store, and only
org.mozilla.{firefox,firefox_beta} are in the Play Store.  (It's not
possible to redirect to a different package because we can't write to
different packages.)

Review commit: https://reviewboard.mozilla.org/r/40199/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/40199/

Comment 7

[Nick Alexander :nalexander [he/him]]

kmoir: the patch I've pushed to review should graft/apply cleanly onto beta (and disable the bouncer entirely).  Please land it onto beta if this has to happen quickly; NI me if you want me to land it onto beta; or request approval if it needs approval.  Thanks!

Comment 8

[Jordan Lund (:jlund)]

(In reply to Nick Alexander :nalexander from comment #5)
> jlund: can you confirm that the "android-partner-sample1" is TC-only, and
> thus can't be run against Beta?  Thanks!

yes, partner-sample1 is TC-only and judging by the branches in tree, we can't run this on beta. If we need partner-sample1 as part of fennec release, I'll have to schedule it via buildbot.

Comment 9

[Jordan Lund (:jlund)]

(In reply to Jordan Lund (:jlund) from comment #8)
> (In reply to Nick Alexander :nalexander from comment #5)
> > jlund: can you confirm that the "android-partner-sample1" is TC-only, and
> > thus can't be run against Beta?  Thanks!
> 
> yes, partner-sample1 is TC-only and judging by the branches in tree, we
> can't run this on beta. If we need partner-sample1 as part of fennec
> release, I'll have to schedule it via buildbot.

whoops, branches in tree being: https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches

Comment 10

[Nick Alexander :nalexander [he/him]]

(In reply to Jordan Lund (:jlund) from comment #8)
> (In reply to Nick Alexander :nalexander from comment #5)
> > jlund: can you confirm that the "android-partner-sample1" is TC-only, and
> > thus can't be run against Beta?  Thanks!
> 
> yes, partner-sample1 is TC-only and judging by the branches in tree, we
> can't run this on beta. If we need partner-sample1 as part of fennec
> release, I'll have to schedule it via buildbot.

No need to schedule it via BB.  I believe TC jobs are on Aurora and will ride to Beta as 47/48 rides, which is fast enough without significant partner interest.

Comment 11

[Kim Moir [:kmoir] ET]

Comment on attachment 8730865 [details]
MozReview Request: Bug 1256765 - Disable building Fennec bouncer APK. r=bustage

landed on beta 
http://hg.mozilla.org/releases/mozilla-beta/rev/bde4a1b01275

Comment 12

[Kim Moir [:kmoir] ET]

There are green builds but reftest failures 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=bde4a1b01275

who can at these failures?

Comment 13

[Kim Moir [:kmoir] ET]

talked to rail and these are intermittent failures so cancelling needinfo

Comment 14

[Kim Moir [:kmoir] ET]

fixed