1168311 - Nightly builds not available

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Teodora Vermesan (:TeoVermesan)]

- I've tried to update Nightly and it appears in about:firefox that (2015-05-21) is the latest build.
- There are no Nightly builds for 22-05 and 23-05
- There are Nightly builds for 24-05 and 25-05, but there are no separate folders for api 9, api 11 or x86. The builds are available in "2015-05-24-03-02-34-mozilla-central" and "2015-05-25-03-02-05-mozilla-central"
- Installing 24-05 build from here and trying to update it, "No updates available" message appears
- Installing 25-05 build from here on a gingerbread device, "There is a problem parsing the package" message appears.

On gingerbread the build could not have been installed
Also installing 24-05 builds from 2015-05-24-03-02-34-mozilla-central din not find any updates.

Comment 1

[Richard Newman [:rnewman]]

It's not surprising that the non-Gingerbread-specific APKs don't install on GB.

The packages are being built (e.g., here's yesterday's Nightly API9), but the copy job doesn't seem to be happening.

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-9/1432548125/en-US/

I pulled that URL from here:

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015-05-25-03-02-05-mozilla-central-android-api-9/mozilla-central-android-api-9-nightly-bm94-build1-build0.txt.gz

Comment 2

[Chris Cooper [:coop] (he/him)]

cc-ing mshal since I suspect this is related to his work getting Android builds off of FTP.

Comment 3

[Michael Shal [:mshal]]

Looks like the 'post_upload_include_platform' logic that is specific to mobile (from bug 557260) was missed when porting to mozharness. I'll try to fix it along with the en-US issue - patch in the works...

Comment 4

[Michael Shal [:mshal]]

Attachment: mobile-upload-platform

This mirrors what buildbotcustom does, which uses 'branch-platform' instead of just 'branch' for mobile builds.

Comment 5

[Michael Shal [:mshal]]

Comment on attachment 8610820 [details] [diff] [review]
mobile-upload-platform

https://hg.mozilla.org/build/mozharness/rev/1411c918fdfe

Comment 6

[Michael Shal [:mshal]]

Attachment: bb-configs-android-balrog.patch

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8612380 [details] [diff] [review]
bb-configs-android-balrog.patch

lgtm

Comment 8

[Michael Shal [:mshal]]

Comment on attachment 8612380 [details] [diff] [review]
bb-configs-android-balrog.patch

https://hg.mozilla.org/build/buildbot-configs/rev/e7d6ecd7bb6a

Comment 9

[Michael Shal [:mshal]]

Attachment: android-update-balrog

This turns on the update step for android builds so that it can submit to balrog. The annoying thing for android is that it doesn't have the usual mar files from an update-packaging step, and instead uses the package itself. See: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#1712

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8612462 [details] [diff] [review]
android-update-balrog

Review of attachment 8612462 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozharness/mozilla/building/buildbase.py
@@ +1390,5 @@
> +            # Find the full path to the package in uploadFiles so we can
> +            # get the size/hash of the mar
> +            for upload_file in files:
> +                if upload_file.endswith(packageName):
> +                    self.set_buildbot_property('completeMarSize',

I think you are forgetting one property here. you have the size and hash but not the filename component of CompleteMar: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#680

I may be wrong but it looks like factory.py sets CompleteMarUrl (based on packageUrl) *and* CompleteMar (based on packageName). The latter, CompleteMar, we set filename, hash, and size properties.

I'm not sure how CompleteMar is used. It might not be needed in mh but the size and hash you have in this patch, iiuc, is useless without filename.

I wrote http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#1432 but I guess you can't use that since it's a copy based on another property

Comment 11

[Michael Shal [:mshal]]

(In reply to Jordan Lund (:jlund) from comment #10)
> I think you are forgetting one property here. you have the size and hash but
> not the filename component of CompleteMar:
> http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#680

It looks like completeMarFilename is only used in partial update generation, which doesn't apply to android:

http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2065

I'm not sure that code is even used anymore since we moved update-packaging in-tree.

> I may be wrong but it looks like factory.py sets CompleteMarUrl (based on
> packageUrl) *and* CompleteMar (based on packageName). The latter,
> CompleteMar, we set filename, hash, and size properties.

For android at least, we need these properties for balrog-submitter.py, which only uses Url, Size, and Hash (not Filename):

http://mxr.mozilla.org/build/source/tools/scripts/updates/balrog-submitter.py#76

> I wrote
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/
> buildbase.py#1432 but I guess you can't use that since it's a copy based on
> another property

Ahh, I didn't know about that. Is it worth switching you think? I was mostly basing the Size/Hash settings off of b2g_build.py and desktop_l10n.py which do similar things.

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 8612462 [details] [diff] [review]
android-update-balrog

Review of attachment 8612462 [details] [diff] [review]:
-----------------------------------------------------------------

okay well let's give it a shot. I guess balrog downloads the mar from the url and then checks the hash/size from the file we had locally (completeMarFilename/packagename). But since we/balrog don't need the filename, we can just store the Size and Hash.

Comment 14

[Michael Shal [:mshal]]

Comment on attachment 8612462 [details] [diff] [review]
android-update-balrog

https://hg.mozilla.org/build/mozharness/rev/0082f1853222

Comment 15

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0256c351bd95

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0256c351bd95

Comment 17

[Michael Shal [:mshal]]

There are now builds in the latest dirs (eg: https://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/ ) and I believe they were submitted to balrog correctly. Can you try and see if it works? If not, please re-open this.

Comment 18

[Jan Manthay]

After downloading from https://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/ Nightly is correct in german language on my device. But after the first udate it switches to english. Can’t say if it is an update or Nightly problem.

Comment 19

[Jan Manthay]

I found a en/us .apk in my download folder, so it looks like the update downloaded this insted of the multi.

Comment 20

[Richard Newman [:rnewman]]

I don't see l10n updates being served:

$ curl "https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150519030204/Android_arm-eabi-gcc3/de/nightly/4.4.3/default/default/41.0a1/update.xml"

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="41.0a1" appVersion="41.0a1" platformVersion="41.0a1" buildID="20150520030205">
        <patch type="complete" URL="http://download.cdn.mozilla.net/pub/mozilla.org/mobile/nightly/2015/05/2015-05-20-03-02-05-mozilla-central-android-api-11-l10n/fennec-41.0a1.de.android-arm.apk" hashFunction="sha512" hashValue="493bfda130dbb86e3dd1237fd7ccb7f0fc4d78b6f4c812592a30ec7c1ce29cf36bd0be7e793363629f42b89b5ff3d62e538799d8edb42ee25a09103d1d16ed25" size="40196302"/>
    </update>
</updates>


Note that 2015-05-20 is shown as the latest l10n build. I do see en-US updates:

$ curl "https://aus4.mozilla.org/update/4/Fennec/41.0a1/20150519030204/Android_arm-eabi-gcc3/en-US/nightly/4.4.3/default/default/41.0a1/update.xml"

<?xml version="1.0"?>
<updates>
    <update type="minor" displayVersion="41.0a1" appVersion="41.0a1" platformVersion="41.0a1" buildID="20150531030228">
        <patch type="complete" URL="http://download.cdn.mozilla.net/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-11/1433066548/en-US/fennec-41.0a1.en-US.android-arm.apk" hashFunction="sha512" hashValue="b5ebd6811cc9bad90df40cf585ed3eec20dd401fdcfa91ad58ba608e789bb59cb7bb2200d98672f5ca816c9f4d971939ef69df79ee0dc4e8d25a47f2c7ee1e81" size="40434258"/>
    </update>
</updates>

and there are .multi. APKs in the parent directory, alongside 'en-US'.

I don't have an update.locale file to hand (which is how Fennec determines what to put in the update URL), so I guessed that "de" is what we use for a German Nightly -- "multi" returns no updates, so I don't know what else it would be.

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Thanks for the debugging. I think there's two issues here.

1, en-US builds are being submitted to the update server instead of multi, eg
09:06:48     INFO -  Balrog request to https://aus4-admin.mozilla.org/api/releases/Fennec-mozilla-central-nightly-20150601075320/builds/Android_arm-eabi-gcc3/en-US
09:06:48     INFO -  Data sent: {'product': u'Fennec', 'hashFunction': u'sha512', 'schema_version': 4, 'alias': 'null', 'copyTo': '["Fennec-mozilla-central-nightly-latest"]', 'version': u'41.0a1', 'data': '{"buildID": "20150601075320", "platformVersion": "41.0a1", "displayVersion": "41.0a1", "appVersion": "41.0a1", "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-11/1433170400/en-US/fennec-41.0a1.en-US.android-arm.apk", "hashValue": "0660b4012b1d115807f1739fc652fa96ff744e04aa858936296755e8dd4eb5d087e234cc4b12ce08264b11d35867bfc0032a069cfa52a34b75757921415ff3b9", "from": "*", "filesize": 40447914}]}'}
(https://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015-06-01-07-53-20-mozilla-central-android-api-11/mozilla-central-android-api-11-nightly-bm73-build1-build3.txt.gz)

Not only is that en-US, but it's the location in tinderbox-builds, when it should be using the multi build from .../mobile/nightly/YYYY/MM/... This is probably from mozharness setting the completeMarUrl property incorrectly, eg not updated when the multi-l10n part happens after en-US. In fact it looks like we don't create taskcluster artifacts after doing make upload for multi (not doing the build step presumably).

2, single locale builds haven't been running since about the 21st. The mozharness build needs to do the equivalent of the trigger which buildbot used to do mozilla-central-android-api-11-l10n.

Comment 23

[Richard Newman [:rnewman]]

Any update on this, mshal or nthomas?

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

mshal is away until next week. Do we need to fix this before then ?

Comment 25

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #24)
> mshal is away until next week. Do we need to fix this before then ?

Not for my issue, since en-US is my language.  BTW is this issue related to the same day the nightly not updating issue was "fixed", bout obviously by offering the wrong build, aurora suddenly stopped auto updating on Android.

Comment 26

[Michael Shal [:mshal]]

Attachment: android-multi-balrog

This is very ugly, but it should address the first issue in #c21. Specifically, during the multi-l10n upload, we override the completeMarUrl and completeMarSize/Hash with the multi apk. This is tricky for a few reasons:

1) The upload of the multi apk is still outside of 'mach build' as a separate 'make upload AB_CD=multi' call, which means it doesn't set any properties in mach_build_properties.json. Instead, I modified the MakeUploadOutputParser to grab the data.

2) Per #c21, we want to use the nightly/YYYY/MM url, which is the first of the package urls in the 'make upload' output. This is likely very fragile.

3) To set the Size/Hash properties, we need the actual file path on the local machine, not the url. And since MakeUploadOutputParser doesn't have access to the query_* functions, there is some extra ugly passing of the 'packageFilename' property back out to the multi_l10n() function.

I'm definitely open to better alternatives, but I think a proper solution will involve moving the multi-l10n bits into the tree with another MOZ_AUTOMATION step or some such. I'd prefer to do that as a followup though so we can get android updates working in the meantime.

I still have no idea on why single locales are broken as well, so I'll have to dig into that separately.

Comment 27

[Michael Shal [:mshal]]

mrrrgn, are you able to look into #c21 part 2 about why single locale builds aren't getting scheduled? The android l10n nightlies still show up in the builder list, but just aren't running anymore. It may have something to do with logic like this in misc.py, but I'm not sure: http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1855

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8617643 [details] [diff] [review]
android-multi-balrog

I'm not very familiar with this code but I'll grant f+ because it looks plausible, with the issues you point out. Fully in favour of maching this though to simplify.

Comment 29

[Jordan Lund (:jlund)]

Comment on attachment 8617643 [details] [diff] [review]
android-multi-balrog

Review of attachment 8617643 [details] [diff] [review]:
-----------------------------------------------------------------

too bad we have to pull MakeUploadOutputParser back in as part of the build process. after the dust settles, ++ to looking into making this upload target in-tree and having the props set correctly in mach_build_properties.json

::: mozharness/mozilla/building/buildbase.py
@@ +129,5 @@
> +                    self.info("Using package as mar file: %s" % m)
> +                    self.matches['completeMarUrl'] = m
> +
> +        if self.use_package_as_marfile:
> +            pat = r'''^Uploading (.*\.(tar\.bz2|dmg|zip|apk|rpm|mar|tar\.gz))$'''

so how does MozillaBuildFactory avoid having to do this?

Comment 30

[Morgan Phillips [:mrrrgn]]

(In reply to Michael Shal [:mshal] from comment #27)
> mrrrgn, are you able to look into #c21 part 2 about why single locale builds
> aren't getting scheduled? The android l10n nightlies still show up in the
> builder list, but just aren't running anymore. It may have something to do
> with logic like this in misc.py, but I'm not sure:
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1855

I'll look at it. It's going to take me time just to get a footing. I've never touched any of this code.

Comment 31

[Michael Shal [:mshal]]

(In reply to Jordan Lund (:jlund) from comment #29)
> ::: mozharness/mozilla/building/buildbase.py
> @@ +129,5 @@
> > +                    self.info("Using package as mar file: %s" % m)
> > +                    self.matches['completeMarUrl'] = m
> > +
> > +        if self.use_package_as_marfile:
> > +            pat = r'''^Uploading (.*\.(tar\.bz2|dmg|zip|apk|rpm|mar|tar\.gz))$'''
> 
> so how does MozillaBuildFactory avoid having to do this?

It should be roughly equivalent to this logic in buildbotcustom: http://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#1712 (the if 'android' block in submitBalrogUpdates()). Does that answer your question? I might be misunderstanding.

Comment 32

[Michael Shal [:mshal]]

Comment on attachment 8617643 [details] [diff] [review]
android-multi-balrog

https://hg.mozilla.org/build/mozharness/rev/e5aff0bd57e8

Comment 33

[Teodora Vermesan (:TeoVermesan)]

Attachment: Screenshot_2015-06-15-11-30-29(1).png

I have installed the latest Nightly build from: ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-11/
and the version and date are missing

Comment 34

[Teodora Vermesan (:TeoVermesan)]

Also, I've installed the 06-06 nightly build and tried to update to the latest:
- "Downloading update" message appears in about:firefox
- Tapping "Install update"  does nothing, "Check for Updates" message appears after that (the same for the "Update available for Nightly" notification)
- Using a 3rd Party File Manager (Astro Files) go to the "Downloads" directory and tap the apk. Installation is done and in about:firefox the version and date are missing as in the screenshot from comment 33.

Comment 35

[Bill Gianopoulos [:WG9s]]

(In reply to Teodora Vermesan (:TeoVermesan) from comment #33)
> Created attachment 8622352 [details]
> Screenshot_2015-06-15-11-30-29(1).png
> 
> I have installed the latest Nightly build from:
> ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-api-
> 11/
> and the version and date are missing

I filed bug 1174506 on this issue.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

The update issue is bug 1174581, updates are disabled on api-11 builds right now.

Comment 37

[Michael Shal [:mshal]]

Attachment: bbcustom-android-scheduler.patch

Are you familiar with the scheduling? The android single-locale l10n builders are still created, but they're no longer triggered because the triggeredSchedulers for is_mobile_l10n=True (http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1904) are only used in the NightlyBuildFactory, not the mozharness factory. This patch adds the is_mobile_l10n logic to generateDesktopMozharnessBuilders, which might be all that's necessary. In staging it seems to correctly trigger the single locale builds after a nightly, at least.

Comment 38

[Jordan Lund (:jlund)]

Comment on attachment 8623449 [details] [diff] [review]
bbcustom-android-scheduler.patch

Review of attachment 8623449 [details] [diff] [review]:
-----------------------------------------------------------------

so I guess this will work since all we need is to match the scheduler name defined here: http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1904

We only need the scheduler name because once we hit lines 1855-1904, they will take care of creating the mobile l10n builders associated with the scheduler regardless if we are using android mh or android bbot-only. At least that's how I'm reading it.

::: misc.py
@@ +1027,5 @@
>              # see bug 1150015
>              l10n_builder = l10nNightlyBuilders[base_name]['l10n_builder']
>              assert(isinstance(l10n_builder, str))
>              triggered_nightly_schedulers.append(l10n_builder)
> +        elif pf.get('is_mobile_l10n'):

it looks like the bbot-only logic requires l10n_chunks, enable_l10n, is_mobile_l10n, and enable_triggered_nightly_scheduler http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1855

iiuc, you are only checking for the last two. Is there a reason for that? does this not apply or maybe is overkill here?

Comment 39

[Michael Shal [:mshal]]

(In reply to Jordan Lund (:jlund) from comment #38)
> @@ +1027,5 @@
> >              # see bug 1150015
> >              l10n_builder = l10nNightlyBuilders[base_name]['l10n_builder']
> >              assert(isinstance(l10n_builder, str))
> >              triggered_nightly_schedulers.append(l10n_builder)
> > +        elif pf.get('is_mobile_l10n'):
> 
> it looks like the bbot-only logic requires l10n_chunks, enable_l10n,
> is_mobile_l10n, and enable_triggered_nightly_scheduler
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1855
> 
> iiuc, you are only checking for the last two. Is there a reason for that?
> does this not apply or maybe is overkill here?

Nope, I think being consistent makes sense. I don't think we'd want to try to trigger if the builders aren't created. I'll upload a new patch...

Comment 40

[Michael Shal [:mshal]]

Attachment: bbcustom-android-scheduler.patch

Comment 41

[Michael Shal [:mshal]]

Comment on attachment 8623920 [details] [diff] [review]
bbcustom-android-scheduler.patch

https://hg.mozilla.org/build/buildbotcustom/rev/bd3ef874017b

Comment 42

[Bill Gianopoulos [:WG9s]]

OK, sorry raising this to blocker.  We really need to close the tree unless this lands soon.  We are not getting adequate test coverage by our nightly testers if they are both not getting updates pushed not seeing what day's updates they are running on the about: page.  Expecting Nightly testers to understand changesets and about:buildconfig is asking too much of them in my opinion.

Comment 43

[Michael Shal [:mshal]]

I believe this is fixed. Please reopen or file a new bug if that's not the case.