1083853 - Enable nightly builds w/ updates on Alder branch

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

We're going to need them Very Soon.

Comment 1

[Rail Aliiev [:rail]]

Attachment: alder-nightly-buildbot-configs.diff

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8506197 [details] [diff] [review]
alder-nightly-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/8bbdafbad6e4

Comment 3

[Rail Aliiev [:rail]]

Attachment: alder-nightly-buildbot-configs-4.diff

* fix the issue with alder being in tb project branches
* kill esr24
* kill not used allLocalesFile

Comment 4

[Rail Aliiev [:rail]]

Attachment: alder-nightly-tools.diff

Kill esr24

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8506264 [details] [diff] [review]
alder-nightly-buildbot-configs-4.diff

Review of attachment 8506264 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/project_branches.py
@@ +99,5 @@
> +        'enable_l10n_onchange': False,
> +        'l10nNightlyUpdate': True,
> +        'l10n_tree': 'fxrel',
> +        'l10n_platforms': ['linux', 'linux64', 'win32', 'macosx64'],
> +        # explicitly set the server to avoid ising variables

s/ising/using/

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8506265 [details] [diff] [review]
alder-nightly-tools.diff

https://hg.mozilla.org/build/tools/rev/7ef457e8c2a2

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8506264 [details] [diff] [review]
alder-nightly-buildbot-configs-4.diff

https://hg.mozilla.org/build/buildbot-configs/rev/d2ef7bf6b94e

Comment 8

[Axel Hecht [:Pike]]

About l10n builds for alder:

It'd be nice to get builds for all locales off of the 33 release tag/branch on the l10n release repos.

In case that doesn't work, we should use a reduced list on the default branch of the l10n release repos, for a reduced list of languages for which we have a good state on default.

That's the list with the privacy button:

ast, da, de, en-GB, es-AR, es-CL, es-ES, es-MX, fi, fr, fy-NL, he, hu, it, ja, ja-JP-mac, ko, lv, nb-NO, nn-NO, pa-IN, pl, pt-BR, rm, ru, sk, sl, zh-TW

plus some to test that the privacy buttons switches off if the localization doesn't have it. Let's use 

ach, af

for that, we should be good with spot-testing that, and those two have their sign-off on the latest revision.

Comment 9

[Rail Aliiev [:rail]]

(In reply to Axel Hecht [:Pike] from comment #8)
> About l10n builds for alder:
> 
> It'd be nice to get builds for all locales off of the 33 release tag/branch
> on the l10n release repos.

That's bug 1083893, I believe.

Comment 10

[Rail Aliiev [:rail]]

This is in production now.

When we have a nightly build and submitted balrog data, we'll need to add a new mapping for this channel.

* Go to https://aus4-admin.mozilla.org/rules.html
* At the top add a new rule with the following values set:
 Mapping: Firefox-alder-nightly-latest
 Background Rate: 100
 Priority: 90
 Channel: nightly-alder

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #10)
> This is in production now.
> 
> When we have a nightly build and submitted balrog data, we'll need to add a
> new mapping for this channel.
> 
> * Go to https://aus4-admin.mozilla.org/rules.html
> * At the top add a new rule with the following values set:
>  Mapping: Firefox-alder-nightly-latest
>  Background Rate: 100
>  Priority: 90
>  Channel: nightly-alder

A quick sanity check is to look at https://aus4.mozilla.org/update/3/Firefox/15.0a1/20120519040218/WINNT_x86-msvc/en-US/nightly-alder/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml?force=1. This returns an empty update xml right now (because we have no data or rule). Once the rule is set-up, it should contain a proper update response.

Comment 12

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 13

[Jordan Lund (:jlund)]

callek pointed out that alder is currently using mozharness mach builds (currently about to only start riding trains)[1]. IIUC, alder is supposed to mirror the 33 release build[2]

to make alder behave like that release branch and not pick up any special configurations like[1], we should do something like:

diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
index 0add8f4..92111c3 100644
--- a/mozilla/project_branches.py
+++ b/mozilla/project_branches.py
@@ -88,6 +88,7 @@ PROJECT_BRANCHES = {
         },
     },
     'alder': {
+        'gecko_version': 33, # or whatever
         # Make every checkin trigger builds, remove after 33.1 (or sooner)
         'enable_perproduct_builds': False,
         'enable_nightly': True,


to remove only mozharness mach from the picture (we probably should do this at the very least), we could:

diff --git a/mozilla/config.py b/mozilla/config.py
index c0ad5eb..a8ce678 100644
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -2536,7 +2536,7 @@ for name, branch in BRANCHES.items():

 # enable mozharness desktop builds across all twigs
 for name, branch in items_at_least(BRANCHES, 'gecko_version', mc_gecko_version):
-    if name in ('mozilla-central', 'mozilla-inbound', 'b2g-inbound', 'try'):
+    if name in ('mozilla-central', 'mozilla-inbound', 'b2g-inbound', 'try', 'alder'):
         # only enable on twigs for now
         continue
     # if true, any platform with mozharness_desktop_build in its config

we should at the very least stop doing mozharnsess mach builds due to it being new and this release testing being important. e.g. since enabling twigs yesterday, nightlies with mach mozharness already have an issue with partials: https://bugzilla.mozilla.org/show_bug.cgi?id=1055918#c10

[1] http://hg.mozilla.org/build/buildbot-configs/file/7121cb89b6df/mozilla/config.py#l2538
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1080354#c7

Comment 14

[Rail Aliiev [:rail]]

Can someone address jlund's comments today (I won't be around today).

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Jordan Lund (:jlund) from comment #13)
> callek pointed out that alder is currently using mozharness mach builds
> (currently about to only start riding trains)[1]. IIUC, alder is supposed to
> mirror the 33 release build[2]
> 
> to make alder behave like that release branch and not pick up any special
> configurations like[1], we should do something like:
> 
> diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
> index 0add8f4..92111c3 100644
> --- a/mozilla/project_branches.py
> +++ b/mozilla/project_branches.py
> @@ -88,6 +88,7 @@ PROJECT_BRANCHES = {
>          },
>      },
>      'alder': {
> +        'gecko_version': 33, # or whatever
>          # Make every checkin trigger builds, remove after 33.1 (or sooner)
>          'enable_perproduct_builds': False,
>          'enable_nightly': True,
> 

This sounds like the best thing to do - we want the lowest risk possible here.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: make alder gecko 33 based

Comment 17

[[e-mail] (:bhearsum)]

Some of the nightlies failed. I'm going to assume that's because something is broken with mozharness+mach. I'll rekick the failed ones after we get mach disabled.

(In reply to Rail Aliiev [:rail] from comment #10)
> This is in production now.
> 
> When we have a nightly build and submitted balrog data, we'll need to add a
> new mapping for this channel.
> 
> * Go to https://aus4-admin.mozilla.org/rules.html
> * At the top add a new rule with the following values set:
>  Mapping: Firefox-alder-nightly-latest
>  Background Rate: 100
>  Priority: 90
>  Channel: nightly-alder

I created this rule and Linux (the only successful nightly so far) is getting an update. Those builds can be downloaded from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-alder/, and the other platforms will show up there as they become ready.

Comment 18

[[e-mail] (:bhearsum)]

Side question: Android builds are enabled. Do we need them? I don't recall explicitly talking about this.

Comment 19

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/2d044474eced

Comment 20

[[e-mail] (:bhearsum)]

With the patch in production I triggered new Mac and Windows nigthtlies. Android ones also failed, but I'm not going to waste resources retriggering them until I know for sure that they're needed.

Comment 21

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #20)
> With the patch in production I triggered new Mac and Windows nigthtlies.
> Android ones also failed, but I'm not going to waste resources retriggering
> them until I know for sure that they're needed.

I was told on Vidyo that there's at least one Android change, so I went ahead and retriggered these nightlies. They'll show up in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android/ and http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android-x86/ when ready.

Comment 22

[Justin Wood (:Callek)]

(In reply to Ben Hearsum [:bhearsum] from comment #21)
> (In reply to Ben Hearsum [:bhearsum] from comment #20)
> > With the patch in production I triggered new Mac and Windows nigthtlies.
> > Android ones also failed, but I'm not going to waste resources retriggering
> > them until I know for sure that they're needed.
> 
> I was told on Vidyo that there's at least one Android change, so I went
> ahead and retriggered these nightlies. They'll show up in
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android/
> and
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android-
> x86/ when ready.

[12:57:29]	Callek	dolske: ooo btw, did we want android builds on alder, currently they are enabled but its unclear to me if they should be
[12:58:19]	dolske	Callek: no, we can leave those off. I check with the Android team, and they're not shipping anything that needs this branch.

And I was told in private by dolske that the "one android change" does not change his answer above. So I'll turn android off.

Comment 23

[Justin Dolske [:Dolske]]

I checked with the Android team earlier today, the only Android change for 33.1 won't be landing on alder (same thing as bug 1061736, which will land directly on mozilla-release later).

Comment 24

[Justin Wood (:Callek)]

Attachment: [configs] bump alder prior and drop android

Per IRC, this bumps alder prior (to just below -release but higher than beta for now)

And drops android jobs from alder as well.

Comment 25

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5515d8b06d73

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8507048 [details] [diff] [review]
[configs] bump alder prior and drop android

https://hg.mozilla.org/build/buildbot-configs/rev/5515d8b06d73

Comment 27

[cmtalbert]

I've been testing tonight, and updates between the alder nightlies don't seem to be working. Is that expected since we're still working here? I'm not sure how "automatic" updates are once we have nightly builds enabled. For instance I have the build from https://hg.mozilla.org/projects/alder/rev/506dbb9cb1c8. And it won't update to either of the two nightlies that have since been produced.

Comment 28

[Rail Aliiev [:rail]]

Hmm, this is an aus4 (the new update server) vs aus3 issue.
https://aus4.mozilla.org/update/3/Firefox/15.0a1/20120519040218/WINNT_x86-msvc/en-US/nightly-alder/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml?force=1 does serve updates, but https://aus3.mozilla.org/update/3/Firefox/15.0a1/20120519040218/WINNT_x86-msvc/en-US/nightly-alder/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml?force=1 doesn't.

Let me dig into this a bit.

Comment 29

[Rail Aliiev [:rail]]

Attachment: cvs-aus3-alder.diff

Of course, aus3 requires these mappings too.

Comment 30

[Rail Aliiev [:rail]]

Comment on attachment 8507870 [details] [diff] [review]
cvs-aus3-alder.diff

$ cvs commit -m "Bug 1083853 - Enable nightly builds w/ updates on Alder branch. r=bhearsum"
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.315; previous revision: 1.314
done

$ cvs tag -F AUS2_PRODUCTION
T config-dist.php

Comment 31

[Rail Aliiev [:rail]]

We should get updates now \o/

Comment 32

[Chris AtLee [:catlee]]

we need to fix http://hg.mozilla.org/projects/alder/file/default/toolkit/mozapps/update/updater/moz.build#l83 to accept mars signed with the nightly keys.

Comment 33

[Chris AtLee [:catlee]]

Attachment: alder should use nightly keys

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 8508813 [details] [diff] [review]
alder should use nightly keys

Sounds good. We might want to delete the existing nightly builds - they won't be able to update to this (they embed dep keys, but we've been and will continue to sign with nightly ones).

Comment 35

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/a752028423fd

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Mac updates for 64bit were not being offered because we were missing a symlink on the update server. Linux and Windows were fine.