Closed Bug 1083853 Opened 5 years ago Closed 5 years ago

Enable nightly builds w/ updates on Alder branch

Categories

(Release Engineering :: General, defect, P2, major)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: rail)

References

Details

Attachments

(7 files)

We're going to need them Very Soon.
Assignee: nobody → rail
Attachment #8506197 - Flags: review?(catlee) → review+
Depends on: 1083893
* fix the issue with alder being in tb project branches
* kill esr24
* kill not used allLocalesFile
Attachment #8506264 - Flags: review?(catlee)
Kill esr24
Attachment #8506265 - Flags: review?(catlee)
Attachment #8506265 - Flags: review?(catlee) → review+
Blocks: 1062777
Comment on attachment 8506264 [details] [diff] [review]
alder-nightly-buildbot-configs-4.diff

Review of attachment 8506264 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/project_branches.py
@@ +99,5 @@
> +        'enable_l10n_onchange': False,
> +        'l10nNightlyUpdate': True,
> +        'l10n_tree': 'fxrel',
> +        'l10n_platforms': ['linux', 'linux64', 'win32', 'macosx64'],
> +        # explicitly set the server to avoid ising variables

s/ising/using/
Attachment #8506264 - Flags: review?(catlee) → review+
About l10n builds for alder:

It'd be nice to get builds for all locales off of the 33 release tag/branch on the l10n release repos.

In case that doesn't work, we should use a reduced list on the default branch of the l10n release repos, for a reduced list of languages for which we have a good state on default.

That's the list with the privacy button:

ast, da, de, en-GB, es-AR, es-CL, es-ES, es-MX, fi, fr, fy-NL, he, hu, it, ja, ja-JP-mac, ko, lv, nb-NO, nn-NO, pa-IN, pl, pt-BR, rm, ru, sk, sl, zh-TW

plus some to test that the privacy buttons switches off if the localization doesn't have it. Let's use 

ach, af

for that, we should be good with spot-testing that, and those two have their sign-off on the latest revision.
(In reply to Axel Hecht [:Pike] from comment #8)
> About l10n builds for alder:
> 
> It'd be nice to get builds for all locales off of the 33 release tag/branch
> on the l10n release repos.

That's bug 1083893, I believe.
This is in production now.

When we have a nightly build and submitted balrog data, we'll need to add a new mapping for this channel.

* Go to https://aus4-admin.mozilla.org/rules.html
* At the top add a new rule with the following values set:
 Mapping: Firefox-alder-nightly-latest
 Background Rate: 100
 Priority: 90
 Channel: nightly-alder
(In reply to Rail Aliiev [:rail] from comment #10)
> This is in production now.
> 
> When we have a nightly build and submitted balrog data, we'll need to add a
> new mapping for this channel.
> 
> * Go to https://aus4-admin.mozilla.org/rules.html
> * At the top add a new rule with the following values set:
>  Mapping: Firefox-alder-nightly-latest
>  Background Rate: 100
>  Priority: 90
>  Channel: nightly-alder

A quick sanity check is to look at https://aus4.mozilla.org/update/3/Firefox/15.0a1/20120519040218/WINNT_x86-msvc/en-US/nightly-alder/Windows_NT%206.1.1.0%20%28x64%29/default/default/update.xml?force=1. This returns an empty update xml right now (because we have no data or rule). Once the rule is set-up, it should contain a proper update response.
Merged to production, and deployed.
callek pointed out that alder is currently using mozharness mach builds (currently about to only start riding trains)[1]. IIUC, alder is supposed to mirror the 33 release build[2]

to make alder behave like that release branch and not pick up any special configurations like[1], we should do something like:

diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
index 0add8f4..92111c3 100644
--- a/mozilla/project_branches.py
+++ b/mozilla/project_branches.py
@@ -88,6 +88,7 @@ PROJECT_BRANCHES = {
         },
     },
     'alder': {
+        'gecko_version': 33, # or whatever
         # Make every checkin trigger builds, remove after 33.1 (or sooner)
         'enable_perproduct_builds': False,
         'enable_nightly': True,


to remove only mozharness mach from the picture (we probably should do this at the very least), we could:

diff --git a/mozilla/config.py b/mozilla/config.py
index c0ad5eb..a8ce678 100644
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -2536,7 +2536,7 @@ for name, branch in BRANCHES.items():

 # enable mozharness desktop builds across all twigs
 for name, branch in items_at_least(BRANCHES, 'gecko_version', mc_gecko_version):
-    if name in ('mozilla-central', 'mozilla-inbound', 'b2g-inbound', 'try'):
+    if name in ('mozilla-central', 'mozilla-inbound', 'b2g-inbound', 'try', 'alder'):
         # only enable on twigs for now
         continue
     # if true, any platform with mozharness_desktop_build in its config

we should at the very least stop doing mozharnsess mach builds due to it being new and this release testing being important. e.g. since enabling twigs yesterday, nightlies with mach mozharness already have an issue with partials: https://bugzilla.mozilla.org/show_bug.cgi?id=1055918#c10

[1] http://hg.mozilla.org/build/buildbot-configs/file/7121cb89b6df/mozilla/config.py#l2538
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1080354#c7
Can someone address jlund's comments today (I won't be around today).
Flags: needinfo?(jlund)
Flags: needinfo?(bugspam.Callek)
(In reply to Jordan Lund (:jlund) from comment #13)
> callek pointed out that alder is currently using mozharness mach builds
> (currently about to only start riding trains)[1]. IIUC, alder is supposed to
> mirror the 33 release build[2]
> 
> to make alder behave like that release branch and not pick up any special
> configurations like[1], we should do something like:
> 
> diff --git a/mozilla/project_branches.py b/mozilla/project_branches.py
> index 0add8f4..92111c3 100644
> --- a/mozilla/project_branches.py
> +++ b/mozilla/project_branches.py
> @@ -88,6 +88,7 @@ PROJECT_BRANCHES = {
>          },
>      },
>      'alder': {
> +        'gecko_version': 33, # or whatever
>          # Make every checkin trigger builds, remove after 33.1 (or sooner)
>          'enable_perproduct_builds': False,
>          'enable_nightly': True,
> 

This sounds like the best thing to do - we want the lowest risk possible here.
Flags: needinfo?(jlund)
Flags: needinfo?(bugspam.Callek)
Attachment #8506906 - Flags: review? → review+
Some of the nightlies failed. I'm going to assume that's because something is broken with mozharness+mach. I'll rekick the failed ones after we get mach disabled.

(In reply to Rail Aliiev [:rail] from comment #10)
> This is in production now.
> 
> When we have a nightly build and submitted balrog data, we'll need to add a
> new mapping for this channel.
> 
> * Go to https://aus4-admin.mozilla.org/rules.html
> * At the top add a new rule with the following values set:
>  Mapping: Firefox-alder-nightly-latest
>  Background Rate: 100
>  Priority: 90
>  Channel: nightly-alder

I created this rule and Linux (the only successful nightly so far) is getting an update. Those builds can be downloaded from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-alder/, and the other platforms will show up there as they become ready.
Side question: Android builds are enabled. Do we need them? I don't recall explicitly talking about this.
Flags: needinfo?(dolske)
With the patch in production I triggered new Mac and Windows nigthtlies. Android ones also failed, but I'm not going to waste resources retriggering them until I know for sure that they're needed.
(In reply to Ben Hearsum [:bhearsum] from comment #20)
> With the patch in production I triggered new Mac and Windows nigthtlies.
> Android ones also failed, but I'm not going to waste resources retriggering
> them until I know for sure that they're needed.

I was told on Vidyo that there's at least one Android change, so I went ahead and retriggered these nightlies. They'll show up in http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android/ and http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android-x86/ when ready.
(In reply to Ben Hearsum [:bhearsum] from comment #21)
> (In reply to Ben Hearsum [:bhearsum] from comment #20)
> > With the patch in production I triggered new Mac and Windows nigthtlies.
> > Android ones also failed, but I'm not going to waste resources retriggering
> > them until I know for sure that they're needed.
> 
> I was told on Vidyo that there's at least one Android change, so I went
> ahead and retriggered these nightlies. They'll show up in
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android/
> and
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-alder-android-
> x86/ when ready.

[12:57:29]	Callek	dolske: ooo btw, did we want android builds on alder, currently they are enabled but its unclear to me if they should be
[12:58:19]	dolske	Callek: no, we can leave those off. I check with the Android team, and they're not shipping anything that needs this branch.

And I was told in private by dolske that the "one android change" does not change his answer above. So I'll turn android off.
Flags: needinfo?(dolske)
I checked with the Android team earlier today, the only Android change for 33.1 won't be landing on alder (same thing as bug 1061736, which will land directly on mozilla-release later).
Per IRC, this bumps alder prior (to just below -release but higher than beta for now)

And drops android jobs from alder as well.
Attachment #8507048 - Flags: review?(bhearsum)
Blocks: 1080354
Attachment #8507048 - Flags: review?(bhearsum) → review+
OS: Linux → All
Hardware: x86_64 → All
I've been testing tonight, and updates between the alder nightlies don't seem to be working. Is that expected since we're still working here? I'm not sure how "automatic" updates are once we have nightly builds enabled. For instance I have the build from https://hg.mozilla.org/projects/alder/rev/506dbb9cb1c8. And it won't update to either of the two nightlies that have since been produced.
Of course, aus3 requires these mappings too.
Attachment #8507870 - Flags: review?(bhearsum)
Attachment #8507870 - Flags: review?(bhearsum) → review+
Comment on attachment 8507870 [details] [diff] [review]
cvs-aus3-alder.diff

$ cvs commit -m "Bug 1083853 - Enable nightly builds w/ updates on Alder branch. r=bhearsum"
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.315; previous revision: 1.314
done

$ cvs tag -F AUS2_PRODUCTION
T config-dist.php
Attachment #8507870 - Flags: checked-in+
Depends on: 1085365
We should get updates now \o/
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
we need to fix http://hg.mozilla.org/projects/alder/file/default/toolkit/mozapps/update/updater/moz.build#l83 to accept mars signed with the nightly keys.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attachment #8508813 - Flags: review?(bhearsum)
Comment on attachment 8508813 [details] [diff] [review]
alder should use nightly keys

Sounds good. We might want to delete the existing nightly builds - they won't be able to update to this (they embed dep keys, but we've been and will continue to sign with nightly ones).
Attachment #8508813 - Flags: review?(bhearsum) → review+
Attachment #8508813 - Flags: checked-in+
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Mac updates for 64bit were not being offered because we were missing a symlink on the update server. Linux and Windows were fine.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.