1102283 - figure out firefox 34 release mechanics

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

We will be producing two different sets of binaries for the Firefox 34 release. Firefox 34.0 will be given to all existing users, and Firefox 34.1 will be served from the website. We also need the ability to ship point releases to both of these sets of users. Ie:
* 34.0, 33.1 and earlier -> 34.0.1
* 34.1 -> 34.1.1

We will eventually merge the streams back together.

A bunch of us talked yesterday and came up with a few options:
1) Symlink mozilla-release to mozilla-release-34.1 on hg.mozilla.org, and create "mozilla-release-34.1" release configs. This was shot down because it would give us links to https://hg.mozilla.org/releases/mozilla-release-34.1 in the product.
2) Do 34.0 and 34.1 off of the "mozilla-release" release config. This was shot down because we couldn't build 34.0 and 34.1 simultaneously with this.
3) Create a fake "mozilla-release-34.1" branch in buildbot-configs/release configs that still ponits at "mozilla-release" on hg.m.o. This is the option we're pursuing at the moment. It should let us do 34.0 and 34.1 simultaneously without causing any in product differences to hg links.

nthomas started work on this in https://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs-34.1/rev/b78f0d6b1e55 and i've been continuing it in https://hg.mozilla.org/users/bhearsum_mozilla.com/buildbot-configs/. We'd like to run staging releases to at least en-US build completion, but running them to full completion would be ideal.

Comment 1

[[e-mail] (:bhearsum)]

I created a story in pivotal for this, I don't know how to link to it though. It's id is 83129450

Comment 2

[[e-mail] (:bhearsum)]

After a lot of mucking around, we've got a staging release going on http://dev-master1.srv.releng.scl3.mozilla.com:8018. There's changes we'll need in my buildbot-configs, buildbotcustom, and tools repositories. There will probably be more still, too, so I'm not posting anything yet.

Comment 3

[Rail Aliiev [:rail]]

Attachment: aus.diff

In case we wanted to go with throttling per locale, attached a proof-of-concept patch (NOT TESTED!).

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8526185 [details] [diff] [review]
aus.diff

@rhelmer, do you think this approach would work to have per locale throttling?

Comment 5

[[e-mail] (:bhearsum)]

Current status: I think we're through the vast majority of the buildbot changes. en-US builds for 34.0 are running on my dev-master (http://dev-master1.srv.releng.scl3.mozilla.com:8018). Hopefully that release will go smoothly from here on out. Once it's done (or done as far as we care to run it), 34.1 needs to be started through http://dev-master1.srv.releng.scl3.mozilla.com:5000/releases.html and my release runner instance. I might be the only person capable of doing that, because it requires my ssh key...

Comment 6

[Rail Aliiev [:rail]]

Attachment: aus.diff

syntax fix

Comment 7

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #6)
> Created attachment 8526320 [details] [diff] [review]
> aus.diff
> 
> syntax fix

I tested the patch on aus3.allizom.org using the following command:

while true; do curl ${URL}?${RANDOM}; sleep 0.5; done

using the following URLs:

# 3% expected, I managed to catch some update responses for this one
https://aus3.allizom.org/update/3/Firefox/32.0.3/20140923175406/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2014.0.0/default/default/update.xml

# 100% expected, got 100%
https://aus3.allizom.org/update/3/Firefox/32.0.3/20140923175406/Darwin_x86_64-gcc3-u-i386-x86_64/pt-BR/release/Darwin%2014.0.0/default/default/update.xml

# 100% expected, got 100%
https://aus3.allizom.org/update/3/Firefox/32.0.3/20140923175406/Darwin_x86_64-gcc3-u-i386-x86_64/en-GB/release/Darwin%2014.0.0/default/default/update.xml

# 100% expected, got 100%
https://aus3.allizom.org/update/3/Firefox/32.0.3/20140923175406/Darwin_x86_64-gcc3-u-i386-x86_64/af/release/Darwin%2014.0.0/default/default/update.xml

The following 3 URLs are set to 0% in $productThrottling, and en-US is set to 3% in $localeThrottling, but I'm getting 0% for all locales...

https://aus3.allizom.org/update/3/Firefox/33.1/20141030112145/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2014.0.0/default/default/update.xml
https://aus3.allizom.org/update/3/Firefox/33.1/20141030112145/Darwin_x86_64-gcc3-u-i386-x86_64/pt-BR/release/Darwin%2014.0.0/default/default/update.xml
https://aus3.allizom.org/update/3/Firefox/33.1/20141030112145/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2014.0.0/default/default/update.xml

Is there any way to get to the box to play around the code/logs to debug this?

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Rail Aliiev [:rail] from comment #7)
> Is there any way to get to the box to play around the code/logs to debug
> this?

jakem might be able to help out here.

Comment 9

[Rail Aliiev [:rail]]

Attachment: tools-l10n-repack-should-use-path.diff

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8526839 [details] [diff] [review]
tools-l10n-repack-should-use-path.diff

https://hg.mozilla.org/build/tools/rev/358aded9c73f

Comment 11

[[e-mail] (:bhearsum)]

Attachment: release runner hack + staging release fix

Here's the rest of the tools changes we need for 34. The first hunk is just a fix for staging releases now that release downloader is dead. The second part is adding hacks to release runner to make sure we choose the right config file for certain versions of 34.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: shortner more stuff

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8526853 [details] [diff] [review]
release runner hack + staging release fix

Review of attachment 8526853 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildbot-helpers/release_sanity.py
@@ +439,4 @@
>                  # verify that l10n changesets match the shipped locales
>                  if releaseConfig.get('shippedLocalesPath'):
>                      sr = releaseConfig['sourceRepositories'][source_repo]
> +                    sourceRepoPath = sr.get('path', sr['path'])

can you use sr["path"] instead sr.get(...). They are equivalent in this case.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: buildbot-configs changes

Surprisingly little here:
* Add fake mozilla-release-34.1 branch + release configs
* Add mozconfig for source builder

That's it!

Comment 15

[Rail Aliiev [:rail]]

Attachment: 34.1-tools-2.diff

A reconfig required for this.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 8526879 [details] [diff] [review]
34.1-tools-2.diff

Review of attachment 8526879 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/util/fabric/actions.py
@@ +234,5 @@
> +        run('ln -s %(bbconfigs_dir)s/mozilla/release-firefox-mozilla-release-34.1.py '
> +            '%(master_dir)s/' % master)
> +        run('ln -s %(bbconfigs_dir)s/mozilla/l10n-changesets_mozilla-release-34.1 '
> +            '%(master_dir)s/' % master)
> +    print OK, "Added esr34.1 symlinks in %(hostname)s:%(basedir)s" % master

s/esr34.1/something else when you land, please.

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8526879 [details] [diff] [review]
34.1-tools-2.diff

https://hg.mozilla.org/build/tools/rev/cd0dea8f5527

The symlinks should be in place now.

Comment 18

[[e-mail] (:bhearsum)]

Attachment: fixes for buildbot-configs

* Add silly required variables
* Strip out mozilla-release-34.1 schedulers/builders, to make sure we don't accidentally double build stuff.

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8526917 [details] [diff] [review]
fixes for buildbot-configs

This is my favourite! :D

Comment 20

[[e-mail] (:bhearsum)]

Attachment: remove verify configs to stop update automation for 34.0.5

Rail and I talked about this and decided that not running any update automation for 34.0.5 would be the safest option. We'll need to revert this (and remove the skip_updates line) if/when we do a 34.0.6. We'll also need patcher configs and update verify configs at that point.

Comment 21

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/c31b3686e3f7

Comment 22

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b8a3e07b35c6

Comment 23

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/c7866b0ffaa8

Comment 24

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/1674ff687951

Comment 25

[Rail Aliiev [:rail]]

Attachment: aus.diff

Bah, I was almost asleep when I realized where I made a logical mistake. The original patch was applying product based throttling when the first if doesn't meat the conditions. Basically it was multiplying the probabilities.

jakem, if getting access to the aus3 staging box is problematic, can you blow away the local changes and apply this patch?

Comment 26

[[e-mail] (:bhearsum)]

Attachment: fix up 34.1 schedulers again

Catlee noticed that my previous attempt to remove 34.1 CI schedulers/change sources accidentally removed all of the release schedulers. This is due to a bad copy paste that worked fine in builder_master.cfg, but doesn't in scheduler_master.cfg due to differing placement of the ENABLE_RELEASE block. This patch re-adds all of the 34.1 schedulers.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 8527685 [details] [diff] [review]
fix up 34.1 schedulers again

catlee r+'ed this over irc. I landed on default+production, because there's unlikely to be a reconfig before the first release build start.

Comment 28

[Rail Aliiev [:rail]]

I filed bug 1104319 to get access to the box.

Comment 29

[Jake Maul [:jakem]]

Are these patches (for AUS) cumulative, or should they be applied one at a time? Comment 25 onward.

Comment 30

[Rail Aliiev [:rail]]

The patch is https://bugzilla.mozilla.org/attachment.cgi?id=8527142 and it should be applied on top of the clean checkout.

Comment 31

[Jake Maul [:jakem]]

https://bugzilla.mozilla.org/attachment.cgi?id=8527142 applied to clean checkout, no errors. Deployed to aus2.allizom.org / aus3.allizom.org (same).

Comment 32

[Rail Aliiev [:rail]]

Hmm, it doesn't look it's working. The patch should disable 33.0.3 updates, but https://aus3.allizom.org/update/3/Firefox/33.0.3/20141105223254/Darwin_x86_64-gcc3-u-i386-x86_64/de/release/Darwin%2014.0.0/default/default/update.xml gives me an update... Probably it's better to wait for bug 1104319 to shorten iterations.

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 8527142 [details] [diff] [review]
aus.diff

OK, I figured out the issue. It turns out that config.php wasn't pointing to config-dist.php.

I played around with locale and product throttling and it looks like this patch should work.

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8527142 [details] [diff] [review]
aus.diff

>Index: inc/config-dist.php
>===================================================================
>+// This defines explicit throttling levels per locale.It overrides global and 
>+// product trottling only for the specified locales.

Nits, space in 'locale.It', s/trottling/throttling/.

I checked the test suite, it's still fine with this patch. Not terribly surprising since it doesn't have any throttling tests, so I take back my query on IRC about adding tests.

Comment 35

[Robert Helmer [:rhelmer]]

Comment on attachment 8527142 [details] [diff] [review]
aus.diff

Review of attachment 8527142 [details] [diff] [review]:
-----------------------------------------------------------------

A few nits, you can take or leave the suggestion about assigning that hard-to-read index into the $localeThrottling array to a variable to avoid duplication (I realize this code is deprecated)

::: index.php
@@ +112,5 @@
>      $aus = new AUS_Object();
>  
>      // Check explicit throttling.
> +    if ( !$aus->isThrottleException($clean['version'], $clean['channel']) ) {
> +        // check if locale based throttling is set. Do not use product based throttling if set

good comment

@@ +113,5 @@
>  
>      // Check explicit throttling.
> +    if ( !$aus->isThrottleException($clean['version'], $clean['channel']) ) {
> +        // check if locale based throttling is set. Do not use product based throttling if set
> +        if ( isset($localeThrottling[$clean['product']][$clean['version']][$clean['locale']]) ) {

You could assign $localeThrottling[$clean['product']][$clean['version']][$clean['locale']] to a variable and then do isset() and the mt_rand comparison against that instead.

I do see that it is (unfortunately) following the style of the existing code, so it probably isn't worth worrying about the duplication of this bit below - this is pretty hard to read and I do worry about one getting changed and not the other, if there's ever a patch to this.

For what it's worth I've tested and played with this locally, and it seems to do the right thing.

@@ +125,2 @@
>      // Check global throttling.
>      } elseif ( defined('THROTTLE_GLOBAL') && THROTTLE_GLOBAL && 

stray space at end of line

::: inc/config-dist.php
@@ +74,5 @@
>  
>  // Turns logging throttled hits on and off.
>  define('THROTTLE_LOGGING',false);
>  
> +// This defines explicit throttling levels per locale.It overrides global and 

stray space at end of line

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

The sendchange for 34.0.5 triggered 34.0 jobs again, reset schedulers, tagging etc. The latter got cancelled.

Comment 37

[Rail Aliiev [:rail]]

Attachment: baseTag-buildbotcustom.diff

This should fix the baseTag issue. I can't remember why we use baseTag there, maybe to match _BUILDN suffix as well?

Comment 38

[Rail Aliiev [:rail]]

Comment on attachment 8527142 [details] [diff] [review]
aus.diff

I landed the patch without any throttling changes in the config (the lines are commented)

$ cvs commit -m "Bug 1102283 - Add suport for locale based throttling. r=nthomas,rhelmer"
Checking in index.php;
/cvsroot/mozilla/webtools/aus/xml/index.php,v  <--  index.php
new revision: 1.41; previous revision: 1.40
done
Checking in inc/config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.324; previous revision: 1.323
done

$ cvs tag -F AUS2_PRODUCTION                                                               
T index.php
T inc/config-dist.php

Comment 39

[[e-mail] (:bhearsum)]

Comment on attachment 8529121 [details] [diff] [review]
baseTag-buildbotcustom.diff

Review of attachment 8529121 [details] [diff] [review]:
-----------------------------------------------------------------

I don't see anything in bug 841550 that talks about required baseTag specifically, though it does seem to suggest that base tag ends with a "_", which is obviously not the case. This seems correct, but landing this before we ship on Monday scares me. I suppose we really don't have a choice though, we need to fix this in case of a rebuild?

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8529121 [details] [diff] [review]
baseTag-buildbotcustom.diff

I'm going to get this in ahead of Fennec 34.0 build3, as I'd prefer it to break early rather than in a chemspill mode.

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8529121 [details] [diff] [review]
baseTag-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/916e55951a41
https://hg.mozilla.org/build/buildbotcustom/rev/bdd8e9617403

Comment 42

[[e-mail] (:bhearsum)]

This is loooooong since fixed.