981777 - Schedule mochitest-browser-chrome on EC2 slaves on all branches

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

In order to get rid of the mac minis (bug 981775) we'll need to get mbc running on EC2 slaves on all branches.

To help facilitate this, we should schedule mbc on EC2 slaves on all our downstream branches (aurora, beta, release, and esr24) so we can run them side-by-side the Fedora runs while we green them up.

Joel, do you think we should schedule them in 3 chunks or just the existing 1 chunk?

Comment 1

[Armen [:armenzg]]

Normally we schedule on trunk and they ride by themselves since we lock to a gecko version.

We can speed the process up if needed, is this what you're requesting in here?

If we schedule it now without fixing them first on trunk they would be running orange.

Comment 2

[Jonathan Griffin (:jgriffin)]

Yes, that's the plan; we'll hide them while they're orange, but we need them running on other branches so we can scope the work.  The work on trunk probably isn't going to be enough, since we have different versions of gecko (and a different set of tests) on those branches.

We don't have time to let this ride the trains, either, so we'll have to do it all simultaneously.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

For opt, single chunk, for debug 3 chunks- this is what we have been testing and it will work better for job run times.

When the tree opens I will land the patch to fix the remaining debug bc issues :)  If we target this for tomorrow on m-c, inbound, etc... we could keep it hidden until we have verified 90%+ green or parity with existing hardware machines and then turn the others off.

Comment 4

[Jonathan Griffin (:jgriffin)]

This bug is explicitly for scheduling the tests on branches with older gecko, but maybe we should make this bug about scheduling the tests on all branches on EC2, side-by-side the Fedora slaves.  That's an easier patch. :)

Comment 5

[Armen [:armenzg]]

It is way easier to schedule for all branches, however, hidding/unhiding for every single branch is tad annoying.

Let me figure out which approach will help.
I will set my first target to help us discover scope on downstream branches.

Comment 6

[Armen [:armenzg]]

Attachment: Simplify our code while we deprecate Fedora jobs + merge day note

(buildbot)armenzg-thinkpad test hg:[default!] $ diff -pU0 old_list_of_builders.txt list_of_builders.txt | grep -v "@@" | sort
+++ list_of_builders.txt	2014-03-11 12:34:06.656256318 -0400
--- old_list_of_builders.txt	2014-03-11 11:38:29.940198960 -0400
+Ubuntu VM 12.04 mozilla-aurora debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 mozilla-beta debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 mozilla-esr24 debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 mozilla-release debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 release-mozilla-beta debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 release-mozilla-esr24 debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 release-mozilla-release debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 mozilla-aurora debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 mozilla-beta debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 mozilla-esr24 debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 mozilla-release debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 release-mozilla-beta debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 release-mozilla-esr24 debug test mochitest-browser-chrome ScriptFactory
+Ubuntu VM 12.04 x64 release-mozilla-release debug test mochitest-browser-chrome ScriptFactory

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8389250 [details] [diff] [review]
Simplify our code while we deprecate Fedora jobs + merge day note

Review of attachment 8389250 [details] [diff] [review]:
-----------------------------------------------------------------

Looks right to me.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8389250 [details] [diff] [review]
Simplify our code while we deprecate Fedora jobs + merge day note

Review of attachment 8389250 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1895,5 @@
>      # Remove the elm exception when we fix b2g reftests
>      # and debug mochitest-browser-chrome bug 837017 and bug 850105
>      if branch == "elm":
>          continue
> +    # MERGE DAY: Remove this loop on 3/17 merge day

You should be using items_before() for something like this. There's examples of how to use it in this file.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 8389250 [details] [diff] [review]
Simplify our code while we deprecate Fedora jobs + merge day note

Review of attachment 8389250 [details] [diff] [review]:
-----------------------------------------------------------------

eh, meh

Comment 10

[Armen [:armenzg]]

Comment on attachment 8389250 [details] [diff] [review]
Simplify our code while we deprecate Fedora jobs + merge day note

I left a block unchecked after further comments on IRC.

Comment 11

[Armen [:armenzg]]

Attachment: Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

To see what our uplift scope is.

f? before r?

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8389390 [details] [diff] [review]
Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

Review of attachment 8389390 [details] [diff] [review]:
-----------------------------------------------------------------

I would like to see a bug where we lower the 'script_maxtime': 7200 to something more in the 4800 range (depending on how these work and how long they take.)

::: mozilla-tests/config.py
@@ +1949,5 @@
>                              except KeyError:
>                                  pass
>  
> +# bug 981177 - schedule debug mochitest_bc_3 on ec2 for release branches
> +for name in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release', 'mozilla-esr24'):

I assume we already have this for trunk branches in another patch?  Also this will likely require a good series of checkins to disable tests in order for these to be green on these branches- we haven't tested chunked browser-chrome on other branches to date.

Comment 13

[Armen [:armenzg]]

We will hide these until we uplift.

I thought we are using bug 982225 for m-i and trunk trees.
We could merge both.

Comment 14

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8389390 [details] [diff] [review]
Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

Review of attachment 8389390 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 15

[Armen [:armenzg]]

Attachment: timeout.diff

The longest job on Cedar took 53 mins.

4200/60=70mins.

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 8389390 [details] [diff] [review]
Enable debug mochitest-browser chrome for Ubuntu {x64} for the release branches

Review of attachment 8389390 [details] [diff] [review]:
-----------------------------------------------------------------

rubberstamp+

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8389396 [details] [diff] [review]
timeout.diff

Review of attachment 8389396 [details] [diff] [review]:
-----------------------------------------------------------------

great, thanks for the work!

Comment 18

[John Hopkins (:jhopkins)]

in production

Comment 19

[Armen [:armenzg]]

I will continue any remaining work on bug 982225.
It is a dupe.