864866 - [tracker] Move away from the rev3 minis

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Armen [:armenzg]]

This bug will track moving away from the rev3 minis.

What do we currently run?
* Fed32 - talos (bug 863903) and b2g (bug 850105)
* Fed64 - talos (bug 863903)
* XP (bug 770579) - unittests and talos
* Win7 (bug 770578) - unittests and talos

For some things we will also need to ride the trains.

Current machine distributions:
fed32 - 100 machines
fed64 -  39 machines
winxp - 121 machines
win7 -  129 machines

Comment 1

[Armen [:armenzg]]

These 3 bugs tracks disabling rev3 jobs on Fedora, Fedora64, WinXP and Win7 on m-b, m-r and m-esr17.

Comment 2

[Armen [:armenzg]]

For WinXP, Win7, Fed & Fed64:
* we still run desktop tests on https://tbpl.mozilla.org/?tree=Mozilla-Esr17
** once esr17 dies, it will be taken care of
** we're one merge day away (6 weeks)

Comment 3

[Armen [:armenzg]]

I found an easy way to evaluate this visually.

Two Fedora debug jobs
#####################
https://tbpl.mozilla.org/?jobname=Rev3&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=Rev3&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-Beta&jobname=Rev3&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-Esr24&jobname=Rev3&showall=1
https://tbpl-dev.allizom.org/?tree=Mozilla-B2g26-v1.2&jobname=Rev3&showall=1

A bunch in here (all 4 rev3 platforms):
#######################################
https://tbpl.mozilla.org/?tree=Mozilla-Esr17&jobname=Rev3&showall=1&rev=4c3e2273eb19
https://tbpl.allizom.org/?tree=Mozilla-B2g18&showall=1&jobname=Rev3
https://tbpl.mozilla.org/?tree=Mozilla-B2g18-v1.1.0hd&jobname=Rev3&showall=1

Comment 4

[Armen [:armenzg]]

I also found these b2g jobs:
############################
https://tbpl-dev.allizom.org/?tree=Mozilla-B2g26-v1.2&showall=1&jobname=b2g_emulator%20mozilla-b2g26_v1_2%20opt%20test%20reftest
https://tbpl-dev.allizom.org/?tree=Mozilla-B2g26-v1.2&showall=1&jobname=2g_emulator%20mozilla-b2g26_v1_2%20opt%20test%20marionette-webapi

Comment 5

[Armen [:armenzg]]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #4)
> I also found these b2g jobs:
> ############################
...
> https://tbpl-dev.allizom.org/?tree=Mozilla-B2g26-v1.
> 2&showall=1&jobname=2g_emulator%20mozilla-
> b2g26_v1_2%20opt%20test%20marionette-webapi

I believe bug 932988 will take care of this.

Comment 6

[Armen [:armenzg]]

I will help drive this to completion.

Comment 7

[Armen [:armenzg]]

bug 948135 will deal with decommissioning talos-r3-w7 machines. Doing that bug now will make us not have to wait until b2g18 dies in March.

Comment 8

[Armen [:armenzg]]

Attachment: [configs] remove talos-r3-w7

Comment 9

[Armen [:armenzg]]

Attachment: [tools] remove talos-r3-w7 plus some old platforms

Comment 10

[Armen [:armenzg]]

Comment on attachment 8346598 [details] [diff] [review]
[configs] remove talos-r3-w7

https://hg.mozilla.org/build/buildbot-configs/rev/357301baeba1

Comment 11

[Armen [:armenzg]]

Comment on attachment 8346600 [details] [diff] [review]
[tools] remove talos-r3-w7 plus some old platforms

https://hg.mozilla.org/build/tools/rev/86744be92bed

Comment 12

[Armen [:armenzg]]

mysql> select count(*) from slaves where name like 'talos-r3-w7-%';
+----------+
| count(*) |
+----------+
|      133 |
+----------+
1 row in set (0.00 sec)

mysql> delete from slaves where name like 'talos-r3-w7-%';
Query OK, 133 rows affected (0.00 sec)

Comment 13

[Armen [:armenzg]]

Attachment: slave_health.cleanup.diff

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8346735 [details] [diff] [review]
slave_health.cleanup.diff

Review of attachment 8346735 [details] [diff] [review]:
-----------------------------------------------------------------

Landed as part of:

https://hg.mozilla.org/users/coop_mozilla.com/slave_health/rev/70ca2a0e8e5f

Comment 15

[Armen [:armenzg]]

I will start looking into the Fedora machines:
* bug 850101
* bug 818968
* bug 948551

Comment 16

[Armen [:armenzg]]

== Status update ==
We have dealt with xp and win7.
We have to deal with Fed and Fed64.
We will first make sure that we have somewhere the right jobs being run (even if orange).
We need to move to not run on Rev3 machines for *any* trunk tree.
We will then look at release dates and see if they work with scl1 colo moves.

* Bug 850101 - Linux 64-bit *debug* mochitest-browser-chrome
https://tbpl.mozilla.org/?jobname=Rev3&showall=1
https://tbpl.mozilla.org/?tree=Cedar&showall=1&jobname=Ubuntu%20VM%2012.04%20cedar%20debug%20test%20mochitest-browser-chrome

It seems that the A-team is also trying to run it as a split job. I'm not 100% there.

* Bug 850105 - Run b2g emulator unit tests on Ubuntu
https://tbpl.mozilla.org/?showall=1&jobname=b2g_emulator.*reftest
https://tbpl.mozilla.org/?tree=Cedar&showall=1&jobname=b2g_emulator.*reftest

The Cedar jobs should have been running either on the EC2 instances or the in-house machines, however, they're running on Cedar :'(

On another note, the jit tests are running on the rev3 minis as well as the Firefox desktop jobs:
https://tbpl-dev.allizom.org/?tree=Cedar&showall=1&jobname=Rev3

Comment 17

[[e-mail] (:bhearsum)]

Something here is in production

Comment 18

[Armen [:armenzg]]

== Status update ==
We need owners from b2g, a-team or fx devs to drive the following two bugs to completion:

* Bug 850101 - Run desktop mochitests-browser-chrome on Ubuntu
https://bugzilla.mozilla.org/showdependencytree.cgi?id=850101&hide_resolved=1
==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850101#c12

* Bug 818968 - B2G emulator tests should be runnable on AWS
==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850105#c8

Comment 19

[Armen [:armenzg]]

It seems that bug 850101 at least has some action in the dep bugs.

Comment 20

[Armen [:armenzg]]

Back in December we heard that Mike Lee will be helping with the b2g reftests and Felipe Gomes will be helping us with mochitest-browser-chrome.

Comment 21

[Armen [:armenzg]]

== Status update ==
We got owners: fgomes & jrmuizel

They're still working on figuring how to make the trees go green.

* Bug 850101 - Run desktop mochitests-browser-chrome on Ubuntu
https://bugzilla.mozilla.org/showdependencytree.cgi?id=850101&hide_resolved=1
==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850101#c12

* Bug 818968 - B2G emulator tests should be runnable on AWS
==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850105#c8

Comment 22

[Armen [:armenzg]]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #21)
> == Status update ==
> We got owners: fgomes & jrmuizel
> 
> They're still working on figuring how to make the trees go green.
> 
> * Bug 850101 - Run desktop mochitests-browser-chrome on Ubuntu
> https://bugzilla.mozilla.org/showdependencytree.cgi?id=850101&hide_resolved=1
> ==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850101#c12
I've heard nothing new from fgomes.

> 
> * Bug 818968 - B2G emulator tests should be runnable on AWS
> ==> Status https://bugzilla.mozilla.org/show_bug.cgi?id=850105#c8

We got some green test jobs on Elm.
I've got bug 975034 blocking it. I hope to make breakthrough this week.

Comment 23

[Armen [:armenzg]]

The A-team has also filed a tracking bug for this initiative: bug 981775.

Comment 24

[Armen [:armenzg]]

== Status update ==
* B2g reftests on AWS
** https://tbpl.mozilla.org/?tree=Elm&jobname=b2g_emulator_vm%20elm
** only R5 is giving us trouble
** Jeff is to investigate it bug 981856

* Debug mochitest-browser-chrome
** https://tbpl.mozilla.org/?tree=Cedar&jobname=Ubuntu.*mochitest-browser-chrome-
** Splitting into 3 chunks might be the way to go
** We're enabling the job on m-i tomorrow - bug 982225
** The work of fgomes might not be needed

Comment 25

[Armen [:armenzg]]

Tracking bugs in here instead:
* Bug 982225 - Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
* Bug 987892 - Ubuntu debug mochitest-browser-chrome-1 is crashing on mozilla-aurora
* Bug 985650 - Allow certain jobs to run on beefier EC2 instances even if more pricey

Comment 26

[Armen [:armenzg]]

== Status update ==
We're running side-by-side on the minis and on EC2:
* chunked debug mochitest-browser-chrome jobs
** https://tbpl.mozilla.org/?jobname=debug.*mochitest-browser-chrome
** NOTE: They will show up later today
* b2g reftests
** https://tbpl.mozilla.org/?jobname=b2g_emulator.*reftest

To finish this up we need to:
* stop running Fedora jobs
** bug 982225 and bug 985650
* uplift patches where required
** bug 987892

Comment 27

[Armen [:armenzg]]

It seems that the EC2-b2g reftests on 15 chunks is on par to the Fedora on 10 chunks wrt to wall time.
I would like to propose to disable the minis where it applies.

Gecko 31 - https://tbpl.mozilla.org/?jobname=b2g_emulator.*reftest
Gecko 30 - https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=b2g_emulator.*reftest

We have to investigate what is required to get these trees green:
https://tbpl.mozilla.org/?tree=Mozilla-B2g28-v1.3&jobname=b2g_emulator_vm.*reftest&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g26-v1.2&jobname=b2g_emulator_vm.*reftest&showall=1
v1.3t is pretty broken right now and b2g18 doesn't matter anymore.

Comment 28

[Armen [:armenzg]]

Disabling the minis does not make sense until we fix the b2g26 & b2g28 trees since we risk introducing oranges on those branches that the minis would have caught on m-c & m-a.

Comment 29

[Armen [:armenzg]]

Attachment: disable jsreftest on rev3

Can we disable these? Who is working on them?

Comment 30

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8399517 [details] [diff] [review]
disable jsreftest on rev3

Review of attachment 8399517 [details] [diff] [review]:
-----------------------------------------------------------------

No one is working on them, and in any case we have them running on EC2 on cedar, so when we do work on them, we should work on them in that context.

Comment 31

[Armen [:armenzg]]

== Status update ==
We're running side-by-side on the minis and on EC2:
* chunked debug mochitest-browser-chrome jobs
** https://tbpl.mozilla.org/?jobname=debug.*mochitest-browser-chrome
* b2g reftests
** https://tbpl.mozilla.org/?jobname=b2g_emulator.*reftest

We need to:
* bug 987892 - green up m-a, m-b, m-r & esr24 for debug mochitest-browser-chrome
* bug 818968 - land approved patches for b2g18, b2g26 and b2g28
** we would hope those would be sufficient to get them green

Comment 32

[Armen [:armenzg]]

jgriffin, I see some fedora jobs running on b2g18:
https://tbpl.mozilla.org/?tree=Mozilla-B2g18&jobname=b2g_emulator%20mozilla-b2g18
I don't want to have to enable EC2 replacements for that job. It's only sanity reftests.
B2g18 is in EOL and I want to ask relman to give explicit consent to simply disable those jobs.

Does this work for you?

Comment 33

[Armen [:armenzg]]

I also see the marionette-webapi:
https://tbpl.mozilla.org/?tree=Mozilla-B2g18&jobname=marionette-webapi

Comment 34

[Armen [:armenzg]]

and even some Linux desktop tests!
https://tbpl.mozilla.org/?tree=Mozilla-B2g18&jobname=Rev3

I think I had deleted b2g18 from my brain past 3-17.

Comment 35

[Jonathan Griffin (:jgriffin)]

I agree wrt removing reftests from b2g18.  For the others, let's move them to EC2 and then simply disable whatever doesn't work.  We should spend as little time as possible trying to fix this there.

Comment 36

[Armen [:armenzg]]

Attachment: On mozilla-b2g18 and v1_1_0hd (which are EOL): disable Linux desktop tests on minis, run marionette-web-api on EC2 (instead of minis) and run b2g emulator reftest sanity on EC2 (instead of minis)

Comment 37

[Armen [:armenzg]]

Attachment: changes from previous patch

Notice that "b2g_emulator" runs on the minis and "b2g_emulator_vm" on EC2.

Comment 38

[Rail Aliiev [:rail]]

Comment on attachment 8401940 [details] [diff] [review]
On mozilla-b2g18 and v1_1_0hd (which are EOL): disable Linux desktop tests on minis, run marionette-web-api on EC2 (instead of minis) and run b2g emulator reftest sanity on EC2  (instead of minis)

woooo!

Comment 39

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8401941 [details]
changes from previous patch

Seems reasonable for b2g18.

Comment 40

[Armen [:armenzg]]

Comment on attachment 8401940 [details] [diff] [review]
On mozilla-b2g18 and v1_1_0hd (which are EOL): disable Linux desktop tests on minis, run marionette-web-api on EC2 (instead of minis) and run b2g emulator reftest sanity on EC2  (instead of minis)

https://hg.mozilla.org/build/buildbot-configs/rev/ad21288639b2

Comment 41

[Armen [:armenzg]]

== Status update ==
We're running side-by-side on the minis and on EC2:
* chunked debug mochitest-browser-chrome jobs
** green up to m-a
** we need to green up m-b, m-r & m-esr24
* b2g reftests
** RyanVM will be landing this morning one of our last patches needing an uplift

Bug involved:
* bug 987892 - green up m-b, m-r & esr24 for debug mochitest-browser-chrome
* bug 983650 - land approved patches for b2g26 and b2g28

* We need to see these green before switching over:
https://tbpl.mozilla.org/?tree=Mozilla-Beta&jobname=Ubuntu.*debug.*mochitest-browser-chrome&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-Release&jobname=Ubuntu.*debug.*mochitest-browser-chrome&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-Esr24&jobname=Ubuntu.*debug.*mochitest-browser-chrome&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g26-v1.2&jobname=b2g_emulator_vm.*reftest&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g28-v1.3&jobname=b2g_emulator_vm.*reftest&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g28-v1.3t&jobname=b2g_emulator_vm.*reftest&showall=1

Comment 42

[[e-mail] (:bhearsum)]

something is in production

Comment 43

[Armen [:armenzg]]

WRT to b2g reftests, the uplift cleared some oranges but it is not completely green (see bug 818968 for summary).

WRT to debug mochitest-browser-chrome, we got m-b green and need some more work for m-r & m-esr24.

Comment 44

[Armen [:armenzg]]

Attachment: Only add debug mochitest-browser-chrome for Fedora/Fedora64 on m-r

Comment 45

[Armen [:armenzg]]

== Status update ==
We're running side-by-side on the minis and on EC2:
* chunked debug mochitest-browser-chrome jobs
** we have disabled the jobs on the minis up to m-a
** only m-r is orange and we will leave like it (merge day in 3 weeks)
** tomorrow/Monday we will only run the jobs on the minis for m-r
* b2g reftests
** bug 994936 - tomorrow/Monday we will have disabled the jobs on the minis for the trunk trees
** next week we will disable some tests on b2g26/b2g28
** at that point we will disable *all* b2g reftests on the minis

* Last orange jobs:
https://tbpl.mozilla.org/?tree=Mozilla-Release&jobname=Ubuntu.*debug.*mochitest-browser-chrome&showall=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g26-v1.2&jobname=b2g_emulator_vm.*reftest&showall=1&onlyunstarred=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g28-v1.3&jobname=b2g_emulator_vm.*reftest&showall=1&onlyunstarred=1
https://tbpl.mozilla.org/?tree=Mozilla-B2g28-v1.3t&jobname=b2g_emulator_vm.*reftest&showall=1&onlyunstarred=1

Comment 46

[Armen [:armenzg]]

Comment on attachment 8405043 [details] [diff] [review]
Only add debug mochitest-browser-chrome for Fedora/Fedora64 on m-r

Something has landed somewhere which makes this patch unnecessary.

Comment 47

[Armen [:armenzg]]

This is how close we're [1]
* [DONE] debug mochitest-browser-chrome
* b2g reftests failing on b2g26 and b2g28
** we will either disable tests or fix the root cause this week

[1]
MacAir buildbot-configs hg:[default!] $ grep "Rev3" old_builders_list.txt
MacAir buildbot-configs hg:[default!] $ grep "emulator" old_builders_list.txt | grep -v "_vm" | grep "reftest-10"
b2g_emulator mozilla-aurora opt test reftest-10 ScriptFactory
b2g_emulator mozilla-b2g26_v1_2 opt test reftest-10 ScriptFactory
b2g_emulator mozilla-b2g28_v1_3t opt test reftest-10 ScriptFactory
b2g_emulator mozilla-b2g28_v1_3 opt test reftest-10 ScriptFactory

Comment 48

[Armen [:armenzg]]

Attachment: fedora.buildbotcustom.diff

Comment 49

[Armen [:armenzg]]

Attachment: fedora.tools.diff

Comment 50

[Armen [:armenzg]]

Attachment: fedora.diff

Comment 51

[Armen [:armenzg]]

Attachment: fedora.braindump.diff

Comment 52

[Armen [:armenzg]]

Attachment: fedora.slavehealth.diff

Comment 53

[Armen [:armenzg]]

Attachment: fedora.diff

This is a no-op change from the previous patch.
It only differs on removing 30+ lines that I had left inside of a commented out set of code.

Comment 54

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8406924 [details] [diff] [review]
fedora.slavehealth.diff

Review of attachment 8406924 [details] [diff] [review]:
-----------------------------------------------------------------

You should remove the html bits here too:

https://hg.mozilla.org/build/slave_health/file/1ef894f23768/index.html#l136

Comment 55

[Rail Aliiev [:rail]]

Comment on attachment 8406902 [details] [diff] [review]
fedora.tools.diff

as we talked on IRC http://hg.mozilla.org/build/tools/file/0d316748aadd/trychooser/index.html#l158 also needs to be removed.

Comment 56

[Rail Aliiev [:rail]]

Comment on attachment 8406901 [details] [diff] [review]
fedora.buildbotcustom.diff

Can you also update http://hg.mozilla.org/build/buildbotcustom/file/9ced1444fbb2/test/test_try_parser.py? A separate patch WFM since this one passes trial tests.

Comment 57

[Armen [:armenzg]]

Attachment: fedora.puppet.diff

Comment 58

[Rail Aliiev [:rail]]

Comment on attachment 8406929 [details] [diff] [review]
fedora.diff

Review of attachment 8406929 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ -1777,5 @@
>      for platform in BRANCHES[branch]['platforms']:
>          if 'slave_platforms' not in BRANCHES[branch]['platforms'][platform]:
>              BRANCHES[branch]['platforms'][platform]['slave_platforms'] = list(PLATFORMS[platform]['slave_platforms'])
>  
>  NON_UBUNTU_BRANCHES = set([name for name, branch in items_before(BRANCHES, 'gecko_version', 22)])

Can you delete NON_UBUNTU_BRANCHES as well? It's not used

::: mozilla-tests/config.py
@@ -2329,5 @@
> -        for suite_type, ubuntu_tests in [('opt_unittest_suites',
> -                                          get_ubuntu_unittests(branch, 'opt_unittest_suites')),
> -                                         ('debug_unittest_suites',
> -                                          get_ubuntu_unittests(branch, 'debug_unittest_suites'))]:
> -            if nested_haskey(BRANCHES[branch]['platforms'], p, ubuntu,

since this is the only place where we use nested_haskey in this file, can you also remove its import?

Comment 59

[Armen [:armenzg]]

We don't run anymore jobs on the minis.

It is time to do any remaining clean up and close this bug.

Comment 60

[Armen [:armenzg]]

At slavealloc:
mysql> delete from slaves where name like 'talos-r3-fed%';
Query OK, 165 rows affected (0.01 sec)

Comment 61

[Armen [:armenzg]]

I only left code for the fedora slaves on buildapi for the sake of looking at the past on our reports.

Comment 62

[Armen [:armenzg]]

I forgot to go and disable buildbot on all of the instances. Most of them had stopped.

This is the command to deal with them:
/tools/buildbot/bin/buildslave stop talos-slave