561437 - [Tracking bug] Turn off packaged unittests on try-as-branch and send to Talos r3

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

As we are able to handle load, turn off the linux, mac 10.5, and win32 unittest builders on try-as-branch.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: sends try linux/mac unittests to talos-r3 builders

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 448029 [details] [diff] [review]
sends try linux/mac unittests to talos-r3 builders

macosx64 and linux64 are missing from the talos branch configs now, is that intentional?

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 448029 [details] [diff] [review]
sends try linux/mac unittests to talos-r3 builders

Will re-do this when talos load issues are unblocked to include whatever OSes we have talos-r3 unittests working for.

Comment 4

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: enable try unittests on talos-r3/talos-staging-pool

Comment 5

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: turn off unittests on try and send to talos-r3

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 453116 [details] [diff] [review]
enable try unittests on talos-r3/talos-staging-pool

http://hg.mozilla.org/build/buildbot-configs/rev/59fac9e01b44

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 453117 [details] [diff] [review]
turn off unittests on try and send to talos-r3

http://hg.mozilla.org/build/buildbot-configs/rev/a49190f482bc

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: add talos masters to try unittest masters list

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

ok so quick update - the linux, linux64, macosx, macosx64 unittests are going to talos-r3 right now.  Tomorrow morning I will turn off the unittests on try builders and let talos-r3 be the only unittest master for these platforms.

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

waiting until morning because i had to fix a unittest master naming typo here: http://hg.mozilla.org/build/buildbot-configs/rev/406f0f8f17c7 to see coverage on both talos masters

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Stop sendchanging unit to talos-r3

The load is out of control on test-master01 a very busy checkin day.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 453959 [details] [diff] [review]
Stop sendchanging unit to talos-r3

http://hg.mozilla.org/build/buildbot-configs/rev/ed0c697531e0

try master on pm02 reconfig'd, and the Maintenance page updated (not optional!).

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: flipping on linux/macosx unittests, adding sendchanges to test masters for try

This patch also turns the 'warn on failure' to True for the unittest_masters that send to production, and it also turns off debug win64 unittests for mozilla-central and mozilla-2.0

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 455180 [details] [diff] [review]
flipping on linux/macosx unittests, adding sendchanges to test masters for try

http://hg.mozilla.org/build/buildbot-configs/rev/739dbf12227a

Comment 15

[Armen [:armenzg]]

Bug 580139 might be a fall off from this bug. We might not be sending emails for unit tests running on the test masters.

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

Update: At the momment win32 is the only platform left on try that is running unittests on the builders.  The tests are *also* running on the test masters, on win7, but are not completely green yet.  All that's left to do for this bug is to turn off the win32 unittests on the try-trunk-master when the win7 tests go green on test master.

Comment 17

[Armen [:armenzg]]

Win32 debug unit tests is all that there is left - bug 562459.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] enables debug win32 for test_masters in try_parser

We'll need this to be able to get debug win32 test runs happening on talos masters, and then once that looks good we can turn off the enable_unittests on the config for the builder_master.

I tested this locally on a test-master and a builder-master as well as added two new tests in the test_try_parser

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] enables debug win32 for test_masters in try_parser and also handles error on expandTestSuites

After getting more comments about mysterious default set builds along the lines of those in bug 615776 I went back and figured out that I should be testing for the existence of unittestSuites before trying to expand them.  This new patch reflects that change, is a little cleaner, and also has been tested on my local master to make sure the exception is in fact cleared up and the desired builds are triggered.

Comment 20

[Chris Cooper [:coop] (he/him)]

Comment on attachment 494588 [details] [diff] [review]
[tested] enables debug win32 for test_masters in try_parser and also handles error on expandTestSuites

>diff --git a/test/test_try_parser.py b/test/test_try_parser.py
>\ No newline at end of file

Fix before committing?

>diff --git a/try_parser.py b/try_parser.py
>+                              custom_builder = "%s tryserver %s %s %s" % (prettyNames[platform], buildType, testType, test)
>+                                # have to check that custom_builder is not already present
>+                              if custom_builder in (builderNames) and custom_builder not in testBuilders:

Fix comment indenting before committing?

>                     # we do all but debug win32 over on test masters so have to check the 

Are we going to need a follow-up bug here for the work Armen is doing to move the debug win32 testing to the test slaves?

>\ No newline at end of file

Again, fix before committing?

Otherwise, looks good.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: [tested] enables debug win32 for test_masters in try_parser and also handles error on expandTestSuites v.2

carrying forward the r+ on this, with nits addressed.

No, we will not need another patch to try_parser to address the move of debug win32 over to talos minis, this will now generate debug test builders for any platform(s) that we need to run on a builder_master for some reason, and at the same time it lets all platforms (now including win32-debug) generate the test builders to be triggered on a tests-master. When the time comes to turn off the win32-debug unittests on the try builder master it will only be a one-line config.py change on 'enableUnittests'

Comment 22

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 494600 [details] [diff] [review]
[tested] enables debug win32 for test_masters in try_parser and also handles error on expandTestSuites v.2

http://hg.mozilla.org/build/buildbotcustom/rev/44292bb9587d

(added removal of valid_builders.py and test/valid_builders.py since they are no longer needed)

Comment 23

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: set enable_unittests to False for win32-debug

just preparing it for when it's time.

Comment 24

[Lukas Blakk [:lsblakk] use ?needinfo]

So what's happening with this now?  Are we still planning to turn off the debug unittests on win2k3?

Comment 25

[Armen [:armenzg]]

I have to make the XP unit tests work before I shut them off.
XP and W2k3 share the same core and it became a "must" before disabling it.

I expect having this running in the next 1-2 weeks but we will another 2-3 weeks for developers to tackle the perma-oranges. This puts us in the last week of January or first of February.

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

Armen, I'm reassigning this to you since you're watching this project closely.  The patch is already reviewed and ready to land once you have the XP tests working properly.

Comment 27

[Armen [:armenzg]]

Sure :)

Comment 28

[Armen [:armenzg]]

Comment on attachment 494753 [details] [diff] [review]
set enable_unittests to False for win32-debug

We finalized the work on this bug this morning with this:
http://hg.mozilla.org/build/buildbot-configs/rev/2f6597acbead