940161 - Schedule mochitests on debug B2G emulators on all trunk trees

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

It looks like we've achieved a < 10% failure rate on mochitests on debug B2G emulators on cedar; we should schedule them across all trunk trees now.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule debug B2G mochitests on trunk branches,

I think this should do the trick.

Comment 2

[Aki Sasaki (not active)]

Attachment: builderdiff

Diff of a clean/patched linux test master.
This looks good.

Created via:
./setup-master.py -l
./setup-master.py bm51-tests1-linux bm51-tests1-linux
cd bm51-tests1-linux
/src/clean/braindump/buildbot-related/builder_list.py master.cfg > ../../clean.dump
# apply patch to buildbot-configs
/src/clean/braindump/buildbot-related/builder_list.py master.cfg > ../../dirty.dump
cd ../..
diff {clean,dirty}.dump > builderdiff

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8338154 [details] [diff] [review]
Schedule debug B2G mochitests on trunk branches,

I also see the debug emulator build running sendchanges on fx-team and m-c, so we should see tests running after this patch lands + reconfigs.

Comment 4

[Aki Sasaki (not active)]

Thanks Jonathan!

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule debug B2G mochitests on trunk branches,

This was bitrotted quite a bit; reflagging for review.  Aki, can you run another builder diff on this to make sure I got it right?

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 8358550 [details] [diff] [review]
Schedule debug B2G mochitests on trunk branches,

>+BRANCHES['mozilla-b2g18_v1_0_1']['platforms']['emulator']['enable_debug_unittests'] = False

We deleted the 1.0.1 branch so this line is causing a KeyError:

Traceback (most recent call last):
  File "/src/clean/braindump/buildbot-related/builder_list.py", line 20, in <module>
    execfile(os.path.basename(sys.argv[1]), g)
  File "master.cfg", line 12, in <module>
    import b2g_config
  File "/src/jgriffin/buildbot-configs/bm51-tests1-linux/b2g_config.py", line 1443, in <module>
    BRANCHES['mozilla-b2g18_v1_0_1']['platforms']['emulator']['enable_debug_unittests'] = False
KeyError: 'mozilla-b2g18_v1_0_1'

r=me with that line removed.

Attaching the new builderdiff.

Comment 7

[Aki Sasaki (not active)]

Attachment: builderdiff

Comment 8

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule debug B2G mochitests on trunk branches,

1_0_1 line deleted, and MOCHITEST changed to MOCHITEST_EMULATOR_DEBUG, so that we get all 15 chunks, instead of 9.

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8358587 [details] [diff] [review]
Schedule debug B2G mochitests on trunk branches,

Review of attachment 8358587 [details] [diff] [review]:
-----------------------------------------------------------------

carry r+ forward

Comment 10

[Jonathan Griffin (:jgriffin)]

Sheriffs, we're turning on debug B2G emulator mochitests with the next buildbot reconfig.  These are green on pine now; we'll deal aggressively with any frequent intermittents by disabling the tests.

Comment 11

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/build/buildbot-configs/rev/29f5bfaea918

Comment 12

[Gregor Wagner [:gwagner]]

(In reply to Jonathan Griffin (:jgriffin) from comment #10)
> Sheriffs, we're turning on debug B2G emulator mochitests with the next
> buildbot reconfig.  These are green on pine now; we'll deal aggressively
> with any frequent intermittents by disabling the tests.

Stop the Presses! Everything is failing right now because of bug 958732.

Comment 13

[Gregor Wagner [:gwagner]]

The patch in bug 958732 fixes this. We just have to make sure it lands first.

Comment 14

[Rail Aliiev [:rail]]

in production

Comment 15

[Ed Morley [:emorley]]

All b2g emulator debug mochitest jobs hidden on b2g-inbound for being orange; the other trees will need doing as soon as jobs have completed on them (and thus are able to be hidden in TBPL).

Comment 16

[Jonathan Griffin (:jgriffin)]

These are running very green again on b2g-inbound; any reason not to unhide them?

Comment 17

[Gregor Wagner [:gwagner]]

Oh they are hidden? Yes please un-hide again.

Comment 18

[Phil Ringnalda (:philor)]

Unhidden on b2g-inbound, where I'm always happy to unhide (amusingly, permaorange on them had just landed). But let's see some of that comment 10 aggressive disabling before we unhide them elsewhere.

Comment 19

[Gregor Wagner [:gwagner]]

(In reply to Phil Ringnalda (:philor) from comment #18)
> Unhidden on b2g-inbound, where I'm always happy to unhide (amusingly,
> permaorange on them had just landed). But let's see some of that comment 10
> aggressive disabling before we unhide them elsewhere.

Well if we know that changeset that regressed it we shouldn't blame the test :)

Comment 20

[Phil Ringnalda (:philor)]

Oh, I don't want that permaorange disabled, I already backed out Nuwa for that (though the end result will probably be disabling a bogus assertion); I mean the rest of the constant but not permaorange failures. They are running "very green" if you look at several pushes where they were all green; they are not so green if you are the single person on the planet who looks at every single orange one on every single tree.

Comment 21

[Jonathan Griffin (:jgriffin)]

(In reply to Phil Ringnalda (:philor) from comment #20)
> Oh, I don't want that permaorange disabled, I already backed out Nuwa for
> that (though the end result will probably be disabling a bogus assertion); I
> mean the rest of the constant but not permaorange failures. They are running
> "very green" if you look at several pushes where they were all green; they
> are not so green if you are the single person on the planet who looks at
> every single orange one on every single tree.

If you see frequent failures that we haven't noticed, please note them here or in bug 929122 and we'll disable them for now.

Comment 22

[Jonathan Griffin (:jgriffin)]

I'm closing this since the tests are running where we want them too; I realize there's some more effort required to green them up, but let's track that against bug 929122 or file new bugs, as needed.