1090641 - Enable e10s devtools and reftests for Linux opt

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: enable-devtools-reftests

It seems better to do this in one bug since most of the burden is in testing it as far as I can tell.

These tests are already running with e10s enabled on holly. They're pretty close to being green. I want to get this bug going in parallel.

This patch makes the necessary changes to buildbot-configs. I haven't tested it since I'm not really sure how. It's pretty similar to other changes in the past though.

I don't think we need to change mozharness at all. It already accepts a --e10s option that is passed through to the test runner. And runreftests.py already accepts a --e10s option.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Also, I just checked that these are the correct job names for tbpl. I'm assuming treeherder works the same way.

Comment 3

[Armen [:armenzg]]

Comment on attachment 8513130 [details] [diff] [review]
enable-devtools-reftests

Review of attachment 8513130 [details] [diff] [review]:
-----------------------------------------------------------------

The patch looks good to me.
It passes test-masters.sh.

billm, I see the devtools and reftest jobs on Holly running orange [1][2]
The only one that I see green is the opt crashtests [3]
Are we ready to enable on trunk?

Asking for nthomas to be sure that there are no issues to add these many more jobs (I think we're fine via test-masters.sh)

[1] https://tbpl.mozilla.org/?tree=Holly&jobname=devtools
[2] https://tbpl.mozilla.org/?tree=Holly&jobname=reftest
[3] https://tbpl.mozilla.org/?tree=Holly&jobname=opt.*crash

Comment 4

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Thanks. I'm still waiting for green on holly. Bug 1090627 needs to land and I also need to do a merge. After that I'll do a bunch of retriggers. Once it looks good this will be ready to land.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8513130 [details] [diff] [review]
enable-devtools-reftests

Currently 3783 builders on the linux64 testers, and you're adding 60, so that's fine.

I'm curious what the end-game is here. Will we still run e10s and non-e10s flavors of everything once we've enabled e10s by default ?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Armen, http://nbviewer.ipython.org/url/people.mozilla.org/~nthomas/check%20builder%20count%20.ipynb to help you self-serve that sort of question. allthethings is generated by https://hg.mozilla.org/build/braindump/file/default/buildbot-related/dump_allthethings.sh.

Comment 7

[Armen [:armenzg]]

billm, once everything is green and the changes are on m-c for a day or so feel free to land.

Comment 8

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The reftest work is taking longer than I expected, but the devtools tests are green. Therefore I landed most of this patch, excepting the parts that enable the reftests:
https://hg.mozilla.org/build/buildbot-configs/rev/db9baccc7994

Here are the holly results for the devtools tests that were enabled:
https://tbpl.mozilla.org/?tree=Holly&rev=03b31879e1c7

Hopefully I'll be able to land the reftest stuff next week.

Comment 9

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/db9baccc7994

Comment 10

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The latest push to holly was green, so here's the second part:
https://hg.mozilla.org/build/buildbot-configs/rev/312dd3f51908

Comment 11

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/312dd3f51908

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Why are we enabling permafailing tests in production by default?
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3617639&repo=mozilla-inbound

Hidden across all trees.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Also, is there a bug on file for shutting off crashtest-ipc and reftest-ipc once these are green and unhidden?

Comment 14

[Armen [:armenzg]]

(In reply to Bill McCloskey (:billm) from comment #10)
> The latest push to holly was green, so here's the second part:
> https://hg.mozilla.org/build/buildbot-configs/rev/312dd3f51908

We can't enable jobs without landing on m-i first and let it percolate through the various trees for a day or two at least.

RyanVM, should we back out for now?

Comment 15

[Ryan VanderMeulen [:RyanVM]]

I already hid them anyway. I just wish a bit more consideration was given instead of dumping permafail on trees that are already in bad shape (as Bill knows).

Comment 16

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I'm sorry this happened. I still don't understand why these jobs are failing. The push to holly was successful and, to the best of my understanding, we're running the same jobs on inbound as on holly. Either there's some difference in the configuration or else something regressed in between the holly push yesterday and now.

Most of the reftests are actually passing, and there aren't any timeouts or crashes, so I think there's still value in running them hidden on every push. I'll try to get this sorted out in the next day or so. I don't think we should consider backing out yet.

Comment 17

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It looks like the tests did regress over one day. I pushed to holly again and they're failing there now, while we were passing before. I guess I'll have to bisect.

Comment 18

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Bug 1083635 was the cause.