1242986 - Enable devtools e10s mochitests on Linux debug

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Julian Descottes [:jdescottes]]

Currently e10s devtools mochitest suites are not enabled in Linux debug builds. 
(https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l2651)

Could we enable them?

Comment 1

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I think we would want this on regular trees like fx-team as well, not just try only.

Comment 2

[Phil Ringnalda (:philor)]

Though we have no real idea whether or not anyone has greened them up, so we don't much want to just plop them down on every tree with no idea what will happen.

Traditionally, we would enable them on some twig, get them green, then enable them everywhere, but since taskcluster is unicorns pooping sugarplums while singing lullabies, perhaps we could bypass that by getting Armen to do taskcluster runs in a try push and then turn them on in the tier-2 taskcluster jobs to find out whether or not they work?

Comment 3

[Armen [:armenzg]]

Not sugarplums but strawberries.

In any case, let's dooo it.

Comment 4

[Armen [:armenzg]]

These ones?
https://treeherder.mozilla.org/#/jobs?repo=try&revision=05e901fb8235&filter-searchStr=devtools&group_state=expanded

I will have them enabled by next week.

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

The first group (without e10s) is already running on try, fx-team, etc.

So, the second group (with e10s) is all that's needed, I believe.

Comment 6

[Armen [:armenzg]]

OK, here's where we really stand with these jobs.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=aba896a9c4e2&filter-searchStr=devtools%20e10s&group_state=expanded

Because a bug we were running the jobs without --e10s.

If someone is available to fix these issues I can paste in here the STR with docker.

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I'm assuming that :jdescottes will investigate.

Comment 8

[Julian Descottes [:jdescottes]]

Sure, I would like to have a look at these issues.

:armenzg can you add the STRs you mentioned?

Comment 9

[Armen [:armenzg]]

Let's wait to look into this next week.

We believe we found a root issue in bug 1245243.
Once we fix it some of these issues will go away.

Best case scenario there's nothing to be done in here (I doubt it) but I assume some (if not all) crashes will clear up.

Comment 10

[Armen [:armenzg]]

We've made progress on bug 1245243, however, the dt jobs have not changed much (We were aiming to clear some crashes due to small shared memory).

In any case, here's where we're at the moment:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=08286a17a375,eb9f9996027b&filter-searchStr=dev&group_state=expanded

The most recent push shows the 4 pushes you will need to trigger devtools jobs on try.
Revision 25554400622b allows you to skip the build and run the tests immediately. It saves about 50 minutes.
Don't use that patch if you need to make changes to the harnesses or the tests.

The try syntax is: try: -b d -p linux64_tc -u mochitest-dt -t none

Revision 809d26b228e4 allows you to use "-p linux64_tc" which saves triggering the Buildbot build.

You can also use --interactive on your push to have ssh-like and VNC-like access to the images running the tests.

Here are the steps to reproduce locally:
https://pastebin.mozilla.org/8858733

Let me know if I can explain further.

Comment 11

[Armen [:armenzg]]

My apologies. I should have not have assigned this to myself.
Let me know if you need help scheduling these on try or reproducing locally.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

BTW, I've been running Linux debug e10s devtools on Ash for awhile now. Mostly green with dead CPOW oranges being the most common failure mode. Same can be said for devtools e10s on other platforms, though, so not sure if that's worth holding things up over.
https://treeherder.mozilla.org/#/jobs?repo=ash&filter-searchStr=linux%20debug%20dev&group_state=expanded

Comment 13

[Ryan VanderMeulen [:RyanVM]]

I attempted to enable these in production over in bug 1275281, but they didn't stick due to a higher than expected rate of intermittent failures.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Attachment: [taskcluster] Enable Linux64 debug devtools e10s tests

This patch can be used for testing. You should be able to push this to Try with |try: -b d -p linux64 -u mochitest-e10s-devtools-chrome| and things will Just Work.

Comment 15

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Bug 1242986 - Enable linux64 debug DevTools e10s tests.

Review commit: https://reviewboard.mozilla.org/r/101892/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/101892/

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

It looks like this is working now.  I believe I updated the patch for TC changes.

Here's a try run with 6 green attempts:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=379703766e842595d767a1afceceb9819f875139

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8823348 [details]
Bug 1242986 - Enable linux64 debug DevTools e10s tests.

https://reviewboard.mozilla.org/r/101892/#review102286

Niiiiiiice. Will you be going through the manifests at some point and re-enabling tests disabled that way as well?

Comment 18

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #17)
> Comment on attachment 8823348 [details]
> Bug 1242986 - Enable linux64 debug DevTools e10s tests.
> 
> https://reviewboard.mozilla.org/r/101892/#review102286
> 
> Niiiiiiice. Will you be going through the manifests at some point and
> re-enabling tests disabled that way as well?

After super quick skim, I did not notice any that had been disabled for only this situation, so perhaps we're okay here.  If we do find some, let's tweak in a follow up.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d768c92253e9
Enable linux64 debug DevTools e10s tests. r=RyanVM

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d768c92253e9