1101482 - browser_988072_sidebar_events.js intermittently fails with --e10s --run-by-dir

Status: RESOLVED FIXED

(Firefox :: Toolbars and Customization, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

I am working on adjust --run-by-dir to work for browser-chrome and --e10s, unfortunately a common failure in the bc1 suite is brower_988072_sidebar_events.js:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2db7095c9402&searchQuery=e10s-browser

I can easily reproduce this locally on my linux64 desktop:
./mach mochitest-browser --e10s browser/components/customizableui/test

In playing with the manifest file, I can remove any one of these files and the error goes away locally:
[browser_989338_saved_placements_not_resaved.js]
[browser_989751_subviewbutton_class.js]
[browser_987177_xul_wrapper_updating.js]
[browser_987185_syncButton.js]

This is the error I get for the test:
13:13:05 INFO - 539 INFO TEST-START | chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js
13:13:05 INFO - 540 INFO checking window state
13:13:05 INFO - 541 INFO Entering test
13:13:05 INFO - 542 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js | Should have the same number of children - Got 0, expected 2
13:13:05 INFO - Stack trace:
13:13:05 INFO - chrome://mochikit/content/browser-test.js:test_is:834
13:13:05 INFO - chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js:compareList:70
13:13:05 INFO - chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js:null:95
13:13:05 INFO - self-hosted:InterpretGeneratorResume:960
13:13:05 INFO - self-hosted:next:886
13:13:05 INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:651:9
13:13:05 INFO - Tester.prototype.nextTest</<@chrome://mochikit/content/browser-test.js:577:7
13:13:05 INFO - SimpleTest.waitForFocus/maybeRunTests/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:683:49
13:13:05 INFO - *************************
13:13:05 INFO - A coding exception was thrown and uncaught in a Task.
13:13:05 INFO - Full message: TypeError: document.getElementById(...) is null
13:13:05 INFO - Full stack: @chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js:99:3
13:13:05 INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:651:9
13:13:05 INFO - Tester.prototype.nextTest</<@chrome://mochikit/content/browser-test.js:577:7
13:13:05 INFO - SimpleTest.waitForFocus/maybeRunTests/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:683:49
13:13:05 INFO - *************************
13:13:05 INFO - 543 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js | Uncaught exception - at chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js:99 - TypeError: document.getElementById(...) is null
13:13:05 INFO - Stack trace:
13:13:05 INFO - @chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js:99:3
13:13:05 INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:651:9
13:13:05 INFO - Tester.prototype.nextTest</<@chrome://mochikit/content/browser-test.js:577:7
13:13:05 INFO - SimpleTest.waitForFocus/maybeRunTests/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:683:49
13:13:05 INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:651:9
13:13:05 INFO - Tester.prototype.nextTest</<@chrome://mochikit/content/browser-test.js:577:7
13:13:05 INFO - SimpleTest.waitForFocus/maybeRunTests/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:683:49
13:13:09 INFO - 544 INFO TEST-OK | chrome://mochitests/content/browser/browser/components/customizableui/test/browser_988072_sidebar_events.js | took 4064ms 


Is there something I can do to help figure this out?

Comment 1

[:Gijs (he/him)]

Does the summary mean that the test fails:

* if you use --e10s and --run-by-dir

or

* if you use *either* --e10s or --run-by-dir ?

I'm confused because this test is already running on tinderbox and passing with e10s, but your commandline doesn't enable run by dir. What do you actually mean?

Comment 2

[:Gijs (he/him)]

(In reply to Joel Maher (:jmaher) from comment #0)
> I am working on adjust --run-by-dir to work for browser-chrome and --e10s,
> unfortunately a common failure in the bc1 suite is
> brower_988072_sidebar_events.js:
> https://treeherder.mozilla.org/ui/#/
> jobs?repo=try&revision=2db7095c9402&searchQuery=e10s-browser
> 
> I can easily reproduce this locally on my linux64 desktop:
> ./mach mochitest-browser --e10s browser/components/customizableui/test
> 
> In playing with the manifest file, I can remove any one of these files and
> the error goes away locally:
> [browser_989338_saved_placements_not_resaved.js]
> [browser_989751_subviewbutton_class.js]

these should be run after this test, so I don't understand how they can influence whether this test fails.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I had the same confusion on the two tests which run afterwards, but I could reproduce the same error 100% locally, and 3 times in a row with one line removed from the manifest and it would always pass.

To answer the first question, running the command (./mach mochitest-browser --e10s browser/components/customizableui/test) I gave treats it like --run-by-dir (since we are just running a single directory).  Right now we run non e10s browser chrome with --run-by-dir in automation, but e10s is blocked on what appears to be only this.

Comment 4

[:Gijs (he/him)]

(In reply to Joel Maher (:jmaher) from comment #3)
> I had the same confusion on the two tests which run afterwards, but I could
> reproduce the same error 100% locally, and 3 times in a row with one line
> removed from the manifest and it would always pass.

That really doesn't make sense. Did you verify the tests run in order, ie the tests we're confused about are genuinely run after the the test this bug is about?

> To answer the first question, running the command (./mach mochitest-browser
> --e10s browser/components/customizableui/test) I gave treats it like
> --run-by-dir (since we are just running a single directory).  Right now we
> run non e10s browser chrome with --run-by-dir in automation, but e10s is
> blocked on what appears to be only this.

Only this test? rs=me to disable this test itself and land run-by-dir. Just this test shouldn't hold up run-by-dir.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

well, I would rather fix this if we can reasonably.  Let me look at the run order when I am back online in 2-3 hours.  I would like to understand this more before disabling this.

For reference, we turned on run-by-dir (disabling 3 tests) for browser-chrome (non e10s).  As far as I know this is the only remaining test to fix for e10s mode browser-chrome.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

ok, my assertions of which tests were causing the failure were after narrowing down the list to just those tests + sidebar_events; in the big scope of things, I cannot see a fix with any of the above mentioned tests removed from the test list :(

Comment 7

[:Gijs (he/him)]

FWIW, I can't reproduce this locally on OS X... can you reproduce locally on any OS? Or is this Linux-only?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I only have linux64 available locally and it reproduces easily on my machine.  The predominant errors are on linux, I recall seeing in another push an error on windows.

We disabled this test yesterday to land support for e10s browser-chrome, I would be happy to adjust this down to a linux only disabling!  Likewise try any patches or gather information locally.  I will get my win7 VM up and running again and maybe that could give us additional information.

Comment 9

[:Gijs (he/him)]

I can't reproduce this on my Linux VM either, which makes this near impossible to debug / figure out in the short term. :-\

If this didn't fail on Windows/Mac, can we enable the test there?

You wrote that you can trivially reproduce on your machine, does it happen if you run the test in isolation, too? If not, any chance I can have a look in Portland? :-)

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I just verified that I could reproduce this on a recent local build on linux64.  I would love to meet up with you in Portland to do this.

Comment 11

[:Gijs (he/him)]

remote:   https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=00f14617f2eb

Comment 12

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #11)
> remote:  
> https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=00f14617f2eb

This looks pretty good considering the retriggers. I'll land it next time I'm near an open tree with a non-empty stomach. :-)

Comment 13

[:Gijs (he/him)]

https://hg.mozilla.org/integration/fx-team/rev/c677242e5be8

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c677242e5be8