1232643 - Add-on SDK tests should run with e10s on on the try servers

Status: RESOLVED INCOMPLETE

(Add-on SDK Graveyard :: General, defect, P2)

Description

[Gabor Krizsanits (INACTIVE)]

Currently we run the SDK tests on try servers with e10s off. We should find a way to turn it on and probably disabling some failing/intermittent tests. As a start I would disable all the tests that has unsafe CPOW warning, since there are good candidates to cause intermittent failures.

Since turning on some test suits can be e10s blocker I flag this one for e10s triage.

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: 0001-Bug-1232643-Skip-on-e10s-Addon-SDK-test-files-with-k.patch

This patch contains changes to the "jepack-package.ini" that skip ("skip-if = e10s") any test files that prevented a "./mach mochitest addon-sdk --e10s" run to complete successfully, when it runs locally on my dev laptop.

More background info:

The following test files fail consistently to run (which means
"mochitest fails even when you run it on the single specific test file"):

- test-context-menu.js (timeouts, See Bug 1239640 for more info)
- test-page-mod.js (failures, See Bug 1239642 for more info)
- test-selection.js (timeouts and failures, see Bug 1239644 for more info)
- test-tabs.js (failures, see Bug 1239645 for more info)
- test-tabs-common.js (failures, see Bug 1239645 for more info)
- test-windows.js (failures, see Bug 1239647 for more info)

The following test files fail only when running in a full 
"./mach run addon-sdk --e10s" run:

- test-tab-observer.js
- test-tab.js
- test-windows-common.js
- test-private-browsing.js
- test-test-utils.js
- test-ui-action-button.js
- test-ui-toolbar.js
- test-url.js

(it seems that they are all raising the same error message,
JavaScript Error: "TypeError: this._messageManager is undefined"
in the file "resource://gre/modules/RemoteWebNavigation.jsm", linea 40)

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Side notes:

- Test Summary when running addon-sdk tests without e10s enabled:

  Jetpack Package Test Summary
        Passed: 12828
        Failed: 0
        Todo: 0

- Test Summary when running addon-sdk tests with e10s enabled 
  (and skipping all the test files listed above):  

  Jetpack Package Test Summary
        Passed: 7432
        Failed: 0
        Todo: 0

Comment 3

[:shell escalante]

issue is SDK uses a different way to black list inside test suite in e10s.  so we can't block out part of the test cases, only all.

ask andy for plan on how to do this - work with gabor on how to do it.

Comment 4

[Andy McKay]

I sadly think that we are too late to do this now, because that we know the SDK tests fail under e10s and require a lot of fixing. Given the remaining lifespan of the SDK its probably not worth the time its going to take.

Comment 5

[Luca Greco [:rpl] [:luca] [:lgreco]]

After yesterday's brief chat over irc, I took another look at how much is needed to be able to run "the subset of SDK tests that are able to complete with e10s enabled" on the build infrastructure (and in particular, being able to check that the blacklist attached to this issue is complete or if more tests have to be added because of consistent failures when running on the build instrustructure), and here is some notes about the results of this test:

I have been able to run the mochitest-jetpack test suite with e10s enabled on linux64 in a try push by changing the related taskcluster file (on the contrary, to be able to change it on other platforms that use buildbot, changes in a separate repository seems to be necessary):

- link to the try results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=57f361bc7146a8b952b15e3c1f9d5754a7d31e01&selectedJob=29395852
- change applied to taskcluster/ci/desktop-test/tests.yml in the above try build: https://hg.mozilla.org/try/rev/d4ba972aa040

The mochitest-jetpack tests have been able to complete on linux64 in e10s mode with the blacklist from the attached patch and one additional test file blacklisted:

- test-window-utils-private-browsing.js (it fails with the assertion error message "testWindowTrackerIgnoresPrivateWindows | there are frames for private window - false == true") 

It is the first time that I look into the building tasks, and so I'm pretty sure that the change in the tasks yaml file is not complete or is not enough to achieve what we want:

the mochitest-jetpack test suite should run with both e10s enabled and disabled but, by looking at the above "try push", it seems that mochitest-jetpack is running in both modes only in the linux64 optimized build, and so I assume that the above change has defaulted mochitest-jetpack to "e10s enabled" everywhere it is executed by taskcluster (while it is still executed without e10s on the rest, the following wiki page is where I looked to get more info about where taskcluster and buildbot are used https://wiki.mozilla.org/ReleaseEngineering/Buildbot_Migration), which is not what we want to achieve.

Comment 6

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: sdk-e10s-blacklist.patch

Updated patch related to the e10s blacklist of the SDK test files (it contains the backlist from attachment 8707878 [details] [diff] [review], with the addition of the blacklisted test file specified in Comment 5, and an updated inline comment on every blacklisted test file with a reference to its related bugzilla issue).

Comment 9

[Firefox Bug Husbandry Bot]

https://bugzilla.mozilla.org/show_bug.cgi?id=1399562