Closed Bug 1232643 Opened 4 years ago Closed 2 years ago
Add-on SDK tests should run with e10s on on the try servers
Currently we run the SDK tests on try servers with e10s off. We should find a way to turn it on and probably disabling some failing/intermittent tests. As a start I would disable all the tests that has unsafe CPOW warning, since there are good candidates to cause intermittent failures. Since turning on some test suits can be e10s blocker I flag this one for e10s triage.
Side notes: - Test Summary when running addon-sdk tests without e10s enabled: Jetpack Package Test Summary Passed: 12828 Failed: 0 Todo: 0 - Test Summary when running addon-sdk tests with e10s enabled (and skipping all the test files listed above): Jetpack Package Test Summary Passed: 7432 Failed: 0 Todo: 0
issue is SDK uses a different way to black list inside test suite in e10s. so we can't block out part of the test cases, only all. ask andy for plan on how to do this - work with gabor on how to do it.
I sadly think that we are too late to do this now, because that we know the SDK tests fail under e10s and require a lot of fixing. Given the remaining lifespan of the SDK its probably not worth the time its going to take.
After yesterday's brief chat over irc, I took another look at how much is needed to be able to run "the subset of SDK tests that are able to complete with e10s enabled" on the build infrastructure (and in particular, being able to check that the blacklist attached to this issue is complete or if more tests have to be added because of consistent failures when running on the build instrustructure), and here is some notes about the results of this test: I have been able to run the mochitest-jetpack test suite with e10s enabled on linux64 in a try push by changing the related taskcluster file (on the contrary, to be able to change it on other platforms that use buildbot, changes in a separate repository seems to be necessary): - link to the try results: https://treeherder.mozilla.org/#/jobs?repo=try&revision=57f361bc7146a8b952b15e3c1f9d5754a7d31e01&selectedJob=29395852 - change applied to taskcluster/ci/desktop-test/tests.yml in the above try build: https://hg.mozilla.org/try/rev/d4ba972aa040 The mochitest-jetpack tests have been able to complete on linux64 in e10s mode with the blacklist from the attached patch and one additional test file blacklisted: - test-window-utils-private-browsing.js (it fails with the assertion error message "testWindowTrackerIgnoresPrivateWindows | there are frames for private window - false == true") It is the first time that I look into the building tasks, and so I'm pretty sure that the change in the tasks yaml file is not complete or is not enough to achieve what we want: the mochitest-jetpack test suite should run with both e10s enabled and disabled but, by looking at the above "try push", it seems that mochitest-jetpack is running in both modes only in the linux64 optimized build, and so I assume that the above change has defaulted mochitest-jetpack to "e10s enabled" everywhere it is executed by taskcluster (while it is still executed without e10s on the rest, the following wiki page is where I looked to get more info about where taskcluster and buildbot are used https://wiki.mozilla.org/ReleaseEngineering/Buildbot_Migration), which is not what we want to achieve.
Updated patch related to the e10s blacklist of the SDK test files (it contains the backlist from attachment 8707878 [details] [diff] [review], with the addition of the blacklisted test file specified in Comment 5, and an updated inline comment on every blacklisted test file with a reference to its related bugzilla issue).
Attachment #8707878 - Attachment is obsolete: true
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.