1495311 - All of our toolkit mochitests are silently being skipped on local builds

Status: RESOLVED FIXED

(WebExtensions :: General, enhancement, P2)

Description

[Tomislav Jovanovic :zombie]

This line seems to disable them:

https://searchfox.org/mozilla-central/rev/819cd31/toolkit/components/extensions/test/mochitest/mochitest.ini#8

Without it, they run twice, in both OOP and non-OOP mode, which doesn't make much sense, but then again, I know nothing about mochitest.

At this point, we should probably be running all test only in oop mode anyway.

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

I took another look at this and it seems that the test files listed in the mochitest-common.ini file are always skipped only when you try to run them by test file path (e.g. as in "./mach mochitest toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html"), but they are actually being executed as expected when you run the entire directory.

On the build infrastructure we also run the entire test directory, and the logs seems to confirm that there these tests are not being skipped in OOP mode (but skipped as intended in their non-OOP mode), e.g. from the following "push to try"'s raw logs: https://taskcluster-artifacts.net/GmXGWJPuRbyDeMvrkhxC0g/0/public/logs/live_backing.log

We can see the following logs (filtered to only include interesting lines related to the single test_ext_background_canvas.html test file):

    11:57:59     INFO - TEST-START | toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
    11:57:59     INFO - TEST-SKIP | toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html | took 0ms 
    ...
    11:58:11     INFO - TEST-START | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_background_canvas.html
    ...    
    11:58:11     INFO - TEST-OK | toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_background_canvas.html | took 131ms

Given that the tests are not being skipped on the build infrastructure, this is not as bad as we were assuming initially and so I'm going to set a lower priority than the current P1.

(I'm going for setting it as a P2 because not being able to run those tests locally without manually editing the mochitest.ini file is pretty annoying and it would be still great to fix it... especially given that the same kind of change that we did for windows is something that we may soon do for both linux and macos, as they are also using the "extensions oop mode" by default now).

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Luca Greco [:rpl] from comment #1)
> I took another look at this and it seems that the test files listed in the
> mochitest-common.ini file are always skipped only when you try to run them
> by test file path (e.g. as in "./mach mochitest
> toolkit/components/extensions/test/mochitest/test_ext_background_canvas.
> html"), but they are actually being executed as expected when you run the
> entire directory.

I have to take this part back: locally they are being skipped even when running the "mach mochitest" on the entire directory.

On the contrary to what happen on the build infrastructure, where (as described in the previous comment) only the "non-OOP" ones are being skipped, as intended.

Comment 3

[Tomislav Jovanovic :zombie]

I agree that this turned out less serious that it looked at first, but we should really get this fixed.

OSX Try servers are overwhelmed, so I want to disable non-OOP testing there as well, which will make this an issue for most of the team.

(I would try to fix this if I knew anything about the build system)

Comment 4

[Rob Wu [:robwu]]

I just confirmed that my patch for bug 1498636 will fix this issue.

Before (without patch for bug 1498636):

$ ./mach mochitest toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
...
 0:00.14 TEST_START: toolkit/components/extensions/test/mochitest/test-oop-extensions/test_ext_background_canvas.html
 0:00.14 TEST_END: SKIP
 0:00.14 TEST_START: toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
 0:00.14 TEST_END: SKIP

After (with patch for bug 1498636):

$ ./mach mochitest toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
...
 0:00.14 TEST_START: toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
 0:00.14 TEST_END: SKIP
 0:00.14 INFO Running manifest: toolkit/components/extensions/test/mochitest/mochitest-common.ini
...
 0:02.46 TEST_START: toolkit/components/extensions/test/mochitest/test_ext_background_canvas.html
 0:03.53 INFO AddTask.js | Entering test test_background_canvas
 0:03.53 INFO Extension loaded
 0:03.60 INFO AddTask.js | Leaving test test_background_canvas
 0:03.61 GECKO(24796) MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
 0:03.61 GECKO(24796) MEMORY STAT | vsize 1286MB | residentFast 106MB | heapAllocated 13MB
 0:03.61 TEST_END: Test OK. Subtests passed 2/2. Unexpected 0

There is one skip because of the skip-if, and a non-skipped test because of the non-skipped include via https://searchfox.org/mozilla-central/rev/6e3cc1535/toolkit/components/extensions/test/mochitest/mochitest-remote.ini#12

(I changed skip-if from "windows" to "linux" to test it on my local Linux machine, but this does not affect the validity of the results)

Comment 5

[Rob Wu [:robwu]]

Patch for bug 1498636 has landed.

Verified fixed on m-c with the test case from comment 4.