1536573 - windows/aarch64 - disable tests from toolkit/content/tests/browser/browser.ini

Status: RESOLVED FIXED

(Toolkit :: General, enhancement)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

This bug tracks disabling of all tests inside the manifest toolkit/content/tests/browser/browser.ini for mochitest-browser-chrome on windows10-aarch64.

All tests here are related to media/playback and with the uncertain state of EME and media/playback stability in windows10-aarch64 it was decided to disable it first.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1536573 - disable manifest toolkit/content/tests/browser/browser.ini for windows10-aarch64

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da1764b86627
disable manifest toolkit/content/tests/browser/browser.ini for windows10-aarch64 r=jmaher

Comment 3

[Dave Townsend [:mossop]]

(In reply to Edwin Gao (:egao) from comment #0)

All tests here are related to media/playback and with the uncertain state of EME and media/playback stability in windows10-aarch64 it was decided to disable it first.

This does not appear to be the case. I see multiple tests here that are unrelated to media. I would appreciate it if those were not disabled here. Not entirely sure why the media tests are here in the first place, it doesn't seem to be an appropriate place for them.

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Dave Townsend [:mossop] (he/him) from comment #3)

(In reply to Edwin Gao (:egao) from comment #0)

All tests here are related to media/playback and with the uncertain state of EME and media/playback stability in windows10-aarch64 it was decided to disable it first.

This does not appear to be the case. I see multiple tests here that are unrelated to media. I would appreciate it if those were not disabled here. Not entirely sure why the media tests are here in the first place, it doesn't seem to be an appropriate place for them.

The patch landed some time ago, but I suppose I could do another patch to remove the skip-if clause.

Could you point me to someone that may know why audio playback tests are in this file? This try push shows the intermittent nature of the failure I'm trying to weed out to achieve a greener run.

Comment 5

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/da1764b86627

Comment 6

[Dave Townsend [:mossop]]

(In reply to Edwin Gao (:egao) from comment #4)

(In reply to Dave Townsend [:mossop] (he/him) from comment #3)

(In reply to Edwin Gao (:egao) from comment #0)

All tests here are related to media/playback and with the uncertain state of EME and media/playback stability in windows10-aarch64 it was decided to disable it first.

This does not appear to be the case. I see multiple tests here that are unrelated to media. I would appreciate it if those were not disabled here. Not entirely sure why the media tests are here in the first place, it doesn't seem to be an appropriate place for them.

The patch landed some time ago, but I suppose I could do another patch to remove the skip-if clause.

Not sure which patch you're referring to here.

Could you point me to someone that may know why audio playback tests are in this file? This try push shows the intermittent nature of the failure I'm trying to weed out to achieve a greener run.

Looks like :alwu landed an umber of them. I guess it could be because the video controls widget is in toolkit/content/widgets and that doesn't seem to have its own test directory. The history probably doesn't matter too much at this point though.

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

:mossop - I was referring to the patch to disable the test manifest toolkit/content/tests/browser/browser.ini for windows10-aarch64 platform. It can be seen https://hg.mozilla.org/mozilla-central/rev/da1764b86627. I can re-enable the tests, but this will cause a whole slew of (possibly intermittent) oranges that will need to be dealt with.

Comment 8

[Dave Townsend [:mossop]]

(In reply to Edwin Gao (:egao) from comment #7)

:mossop - I was referring to the patch to disable the test manifest toolkit/content/tests/browser/browser.ini for windows10-aarch64 platform. It can be seen https://hg.mozilla.org/mozilla-central/rev/da1764b86627. I can re-enable the tests, but this will cause a whole slew of (possibly intermittent) oranges that will need to be dealt with.

Right, I was confused since that patch only landed 40 minutes before your comment. I'm not sure why you can't remove the global skip-if and add it just to the tests that are causing problems.

Comment 9

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Dave Townsend [:mossop] (he/him) from comment #8)

(In reply to Edwin Gao (:egao) from comment #7)

:mossop - I was referring to the patch to disable the test manifest toolkit/content/tests/browser/browser.ini for windows10-aarch64 platform. It can be seen https://hg.mozilla.org/mozilla-central/rev/da1764b86627. I can re-enable the tests, but this will cause a whole slew of (possibly intermittent) oranges that will need to be dealt with.

Right, I was confused since that patch only landed 40 minutes before your comment. I'm not sure why you can't remove the global skip-if and add it just to the tests that are causing problems.

The reason I chose to do a global skip is because the sheer number of tests that fail, in the range of 30-40 tests, so the time taken to cross-reference the tests and results would have been time consuming.

I will put this as a todo for me to re-enable some of the non-failing tests.

Comment 10

[Dave Townsend [:mossop]]

(In reply to Edwin Gao (:egao) from comment #9)

(In reply to Dave Townsend [:mossop] (he/him) from comment #8)

(In reply to Edwin Gao (:egao) from comment #7)

:mossop - I was referring to the patch to disable the test manifest toolkit/content/tests/browser/browser.ini for
The reason I chose to do a global skip is because the sheer number of tests that fail, in the range of 30-40 tests, so the time taken to cross-reference the tests and results would have been time consuming.

I will put this as a todo for me to re-enable some of the non-failing tests.

I haven't heard what sorts of shortcuts we're willing to take to get aarch64 off the ground so maybe I'm missing something but I really hope they don't include blindly disabling tests based on an incorrect justification, that sounds like a recipe for poor quality to me.

I've gone ahead and fixed this in bug 1538370 since it's not really worth it for either of us to spend more time on this. In the future though please note that test changes, including those that just disable tests, should be reviewed by the owners or peers of the tests.