1163684 - Disable all functional and remote tests except security tests

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We recently have massive amounts of test failures for our mozmill tests and I absolutely don't have the time to partially skip them once failures are starting to occur. So I discussed a complete disabling of the test with Robert and Otilia via IRC and we all agree to that. It's even something we wanted to do already end of last quarter but I was in the hope partially disabling tests will still give us a better test coverage. But the maintenance burden is simply too high.

So we will disable all functional tests, and all remote tests except the security ones. The latter will be replaced soon by the Marionette tests. The same applies to update tests, which I will also replace soon. Both for Nightly first and then riding the train, or if all works well, even with backports so we can reach 38ESR.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that the security tests will be the first ones to transition over to Marionette, i will also leave the functional ones enabled. Those are not failing, and we can better keep up with regressions which would also affect the Marionette tests.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: skip

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://hg.mozilla.org/qa/mozmill-tests/rev/91aa0e663506 (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/6402ee647170 (aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/c8ed0b658ee3 (beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/940b5ac4f690 (release)
https://hg.mozilla.org/qa/mozmill-tests/rev/19fe2379bd0b (esr38)
https://hg.mozilla.org/qa/mozmill-tests/rev/b30c7699207c (esr31)

Given that we wont enable those tests again I'm going to mark the bug as fixed.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: unskip security functional tests for esr31

Somehow on esr31 all functional tests got skipped. Sorry for that. This patch will re-enable the security tests which we should still run.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://hg.mozilla.org/qa/mozmill-tests/rev/4234a257a992 (esr31)