1336761 - Separate out web extension tests into its own suite

Status: REOPENED

(WebExtensions :: General, defect, P5)

Description

[Tomislav Jovanovic :zombie]

+++ This bug was initially created as a clone of Bug #1336757 +++

Can we get our own test designation (is this "suite"?) for all WE tests in one place?  

As things currently stand, mistakes like bug 1336757 are easy to miss and likely to happen again -- it was pure coincidence that I noticed this so quickly, this could have gone unnoticed for weeks.

Also, when hunting for intermittent failures, searching through results of WE tests in mochitest logs 1 through 10 (which also varies between platforms) is an awful dev experience.

Comment 1

[Andy McKay]

Would you know how to do this?

Comment 2

[Andrew Halberstadt [:ahal]]

Yep, you should be able to do it self-serve even:
1) Copy+paste then tweak a config like this:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/test/tests.yml#664

2) Add it to the mozharness configs:
https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/unittests/linux_unittest.py#186

3) Whatever you chose as your string in --subsuite=<value>, just add subsuite=<value> to the [DEFAULT] section of whatever manifests should be run. That's it, test it out by pushing to try.

This will only work with taskcluster tasks (aka linux only for now). Mac and windows support should be coming later this year. Also, there is an effort underway in treeherder to completely re-think how we approach displaying results. As part of this, having to deal with chunks should be a thing of the past.. though that will be a bit longer term.

Comment 3

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.