1892351 - manifests with the same test and reference but different prefs or loading, can be difficult to find the failure, split into separate manifests

Status: RESOLVED WONTFIX

(Testing :: Reftest, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

many tools decide which test to run, skip, or file a bug on a failure- but in well over 100 cases there are the same test and reference file tested but we switch things such as prepending HTTP or setting defaults pref(...) in the middle of the file. From a sheriffing point of view, this is confusing, what if you need to add a fuzzy-if, which instance do you add it to?

I propose for most of the cases to split into separate manifest files, then at least it will be easier to distinguish which failure needs attention.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1892351 - split reftest.list when same test is repeated in manifest. r=tnikkel

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

going with a different approach in bug 1893537