Bugzilla

(In reply to Rob Wu [:robwu] from comment #4)
> (In reply to :Gijs (he/him) from comment #3)
> > How badly do things fail if we make this the default behaviour instead of an opt-in flag - could we do a trypush and find out? (I'm not sure I understand why this was originally added as an opt-in flag for mochitest-plain...)
> 
> Try push: `./mach try fuzzy -q "'mochitest"` results in 56446 failures (https://treeherder.mozilla.org/jobs?repo=try&revision=3cb255849869c5f91d905d393eb99fb728520715).
> 

Sorry to push back, but should we keep this on by default for test-verify?

I am trying to green up devtools tests in Bug 1974700. I'm close but there are a bunch of tests where the preferences are outside of devtools' control, and more importantly where the list of changed preferences is not stable. For instance, one test opens about:preferences which seems to flip some services.sync preferences, but not always the same ones. 

My pitch is that I'm relying on test-verify to investigate intermittent failures. I don't know how many individual tests are affected by the 50000+ failures from the try push mentioned above but that means test-verify can no longer be used for any of those unless we fix the test first. Which might be trivial, or might mean rewriting the test. That also means a TV bug will be opened the next time any of those bugs is modified. Those bugs are already a low quality signal - they are usually incorrectly flagged as regressions of the bug which triggered the TV run. 

I will file a follow up to allow individual tests to opt out of the comparePrefs checks (I think that's a must-have for some tests). But still wanted to raise the issue, maybe we should roll this out more progressively?

(In reply to Rob Wu [:robwu] from comment #4)
> (In reply to :Gijs (he/him) from comment #3)
> > How badly do things fail if we make this the default behaviour instead of an opt-in flag - could we do a trypush and find out? (I'm not sure I understand why this was originally added as an opt-in flag for mochitest-plain...)
> 
> Try push: `./mach try fuzzy -q "'mochitest"` results in 56446 failures (https://treeherder.mozilla.org/jobs?repo=try&revision=3cb255849869c5f91d905d393eb99fb728520715).
> 

Sorry to push back, but should we keep this on by default for test-verify?

I am trying to green up devtools tests in Bug 1974700. I'm close but there are a bunch of tests where the preferences are outside of devtools' control, and more importantly where the list of changed preferences is not stable. For instance, one test opens about:preferences which seems to flip some services.sync preferences, but not always the same ones. 

My pitch is that I'm relying on test-verify to investigate intermittent failures. I don't know how many individual tests are affected by the 50000+ failures from the try push mentioned above but that means test-verify can no longer be used for any of those unless we fix the test first. Which might be trivial, or might mean rewriting the test. That also means a TV bug will be opened the next time any of those tests is modified. Those bugs are already a low quality signal - they are usually incorrectly flagged as regressions of the bug which triggered the TV run. 

I will file a follow up to allow individual tests to opt out of the comparePrefs checks (I think that's a must-have for some tests). But still wanted to raise the issue, maybe we should roll this out more progressively?