Bugzilla

(In reply to :Ehsan Akhgari from comment #9)

> How are the test failures in the devtools test related to the test changes made in this patch?  I see zero evidence of any such connection.  In fact, it is impossible for there to be any such connection, since the patch that landed here makes changes to tests that run in a *different suite*, so there is zero chance of the changes here to have made any impact whatsoever to the test that fails intermittently here.
This is an issue how the backout was performed, maybe the first revision was still in the clipboad when then end of the range which shall be backed out got pasted.

> You may ask yourself, could the failure in https://searchfox.org/mozilla-central/source/devtools/client/webconsole/test/mochitest/browser_webconsole_trackingprotection_errors.js have anything to do with bug 1525458 which also landed together with this patch?  Looking at that test, the answer is no, since that test doesn't exercise the feature that this bug made changes to (url classifier notifications when the top-level page is on the content blocking allow list -- the test doesn't use the allow list at all.)  So again no evidence for a connection there.
When bug 1525458 landed for the first time, it caused this test to fail: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&tochange=66ba790db6ff5210b1429173732a5816b39849d2&fromchange=d7d6b249119a5c8a8b179407f801152f09acb0b3&searchStr=devtools%2Clinux%2Copt&selectedJob=231055048 That's the first failure when this test started to fail frequently.

> Looking at the log of the test, has this test been modified recently?  Yes, it was modified just a couple of weeks ago in bug 1514824.
Frequent fails (except for edge cases) aren't triggered by a change weeks after it landed. It was correct to look for a recent change.

> Looking at all of the available evidence, it seems like what happened was the test had "tracking protection" in its name, and it had an intermittent failure bug in it, and instead of a bug being filed for it and the author of the test being asked to look at it, the latest patch that made a change to an unrelated test with "tracking" in its name got backed out.  Based on this, I'm going to reland my patch and ask you to file a bug on this test failure and talk to Nicolas about looking at it.  Thanks!
I will remind the sheriffs to talk to the developers if they believe to have found the change which causes a frequent failure. 

Please be aware that the code sheriffs are not coding and thus are searching the culprits of failures by searching for test and folder names and doing backfills and retriggers.

(In reply to :Ehsan Akhgari from comment #9)

> How are the test failures in the devtools test related to the test changes made in this patch?  I see zero evidence of any such connection.  In fact, it is impossible for there to be any such connection, since the patch that landed here makes changes to tests that run in a *different suite*, so there is zero chance of the changes here to have made any impact whatsoever to the test that fails intermittently here.

This is an issue how the backout was performed, maybe the first revision was still in the clipboad when then end of the range which shall be backed out got pasted.

> You may ask yourself, could the failure in https://searchfox.org/mozilla-central/source/devtools/client/webconsole/test/mochitest/browser_webconsole_trackingprotection_errors.js have anything to do with bug 1525458 which also landed together with this patch?  Looking at that test, the answer is no, since that test doesn't exercise the feature that this bug made changes to (url classifier notifications when the top-level page is on the content blocking allow list -- the test doesn't use the allow list at all.)  So again no evidence for a connection there.

When bug 1525458 landed for the first time, it caused this test to fail: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&tochange=66ba790db6ff5210b1429173732a5816b39849d2&fromchange=d7d6b249119a5c8a8b179407f801152f09acb0b3&searchStr=devtools%2Clinux%2Copt&selectedJob=231055048 That's the first failure when this test started to fail frequently.

> Looking at the log of the test, has this test been modified recently?  Yes, it was modified just a couple of weeks ago in bug 1514824.

Frequent fails (except for edge cases) aren't triggered by a change weeks after it landed. It was correct to look for a recent change.

> Looking at all of the available evidence, it seems like what happened was the test had "tracking protection" in its name, and it had an intermittent failure bug in it, and instead of a bug being filed for it and the author of the test being asked to look at it, the latest patch that made a change to an unrelated test with "tracking" in its name got backed out.  Based on this, I'm going to reland my patch and ask you to file a bug on this test failure and talk to Nicolas about looking at it.  Thanks!

I will remind the sheriffs to talk to the developers if they believe to have found the change which causes a frequent failure. 

Please be aware that the code sheriffs are not coding and thus are searching the culprits of failures by searching for test and folder names and doing backfills and retriggers.