Bugzilla

Agreed, I will keep the patch will be in changes planned until we know what is the decision with TV and compare-preferences.

> or general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself.

I considered the options and did test cleanup as much as possible. But some tests devtools tests will indirectly trigger preferences completely unrelated to devtools by using the browser (opening a page, triggering a download etc...). I don't want to add them to ignorePrefs.json because I don't really know much about those preferences and folks maintaining them might disagree. I could just have an ignore list for each of those tests and cleanup prefs manually at the end but that's fragile. I have already seen yesterday new bugs filed because of preferences introduced to implement changes unrelated to devtools.

> motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests.

Quick clarification, my motivation is not address the bulk of DevTools failures. I already landed patches which fix 99% of the issues in /devtools. But TV is an important tool for me when investigating and fixing intermittent failures, and so the few tests which fall in the category described above, I wanted a solution to be able to keep using test-verify.

Agreed, I will keep the patch will be in changes planned until we know what is the decision with TV and compare-preferences.

> or general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself.

I considered the options and did test cleanup as much as possible. But some tests devtools tests will indirectly trigger preferences completely unrelated to devtools by using the browser (opening a page, triggering a download etc...). I don't want to add them to ignorePrefs.json because I don't really know much about those preferences and folks maintaining them might disagree. I could just have an ignore list for each of those tests and cleanup prefs manually at the end but that's fragile. I have already seen yesterday new bugs filed because of preferences introduced to implement changes unrelated to devtools.

> motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests.

Quick clarification, my motivation is not to address the bulk of DevTools failures. I already landed patches which fix 99% of the issues in /devtools. But TV is an important tool for me when investigating and fixing intermittent failures, and so the few tests which fall in the category described above, I wanted a solution to be able to keep using test-verify.

Agreed, I will keep the patch will be in changes planned until we know what is the decision with TV and compare-preferences.

> or general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself.

I considered the options and did test cleanup as much as possible. But some devtools tests will indirectly trigger preferences completely unrelated to devtools by using the browser (opening a page, triggering a download etc...). I don't want to add them to ignorePrefs.json because I don't really know much about those preferences and folks maintaining them might disagree. I could just have an ignore list for each of those tests and cleanup prefs manually at the end but that's fragile. I have already seen yesterday new bugs filed because of preferences introduced to implement changes unrelated to devtools.

> motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests.

Quick clarification, my motivation is not to address the bulk of DevTools failures. I already landed patches which fix 99% of the issues in /devtools. But TV is an important tool for me when investigating and fixing intermittent failures, and so the few tests which fall in the category described above, I wanted a solution to be able to keep using test-verify.

Agreed, I will keep the patch in changes planned until we know what is the decision with TV and compare-preferences.

> or general preferences unrelated to the specific test, an exception should be added to the ignorePrefs.json file If a test expects a pref to change, it should clean it up itself.

I considered the options and did test cleanup as much as possible. But some devtools tests will indirectly trigger preferences completely unrelated to devtools by using the browser (opening a page, triggering a download etc...). I don't want to add them to ignorePrefs.json because I don't really know much about those preferences and folks maintaining them might disagree. I could just have an ignore list for each of those tests and cleanup prefs manually at the end but that's fragile. I have already seen yesterday new bugs filed because of preferences introduced to implement changes unrelated to devtools.

> motivated by the fact that TV (--verify) test jobs are currently failing for many DevTools tests.

Quick clarification, my motivation is not to address the bulk of DevTools failures. I already landed patches which fix 99% of the issues in /devtools. But TV is an important tool for me when investigating and fixing intermittent failures, and so the few tests which fall in the category described above, I wanted a solution to be able to keep using test-verify.