> Is there a way to add the prefs that devtools is adjusting to this file and then move forward? I implemented a logic which simply clears preferences at the end of the test, but limited to devtools preferences. I could just do it for all preferences, but in that case, that should really be the behavior of the test harness. IMO this feature is not helping to find issues in the implementation, but issues in tests which don't cleanunp prefs correctly. So if the test harness would be responsible for cleaning up most (all?) of the prefs modified during the test, that would be even more beneficial. > In a perfect world every test that is tier-1 would pass test-verify. Also test-verify would do a lot more to ensure the test is independent. It would be better if this validation was performed earlier than test-verify, which only runs when a test is updated. I think the end goal should be to enable this by default on all jobs, so that issues are detected very early. But prior to that there should be an effort to green up the tree, or to have an easy way to disable it in various test suites and let suite owners fix the tests when they have the bandwidth to do that.
Bug 1816549 Comment 13 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
> Is there a way to add the prefs that devtools is adjusting to this file and then move forward? I implemented a logic which simply clears preferences at the end of the test, but limited to devtools preferences. I could just do it for all preferences, but in that case, that should really be the behavior of the test harness. IMO this feature is not helping to find issues in the implementation, but issues in tests which don't cleanup prefs correctly. So if the test harness would be responsible for cleaning up most (all?) of the prefs modified during the test, that would be even more beneficial. > In a perfect world every test that is tier-1 would pass test-verify. Also test-verify would do a lot more to ensure the test is independent. It would be better if this validation was performed earlier than test-verify, which only runs when a test is updated. I think the end goal should be to enable this by default on all jobs, so that issues are detected very early. But prior to that there should be an effort to green up the tree, or to have an easy way to disable it in various test suites and let suite owners fix the tests when they have the bandwidth to do that.