Tests often So instead of: [test_foo] skip-if = (os == "mac" && debug) || (os == "linux" && !debug) || os == "windows" # bug 123456, bug 123457, disabled during initial triage We could write: [test_foo] skip-if = os == "mac" && debug # bug 123456 os == "linux" && !debug # bug 123457 os == "windows" # disabled during initial triage That way it would be really obvious which condition is disabled for what reason. I think this change would be relatively easy to make in manifestparser. Going a step further, we could even make the reason part of the skip-if line that gets parsed into the metadata. Then we could have a list of well defined reasons that people could search for using `mach test-info`. So the above example could become: [test_foo] skip-if = os == "mac" && debug ; intermittent, bug 123456 os == "linux" && !debug; failure, bug 123457 os == "windows"; initial-triage, bug 123458 This way the bug and reason metadata would be included in the test object, and would therefore be searchable via tools like `mach test-info`.
Bug 1675319 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Tests often fail for different reasons on different configurations. Usually we just append a comment and list bugs separate by a comma. But it's not always clear which bug is for which reason. It's sometimes not even clear which parts of the expression belong to which issue. I propose we try to encourage multi-line skip-if statements. So instead of: ``` [test_foo] skip-if = (os == "mac" && debug) || (os == "linux" && !debug) || os == "windows" # bug 123456, bug 123457, disabled during initial triage ``` We could write: ``` [test_foo] skip-if = os == "mac" && debug # bug 123456 os == "linux" && !debug # bug 123457 os == "windows" # disabled during initial triage ``` That way it would be really obvious which condition is disabled for what reason. I think this change would be relatively easy to make in manifestparser. Going a step further, we could even make the reason part of the skip-if line that gets parsed into the metadata. Then we could have a list of well defined reasons that people could search for using `mach test-info`. So the above example could become: ``` [test_foo] skip-if = os == "mac" && debug ; intermittent, bug 123456 os == "linux" && !debug; failure, bug 123457 os == "windows"; initial-triage, bug 123458 ``` This way the bug and reason metadata would be included in the test object, and would therefore be searchable via tools like `mach test-info`.
Tests often fail for different reasons on different configurations. Usually we just append a comment and list bugs separate by a comma. But it's not always clear which bug is for which reason. It's sometimes not even clear which parts of the expression belong to which issue. I propose we try to encourage multi-line skip-if statements. So instead of: ``` [test_foo] skip-if = (os == "mac" && debug) || (os == "linux" && !debug) || os == "windows" # bug 123456, bug 123457, disabled during initial triage ``` We could write: ``` [test_foo] skip-if = os == "mac" && debug # bug 123456 os == "linux" && !debug # bug 123457 os == "windows" # disabled during initial triage ``` That way it would be really obvious which condition is disabled for what reason. I think this change would be relatively easy to make in manifestparser. Going a step further, we could even make the reason part of the skip-if line that gets parsed into the metadata. Then we could have a list of well defined reasons that people could search for using `mach test-info`. So the above example could become: ``` [test_foo] skip-if = os == "mac" && debug; intermittent, bug 123456 os == "linux" && !debug; failure, bug 123457 os == "windows"; initial-triage, bug 123458 ``` This way the bug and reason metadata would be included in the test object, and would therefore be searchable via tools like `mach test-info`.