Closed Bug 1300892 Opened 4 years ago Closed 4 years ago
Autophone - should S1S2 tests which fail to get a single measurement be treated as failures?
Autophone currently treats the failure to obtain a single measurement out of all of the iterations as a test failure which turns the test orange. We have not made any effort or headway in triaging or fixing these issues which results in the Sheriffs spending time classifying failures for an issue which will not be fixed. I'm a firm believer that if a condition isn't something that we will actively work to resolve it shouldn't turn the test orange. One possibility is to automatically retry the iteration (as long as the failure is not due to a crash). This may eliminate the situation where we have fewer than the requested number of iterations for most situations. However, if we fail to get the measurement during the retry and were able to obtain at least N-1 of the requested iterations I think we should treat the test as a success and not a failure. Thoughts?
Auto retry glosses over the real issues- although fixing the real issues might be a wasted effort. I think it is reasonable to do what is proposed- we will have real data, just a small bit less than ideal, but useful enough.
Sounds fine to me
Attachment #8789436 - Flags: review?(jmaher)
Comment on attachment 8789436 [details] [diff] [review] bug-1300892-v1.patch Review of attachment 8789436 [details] [diff] [review]: ----------------------------------------------------------------- great, I am eager to see if this will greatly reduce our oranges.
Attachment #8789436 - Flags: review?(jmaher) → review+
https://github.com/mozilla/autophone/commit/086707f07f47440a46569c42729ad098615142f9 deployed 2016-09-08 ~10:15 PDT
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.