Closed Bug 1300892 Opened 4 years ago Closed 4 years ago

Autophone - should S1S2 tests which fail to get a single measurement be treated as failures?

Categories

(Testing :: Autophone, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(1 file)

Autophone currently treats the failure to obtain a single measurement out of all of the iterations as a test failure which turns the test orange. We have not made any effort or headway in triaging or fixing these issues which results in the Sheriffs spending time classifying failures for an issue which will not be fixed. I'm a firm believer that if a condition isn't something that we will actively work to resolve it shouldn't turn the test orange.

One possibility is to automatically retry the iteration (as long as the failure is not due to a crash). This may eliminate the situation where we have fewer than the requested number of iterations for most situations.

However, if we fail to get the measurement during the retry and were able to obtain at least N-1 of the requested iterations I think we should treat the test as a success and not a failure.

Thoughts?
Flags: needinfo?(snorp)
Flags: needinfo?(jmaher)
Auto retry glosses over the real issues- although fixing the real issues might be a wasted effort.

I think it is reasonable to do what is proposed- we will  have real data, just a small bit less than ideal, but useful enough.
Flags: needinfo?(jmaher)
Sounds fine to me
Flags: needinfo?(snorp)
Comment on attachment 8789436 [details] [diff] [review]
bug-1300892-v1.patch

Review of attachment 8789436 [details] [diff] [review]:
-----------------------------------------------------------------

great, I am eager to see if this will greatly reduce our oranges.
Attachment #8789436 - Flags: review?(jmaher) → review+
https://github.com/mozilla/autophone/commit/086707f07f47440a46569c42729ad098615142f9
deployed 2016-09-08 ~10:15 PDT
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.