1036325 - Support bisection for failing intermittent mochitests

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Vaibhav (:vaibhav1994)]

This bug is in continuation of bug 1014125 . We need to support bisection for failing intermittent tests. For this, we need to run the failing test 10 times independently and ensure it passes every time.

Comment 1

[Vaibhav (:vaibhav1994)]

Attachment: intermittent.patch

Try Run: https://tbpl.mozilla.org/?tree=Try&rev=05d069406c9d

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8461136 [details] [diff] [review]
intermittent.patch

Review of attachment 8461136 [details] [diff] [review]:
-----------------------------------------------------------------

looking great, and thanks for ensuring you have tested this!

::: testing/mochitest/bisection.py
@@ +259,5 @@
> +            self.summary.append("\t\tThe test %s failed." % self.contents['testsToRun'][0])
> +            self.failcount += 1
> +            self.contents['loop'] = 1
> +            self.repeat -= 1
> +            if self.failcount < 3:

right here is a hardcoded assumption which we have determined in a closed box as the right solution.  Could we make this a variable, say self.maxFailures

set it up on line 16, and then add a comment explaining how it works above where it is defined?

Comment 3

[Vaibhav (:vaibhav1994)]

Attachment: intermittent.patch

Fixed the nits. Carrying over the r+

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dc5dfa48e98d

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/dc5dfa48e98d