This failed with a timeout on OS X opt RGL today: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286456060.1286456889.14338.gz However, another test run on the same changeset passed, so it is apparently an intermittent issue.
*** Bug 602505 has been marked as a duplicate of this bug. ***
For future reference, add "[orange]" to whiteboard when filing intermittent orange bugs, so that tbpl can find them. http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1286463047.1286463938.17395.gz Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test opengl on 2010/10/07 07:50:47 s: talos-r3-leopard-025
*** Bug 602519 has been marked as a duplicate of this bug. ***
also, looks like both instances of this so far have been reftest failures, not timeouts.
as philor says in duplicate bug 602519 about the reftest snapshot: > The log isn't terribly exciting: pure red instead of pure green.
dholbert%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1290201341.1290201980.899.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central opt test reftest on 2010/11/19 13:15:41 s: talos-r3-snow-027 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/mozilla-central_snowleopard_test-reftest/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
philringnalda%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1291926075.1291926705.7411.gz Rev3 MacOSX Snow Leopard 10.6.2 tracemonkey opt test reftest on 2010/12/09 12:21:15 s: talos-r3-snow-026 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/tracemonkey_snowleopard_test-reftest/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
mounir.lamouri%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1294945777.1294946899.22728.gz Rev3 Fedora 12 tryserver opt test reftest on 2011/01/13 11:09:37 s: talos-r3-fed-018 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
philringnalda%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1295477336.1295478253.6484.gz Rev3 Fedora 12 mozilla-central opt test reftest on 2011/01/19 14:48:56 s: talos-r3-fed-041 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
ehsan%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1295477336.1295478253.6484.gz Rev3 Fedora 12 mozilla-central opt test reftest on 2011/01/19 14:48:56 s: talos-r3-fed-041 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
sphink%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1297215820.1297216408.23217.gz Rev3 MacOSX Leopard 10.5.8 tracemonkey opt test reftest on 2011/02/08 17:43:40 s: talos-r3-leopard-006 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mounir.lamouri%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1298284631.1298286230.16591.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-central debug test reftest on 2011/02/21 02:37:11 s: talos-r3-snow-040 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
mwu%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Build-System/1300331115.1300332653.31657.gz Rev3 Fedora 12x64 build-system debug test reftest on 2011/03/16 20:05:15 s: talos-r3-fed64-016 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
ted.mielczarek%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=Build-System/1300331115.1300332653.31657.gz Rev3 Fedora 12x64 build-system debug test reftest on 2011/03/16 20:05:15 s: talos-r3-fed64-016 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
mounir http://tinderbox.mozilla.org/showlog.cgi?log=Cedar/1303146017.1303147433.24351.gz Rev3 Fedora 12x64 cedar debug test reftest on 2011/04/18 10:00:17 s: talos-r3-fed64-045 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
rcampbell%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Devtools/1303408518.1303410310.27856.gz Rev3 MacOSX Snow Leopard 10.6.2 devtools debug test reftest on 2011/04/21 10:55:18 s: talos-r3-snow-009 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed) PROCESS-CRASH | Main app process exited normally | application crashed (minidump found) Thread 0 (crashed)
cdleary%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1303830386.1303831210.18614.gz Rev3 MacOSX Leopard 10.5.8 tracemonkey opt test reftest on 2011/04/26 08:06:26 s: talos-r3-leopard-003 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mbrubeck%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1304961383.1304962949.10978.gz Rev3 Fedora 12x64 mozilla-central debug test reftest on 2011/05/09 10:16:23 s: talos-r3-fed64-036 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
bz%mit.edu http://tinderbox.mozilla.org/showlog.cgi?log=Try/1305513319.1305514795.12324.gz Rev3 Fedora 12 try debug test reftest on 2011/05/15 19:35:19 s: talos-r3-fed-020 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
joshmoz http://tinderbox.mozilla.org/showlog.cgi?log=Try/1305664218.1305666089.1231.gz Rev3 Fedora 12 try debug test reftest on 2011/05/17 13:30:18 s: talos-r3-fed-034 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mounir http://tinderbox.mozilla.org/showlog.cgi?log=Cedar/1306381704.1306383453.15806.gz Rev3 WINNT 5.1 cedar opt test reftest on 2011/05/25 20:48:24 s: talos-r3-xp-019 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
zackw%panix.com http://tinderbox.mozilla.org/showlog.cgi?log=Try/1306739741.1306742753.10771.gz Rev3 WINNT 6.1 try debug test reftest on 2011/05/30 00:15:41 s: talos-r3-w7-028 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mak77%bonardo.net http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1307524784.1307526351.9512.gz Rev3 Fedora 12 mozilla-central debug test reftest on 2011/06/08 02:19:44 s: talos-r3-fed-036 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mounir http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1307524784.1307526351.9512.gz Rev3 Fedora 12 mozilla-central debug test reftest on 2011/06/08 02:19:44 s: talos-r3-fed-036 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
dbaron http://tinderbox.mozilla.org/showlog.cgi?log=Try/1307920310.1307921947.17448.gz Rev3 Fedora 12 try debug test reftest on 2011/06/12 16:11:50 s: talos-r3-fed-041 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
smontagu%smontagu.org http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Aurora/1309076858.1309078427.19290.gz Rev3 MacOSX Leopard 10.5.8 mozilla-aurora debug test reftest on 2011/06/26 01:27:38 s: talos-r3-leopard-027 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
benjamin%smedbergs.us http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Aurora/1309203809.1309206764.2680.gz Rev3 WINNT 6.1 mozilla-aurora debug test reftest on 2011/06/27 12:43:29 s: talos-r3-w7-030 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
dbaron http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1309414871.1309416820.20920.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-inbound debug test reftest on 2011/06/29 23:21:11 s: talos-r3-snow-033 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
philor http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1309414871.1309416820.20920.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-inbound debug test reftest on 2011/06/29 23:21:11 s: talos-r3-snow-033 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
philor http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1309839616.1309841056.12331.gz Rev3 MacOSX Snow Leopard 10.6.2 mozilla-inbound opt test reftest on 2011/07/04 21:20:16 s: talos-r3-snow-039 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
bz%mit.edu http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Aurora/1310522681.1310525341.7477.gz Rev3 WINNT 5.1 mozilla-aurora opt test reftest on 2011/07/12 19:04:41 s: talos-r3-xp-034 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
tnikkel%gmail.com http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1311632113.1311634246.6816.gz Rev3 WINNT 6.1 mozilla-inbound opt test reftest on 2011/07/25 15:15:13 s: talos-r3-w7-007 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mbrubeck%mozilla.com http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-Inbound/1311790530.1311792173.27895.gz Rev3 Fedora 12x64 mozilla-inbound debug test reftest on 2011/07/27 11:15:30 s: talos-r3-fed64-051 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
ttaubert http://tinderbox.mozilla.org/showlog.cgi?log=Fx-Team/1313875755.1313877471.19462.gz Rev3 WINNT 6.1 x64 fx-team opt test reftest on 2011/08/20 14:29:15 s: t-r3-w764-005 REFTEST TEST-UNEXPECTED-FAIL | file:///C:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
philor https://tbpl.mozilla.org/php/getParsedLog.php?id=6246719 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test reftest on 2011-09-02 08:35:56 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
mats https://tbpl.mozilla.org/php/getParsedLog.php?id=6289190 Rev3 WINNT 5.1 try debug test reftest on 2011-09-06 04:35:08 REFTEST TEST-UNEXPECTED-FAIL | file:///c:/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
ehsan https://tbpl.mozilla.org/php/getParsedLog.php?id=6377515 Rev3 MacOSX Snow Leopard 10.6.2 mozilla-inbound opt test reftest on 2011-09-12 09:14:12 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
edmorley https://tbpl.mozilla.org/php/getParsedLog.php?id=6465577&tree=Try Rev3 Fedora 12x64 try debug test reftest on 2011-09-20 05:52:21 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
Created attachment 561246 [details] [diff] [review] Disable the test This was currently random-if(Android) but it seems like it doesn't run reliably on desktop either, so we should probably make it random everywhere.
honzab.moz%firemni.cz https://tbpl.mozilla.org/php/getParsedLog.php?id=6468520&tree=Mozilla-Inbound Rev3 MacOSX Snow Leopard 10.6.2 mozilla-inbound opt test reftest on 2011-09-20 10:14:03 REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
(In reply to Jeff Muizelaar [:jrmuizel] from comment #40) > Created attachment 561246 [details] [diff] [review] > Disable the test > > This was currently random-if(Android) but it seems like it doesn't run > reliably on desktop either, so we should probably make it random everywhere. Thanks very much for looking into this. Sorry I hadn't got to it yet. I've looked over the test case and I think what's happening is that we don't get to process the repeatEvent within the 100ms timeout used in this test. Probably what we want to do is rework the test to pass early and fail slow--i.e. add an event listener, if the animation starts end the test immediately, otherwise add a long timeout before failing. I'm pretty sure we're doing that elsewhere. I'll look into it next week. In the meantime, one "fix" is to just adjust the 100ms timeout to 250ms which should reduce the failure rate from once/twice a week to perhaps once a month or so? Not a great long-term solution though but at any rate, I'd rather do that than mark this test as random.
Created attachment 561375 [details] [diff] [review] Extend the timeout Extend the timeout. This is not the final fix but might be useful in the meantime for reducing the failure rate until a more permanent fix is done.
Try run for e775f0cd2fe9 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=e775f0cd2fe9 Results (out of 103 total builds): success: 87 warnings: 15 failure: 1 Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bbirtles@mozilla.com-e775f0cd2fe9
Created attachment 562627 [details] [diff] [review] Proposed patch v1a This patch implements the solution indicated in comment 42, i.e. detect the success case and end right away, or wait longer for the failure case.
Comment on attachment 561246 [details] [diff] [review] Disable the test Cancelling review for the patch to disable the test since I think we can fix this.
Comment on attachment 562627 [details] [diff] [review] Proposed patch v1a r=me, with two nits: - I worry that the 5 second timeout might be short enough that it could sporadically fail on really crappy hardware with GC pauses & other stuff going on. Could we up that to 30 or 60 seconds? - Could you declare the timeout amount as a constant at the top of the JS block, so it's clear what it's there for (and easy to tweak if we want to increase / decrease it)? Thanks!
Created attachment 562629 [details] [diff] [review] Proposed patch v1b Address review feedback (comment 47).
Created attachment 562630 [details] [diff] [review] Proposed patch v1c; r=dholbert Forgot to add the r=dholbert to the check-in comment
I've sent an older version of this test off to Try so I'll wait to check that everything is ok there before landing this.
Try run for 3e287957617f is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=3e287957617f Results (out of 33 total builds): success: 30 warnings: 3 Builds available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/bbirtles@mozilla.com-3e287957617f
Pushed patch to m-i: https://hg.mozilla.org/integration/mozilla-inbound/rev/3548d5c7cde5 I'd like to leave this bug open for about 6 weeks though to be sure this has fixed it.
https://hg.mozilla.org/mozilla-central/rev/3548d5c7cde5
mbrubeck%mozilla.com https://tbpl.mozilla.org/php/getParsedLog.php?id=7029989&tree=Mozilla-Aurora Rev3 Fedora 12x64 mozilla-aurora debug test reftest on 2011-10-25 18:09:48 REFTEST TEST-UNEXPECTED-FAIL | file:///home/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/svg/smil/event/event-begin-timeevent-3.svg | image comparison (==)
Ignoring comment 54 since it's on aurora which is moz 9 and the patch I'm hoping fixes this has target milestone moz 10.
It's six weeks now since this landed and there have been no failures (on any branch where this has landed) so I'm pretty confident this is fixed.
or
