The default bug view has changed. See this FAQ.

sporadic REFTEST TEST-UNEXPECTED-FAIL | reftest/tests/layout/reftests/bugs/598726-1.html | image comparison (==)

RESOLVED FIXED in mozilla12

Status

()

Core
CSS Parsing and Computation
RESOLVED FIXED
5 years ago
4 years ago

People

(Reporter: dholbert, Assigned: fryn)

Tracking

(Depends on: 1 bug, {intermittent-failure})

Trunk
mozilla12
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(4 attachments, 1 obsolete attachment)

(Reporter)

Description

5 years ago
Created attachment 574229 [details]
reftest log from comment 0 (57px diff, before frequent failures started)

https://tbpl.mozilla.org/php/getParsedLog.php?id=7379176&tree=Mozilla-Inbound
Rev3 MacOSX Leopard 10.5.8 mozilla-inbound debug test reftest on 2011-11-13 17:44:50 PST for push 9f2d2db76c57
{
REFTEST TEST-UNEXPECTED-FAIL | file:///Users/cltbld/talos-slave/test/build/reftest/tests/layout/reftests/bugs/598726-1.html | image comparison (==)
}

The difference isn't visually noticeable (to me at least) -- it's a 53px difference, and it seems to be in antialiasing along the edges of the characters in the text field.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
OS: Windows 7 → All
Hardware: x86_64 → All
(Reporter)

Comment 177

5 years ago
Timeline of this bug:

* 2011-01-19: This test was checked in
    (...10 months elapse...)
* 2011-11-13: random orange reported (comment 0)
    (...1 month elapses...)
* 2011-12-20: second instance of this random orange (comment 2), and now it starts failing ~13 times per day, continuing up to the present.

So apparently something changed to make this really bad on (or just before) 12/20.

(note: I'm explicitly ignoring comment 1, because it's not this bug -- it's a timeout)

Also: glancing at the last few logs' reftest snapshots, the difference is much more substantial (and "real bug" looking) than comment 0.

Comment 0 just had 53px different, and they each are e.g. rgb(n,n,n) vs rgb(n+1,n+1,n+1) antialiased edges of text. (not visually noticeable)  In contrast, the most recent few comments have hundreds to thousands of pixels different, and it's very visually noticeable -- the text box in the testcase has a different size / position / styling (depending on the platform).
So seems like the problem here is that the input box is not correctly painted by the time that the transitionend event is fired.  Can you guys imagine why?
(Reporter)

Comment 179

5 years ago
The "guilty" pushlog (on m-i) doesn't suggest anything obvious...

The earliest report in the spammy-range was at 2011-12-20 7:25 AM, and it failed 4 times in the ~2 hours thereafter.  Just to be super-generous, here's the pushlog from 8am the previous day up to 8am that day:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?startdate=2011-12-19+08%3A00%3A00&enddate=2011-12-20+08%3A00%3A00

Nothing jumps out at me there.
Comment hidden (Treeherder Robot)
(Reporter)

Comment 181

5 years ago
Hmm, http://hg.mozilla.org/integration/mozilla-inbound/rev/21b0f339bf52 was a non-trivial change to the reftest harness itself at 4:46 that morning. I wonder if that might have been responsible?

http://hg.mozilla.org/integration/mozilla-inbound/rev/7f8c69ad0895 might also be relevant... (only if nsGfxScrollFrame gets used at all for simple <input> text boxes... I forget if it does)

I think we can also ignore the m-c to m-i merges in that pushlog.  If this had come from a m-c merge, then we would've gotten spammy-report from m-c first -- but we didn't.
(Reporter)

Updated

5 years ago
Attachment #574229 - Attachment description: reftest log → reftest log from comment 0 (before this got spammy)
CCing jmaher, author of http://hg.mozilla.org/integration/mozilla-inbound/rev/21b0f339bf52.
(Reporter)

Comment 183

5 years ago
Created attachment 585639 [details]
reftest log from comment 180 (888px diff, fails frequently)
(Reporter)

Updated

5 years ago
Attachment #574229 - Attachment description: reftest log from comment 0 (before this got spammy) → reftest log from comment 0 (57px diff, before frequent failures started)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 189

5 years ago
(In reply to Daniel Holbert [:dholbert] from comment #181)
> Hmm, http://hg.mozilla.org/integration/mozilla-inbound/rev/21b0f339bf52 was
> a non-trivial change to the reftest harness itself at 4:46 that morning. I
> wonder if that might have been responsible?
(In reply to Ehsan Akhgari [:ehsan] from comment #182)
> CCing jmaher, author of
> http://hg.mozilla.org/integration/mozilla-inbound/rev/21b0f339bf52.

I think that one's innocent, actually.  I just did 2 TryServer pushes, for that cset and its parent (b52b4576f9f1), and gave both pushes tons of reftest runs.  The parent cset has already hit this failure on 3/13 WinXP Debug runs, which I think is enough to classify it in the "spammy" range -- hence, its child 21b0f339bf52 from jmaher is innocent.

Hence, un-CC'ing jmaher to spare him the tbplbot mail.

For reference, the TryServer run for that parent cset is:
  https://tbpl.mozilla.org/?tree=Try&rev=3ca52aea0573
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 585938 [details] [diff] [review]
wallpaper

Here's a wallpaper that I think would cure the orange problem.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 211

5 years ago
(In reply to Daniel Holbert [:dholbert] from comment #179)
> The earliest report in the spammy-range was at 2011-12-20 7:25 AM, and it
> failed 4 times in the ~2 hours thereafter.  Just to be super-generous,
> here's the pushlog from 8am the previous day up to 8am that day:
> http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?startdate=2011-
> 12-19+08%3A00%3A00&enddate=2011-12-20+08%3A00%3A00

I did a few more try pushes from that range, and I got a lucky winner - Mats' push for bug 619273:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=0aa9c3a5b7e1

Try run from that push (0aa9c3a5b7e1):
  https://tbpl.mozilla.org/?tree=Try&rev=7d2060424376

Try run from its parent (feaccb6a4c35):
  https://tbpl.mozilla.org/?tree=Try&rev=a6cecf97e645

I gave both of the above Try runs 40-50 reftest jobs on WinXP Debug.
- On the blamed 0aa9c3a5b7e1 push, 11 of those jobs suffered from this bug.
- On the parent feaccb6a4c35 push, 0  of those jobs suffered from this bug.
That seems pretty definitive to me.

Hence, setting this as blocking as blocking bug 619273.  (While comment 0 was from before that bug landed, the remainder of the reports here are from the "spammy range" after it landed, so I think it's reasonable to consider this largely a regression from bug 619273.)
Blocks: 619273
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to Daniel Holbert [:dholbert] from comment #211)
> so I think it's reasonable to consider
> this largely a regression from bug 619273.)

Yeah, I can see how bug 619273 could have affected this test through the
auto-selection of the <input> value on focus.
I doubt there's a bug in the code though, the test works fine with the
attached wallpaper.  I did ~50 reftest runs on it yesterday without
any orange.

Updated

5 years ago
Attachment #585938 - Flags: review?(dholbert)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 219

5 years ago
Comment on attachment 585938 [details] [diff] [review]
wallpaper

ehsan's probably a better reviewer for this, as he wrote the test being tweaked.
Attachment #585938 - Flags: review?(dholbert) → review?(ehsan)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment on attachment 585938 [details] [diff] [review]
wallpaper

Does the orange happen without the 100ms timeouts?  If you expect the flushing to help, we should really keep the timeouts 0.  If not, then this will only make the orange happen less frequently, I think.
Attachment #585938 - Flags: review?(ehsan) → review-
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 586540 [details] [diff] [review]
wallpaper, v2

Yeah, that works - none of the oranges are for this bug:
https://tbpl.mozilla.org/?tree=Try&rev=b70564cb4ef5
Attachment #585938 - Attachment is obsolete: true
Attachment #586540 - Flags: review?(ehsan)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment on attachment 586540 [details] [diff] [review]
wallpaper, v2

Review of attachment 586540 [details] [diff] [review]:
-----------------------------------------------------------------

r=me, thanks!
Attachment #586540 - Flags: review?(ehsan) → review+
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://hg.mozilla.org/integration/mozilla-inbound/rev/0f5d650bf2eb
Whiteboard: [orange] → [orange][inbound]
Target Milestone: --- → mozilla12
Comment hidden (Treeherder Robot)
FTR, comment 303 is for a build before the fix landed.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://hg.mozilla.org/mozilla-central/rev/0f5d650bf2eb

\o/
Assignee: nobody → matspal
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
https://tbpl.mozilla.org/php/getParsedLog.php?id=8637879&tree=Mozilla-Aurora
Whiteboard: [orange][inbound] → [orange]
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
This just reappeared

https://tbpl.mozilla.org/php/getParsedLog.php?id=9381057&tree=Mozilla-Inbound
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)