Closed Bug 771150 Opened 8 years ago Closed 6 years ago

Eideticker failed to pick up checker boarding regression caused by dlbi

Categories

(Testing Graveyard :: Eideticker, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: jrmuizel, Unassigned)

References

Details

A large regression is clear on tcheckerboard. I don't really see the same on eideticker.

http://graphs.mozilla.org/graph.html#tests=[[175,63,20]]&sel=1340894474844,1341499274844&displayrange=7&datatype=running
(In reply to Jeff Muizelaar [:jrmuizel] from comment #0)
> A large regression is clear on tcheckerboard. I don't really see the same on
> eideticker.
> 
> http://graphs.mozilla.org/graph.html#tests=[[175,63,20]]&sel=1340894474844,
> 1341499274844&displayrange=7&datatype=running

tcheckerboard tests against a copy of timecube.com, so I guess that really means we should add a test against a copy of that to the eideticker suite.

If you look at another checkerboarding benchmark (tcheck2) which tests against CNN you see that there was no regression there either: 

http://graphs.mozilla.org/graph.html#tests=[[175,63,20],[201,63,20]]&sel=1340894474844,1341499274844&displayrange=7&datatype=running
Summary: Eideticker failed to pick up checker boarding regression caused by dlbi → Should add test for timecube.com
I believe the same regression should be happening on the task.js page and it doesn't really show up there either.
(In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> I believe the same regression should be happening on the task.js page and it
> doesn't really show up there either.

Hmm, any ideas why that would be the case? I'm pretty sure there's no issue with the HDMI capture or analysis code.
(In reply to William Lachance (:wlach) from comment #3)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> > I believe the same regression should be happening on the task.js page and it
> > doesn't really show up there either.
> 
> Hmm, any ideas why that would be the case? I'm pretty sure there's no issue
> with the HDMI capture or analysis code.

It could be that presentation of the data just makes it hard to see changes. i.e. You can't just look at data from the last couple of days.

It also seems like the tests are not run often enough. There seems to be no data between 6/30 and 7/4 on cnn.com and reddit
Summary: Should add test for timecube.com → Eideticker failed to pick up checker boarding regression caused by dlbi
(In reply to Jeff Muizelaar [:jrmuizel] from comment #4)
> (In reply to William Lachance (:wlach) from comment #3)
> > (In reply to Jeff Muizelaar [:jrmuizel] from comment #2)
> > > I believe the same regression should be happening on the task.js page and it
> > > doesn't really show up there either.
> > 
> > Hmm, any ideas why that would be the case? I'm pretty sure there's no issue
> > with the HDMI capture or analysis code.
> 
> It could be that presentation of the data just makes it hard to see changes.
> i.e. You can't just look at data from the last couple of days.

I just added support to zoom in on a particular region of an Eideticker graph on the dashboard, let me know if it helps. I'm not seeing much of a pattern on the dashboard, but as you say, we are missing some data.

> It also seems like the tests are not run often enough. There seems to be no
> data between 6/30 and 7/4 on cnn.com and reddit

There's still issues of stability with the test harness. I think I've nailed down at least a few issues recently, but some still remain. I now have a tool to run eideticker against a particular range of builds -- I'll try running that for taskjs to see if it caught the regression.
(In reply to William Lachance (:wlach) from comment #5)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #4)
> > (In reply to William Lachance (:wlach) from comment #3)
> > It also seems like the tests are not run often enough. There seems to be no
> > data between 6/30 and 7/4 on cnn.com and reddit
> 
> There's still issues of stability with the test harness. I think I've nailed
> down at least a few issues recently, but some still remain.

I just filed bug 771375 about the dashboard update script's flakiness. Hopefully when this is fixed the dashboard update process should be more consistent.
This bug is pretty old, and I don't think there's enough specific actionable information to go anywhere with it.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INVALID
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.