Open Bug 598879 Opened 9 years ago Updated 5 months ago

Create reftest-style framework for comparing <browser> and <browser remote> rendering

Categories

(Testing :: Reftest, defect)

x86_64
Linux
defect
Not set

Tracking

(blocking2.0 -)

Tracking Status
blocking2.0 --- -

People

(Reporter: cjones, Unassigned)

References

Details

Ideally, we would modify the reftest harness so as to allow us to compare four renderings instead of two for each test T.  We would want to run the normal reftest process on T and and its reference T_r once for <browser> and again for <browser remote>.  This would yield renderings T, T remote, T_r, and T_r remote.  We could gather all four in parallel, in theory.  Then, for simplicity, assume we want T's reference T_r to be != to the test T.  We would check
  T   == T remote
  T_r == T_r remote
  T   != T_r

It would be nice to run this continuously on all platforms as extension of normal reftests.  It's unclear what perf overhead this would incur.

It's not totally obvious how the browser process would know that the content process was done painting.  However, Rob and I discussed this a bit at the work week, and it seems that we could probably just load the reftest script into the content process and re-use pretty much all the logic in there.
(Hm, no blocking-fennec flag?)

IMHO this needs to block fennec-betaN.  We really really need to be continuously running tests of all the APIs we've added and of shadow-layer rendering.
blocking2.0: --- → ?
I was going to suggest adding a --remote pref that ran reftest with all the drawing taking place in a content process, but that's not quite what you want, is it? You want to compare local rendering vs. remote rendering.

Is it overkill to run this as part of regular reftest? Do you expect that people hacking layout will expose bugs in remote rendering, or do you think that it's more likely to be other things, and reftest is just a convenient seed of "try drawing a lot of things"? It feels like it'd be odd to start adding tests to reftest for bugs in the remote rendering, if they don't really test bugs in our layout engine so much as our content process rendering.
(In reply to comment #2)
> I was going to suggest adding a --remote pref that ran reftest with all the
> drawing taking place in a content process, but that's not quite what you want,
> is it? You want to compare local rendering vs. remote rendering.
> 

Right.

> Is it overkill to run this as part of regular reftest?

Possibly, experience would tell.

> Do you expect that
> people hacking layout will expose bugs in remote rendering, or do you think
> that it's more likely to be other things, and reftest is just a convenient seed
> of "try drawing a lot of things"?

I don't understand why it's "either or"; both can happen, there are lots of ways to break remote rendering.

> It feels like it'd be odd to start adding
> tests to reftest for bugs in the remote rendering, if they don't really test
> bugs in our layout engine so much as our content process rendering.

We do need reftests to check new APIs we've added for fennec with layout implementations that are different for local vs. remote browser (by necessity), regardless of whether we support comparing *all* reftests between local vs. remote.  But we use reftests to check HW-accelerated rendering, even though HW accel has nothing to do with layout, and it's been fabulously useful.  That's the idea I have in mind.
Also FWIW I believe Rob proposed something similar for comparing SW rendering vs. HW-accelerated rendering, which would have been very cool, but unfortunately there are various GPU details that make that infeasible.
I'm not sure we actually need to compare <browser> against <browser remote> rendering. If we just run reftests and take snapshots of the chrome window containing <browser remote>, isn't that going to exercise the shadow layer system pretty well?
In other words, just run reftests using <browser remote> on mobile.
(In reply to comment #5)
> I'm not sure we actually need to compare <browser> against <browser remote>
> rendering. If we just run reftests and take snapshots of the chrome window
> containing <browser remote>, isn't that going to exercise the shadow layer
> system pretty well?

Sure, but that leaves open the possibility for all sorts of glitches in remote=true that don't show up for remote=false.

To be clear, I filed this bug to cover making a system that allows me to compare remote=true to remote=false renderings, maybe re-using the reftest framework.  Whether or not we run this on tinderbox or for what platforms is a totally separate discussion.
(In reply to comment #7)
> (In reply to comment #5)
> > I'm not sure we actually need to compare <browser> against <browser remote>
> > rendering. If we just run reftests and take snapshots of the chrome window
> > containing <browser remote>, isn't that going to exercise the shadow layer
> > system pretty well?
> 
> Sure, but that leaves open the possibility for all sorts of glitches in
> remote=true that don't show up for remote=false.

So you're worried about bugs that affect the test and reference the same way on every single test, and therefore aren't detected at all?

That's already a potential problem with reftests, which we're not bothered by.

> To be clear, I filed this bug to cover making a system that allows me to
> compare remote=true to remote=false renderings, maybe re-using the reftest
> framework.  Whether or not we run this on tinderbox or for what platforms is a
> totally separate discussion.

OK.
I'm worried about bugs that would affect a particular test+reference the same for remote but not local browser.  Clipping, snapping, etc. of shadow trees e.g.
so we really want to just load the test and reference in a browser=true environment while having the core reftest extension running in chrome.  Then we can snapshot and compare in chromeland.

This would be reftest only?  What about jsreftests and crashtests?
(In reply to comment #10)
> so we really want to just load the test and reference in a browser=true
> environment while having the core reftest extension running in chrome.  Then we
> can snapshot and compare in chromeland.
> 

For <browser remote>, yes that's mostly true.  We'll need the reftest code that detects when the page load/animation has reached a snapshot-able state to live in the content process, though.

> This would be reftest only?  What about jsreftests and crashtests?

I don't know much about jsreftests and crashtests, but if they're not comparing renderings, then just running them with something like the --remote option Ted suggested in comment 2 should be enough.
Filed bug 615386 on the reftest --remote-style framework mentioned in comment 2.
blocking2.0: ? → -
You need to log in before you can comment on or make changes to this bug.