1128211 - Reduce the reftest resolution on all platforms

Status: NEW

(Testing :: Reftest, defect)

Description

[Seth Fowler [:seth] [:s2h]]

In bug 737961, we annotated the reftests that fail on small screens, but we need to go further and actually change the default reftest resolution.

The current reftest resolution is 800x100, which is far too large. Many modern laptops (including all Apple laptops) have a vertical resolution of 900 pixels by default, which causes many reftest failures when running locally because we use a different code path in the layer system (this is what the USE_WIDGET_LAYERS warning you get in this situation means). Even when the reftest doesn't fail, the fact that we use a different code path is problematic enough; it means that developers running their tests locally are not running the same code that we run on our CI infrastructure.

I've seen the number 600x600 proposed before, and it seems like a reasonable choice to me. Let's fix tests which depend on a larger resolution and move forward with changing the default.

I should note that in bug 737961, there was concern about cooperation with other browser vendors, since apparently the reftest resolution is a shared standard. If that continues to block this work, we need to push forward there as well.

Comment 1

[Jesse Ruderman]

Does this depend on bug 795022 and bug 795278?

Comment 2

[Seth Fowler [:seth] [:s2h]]

(In reply to Jesse Ruderman from comment #1)
> Does this depend on bug 795022 and bug 795278?

Looks like it, yeah.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

oh crazy- maybe we can get those patches reviewed after all these years!

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

We also need to go through the tests and make sure not only that they still pass, but also that they're still testing what they were intended to test.  This requires manual examination.

It should probably be done one directory at a time -- which requires that the reftest harness be able to run different directories at different sizes.  (Did someone implement that somewhere?)

Comment 5

[Seth Fowler [:seth] [:s2h]]

(In reply to David Baron [:dbaron] ⏰UTC-4 (busy, returning May 21) from comment #4)
> It should probably be done one directory at a time -- which requires that
> the reftest harness be able to run different directories at different sizes.
> (Did someone implement that somewhere?)

I haven't seen that implemented. I guess we could read the reftest resolution from a pref and piggyback on default-preferences? That seems like the most straightforward solution.

Comment 6

[Sam Sneddon [:gsnedders]]

(In reply to Seth Fowler [:seth] [:s2h] from comment #0)
> I've seen the number 600x600 proposed before, and it seems like a reasonable
> choice to me. Let's fix tests which depend on a larger resolution and move
> forward with changing the default.
> 
> I should note that in bug 737961, there was concern about cooperation with
> other browser vendors, since apparently the reftest resolution is a shared
> standard. If that continues to block this work, we need to push forward
> there as well.


See https://github.com/w3c/web-platform-tests/issues/7598 when it comes to size for screenshots in web-platform-tests; stuff is currently inconsistent as to whether 600x600 or 800x600 is used. Given everything except wptrunner currently uses 800-wide screenshots, I'd favour standardising on 800x600.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:tnikkel, could you have a look please?
For more information, please visit auto_nag documentation.