713422 - Run reftests on Windows 7 with no hardware acceleration everywhere, not just try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

Because we have the attention span of a hey, a squirrel! a little over a year ago we got you to run Win7 reftest-no-accel on try, and then we'd get it green by ooh, shiny! and then run it everywhere.

Eventually I got tired of seeing it orange on try, and annotated the failures so it's green now. We should run it everywhere now, because otherwise we'd have to admit that even though we told Santa it was the most important thing to have ever and we'd just die without it, we were already tired of playing with it after 48 hours.

Comment 1

[Phil Ringnalda (:philor)]

Attachment: fix?

One possibility is that it's just this easy.

Another is that I've got a typo that would break everything, plus it doesn't really work that way.

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 585672 [details] [diff] [review]
fix?

Review of attachment 585672 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me. Here are the new builders that get added:

diff -u old.txt new.txt | grep -e '^[+-]'
--- old.txt	2012-01-04 08:50:12.940476325 -0500
+++ new.txt	2012-01-04 08:49:36.623639202 -0500
+Rev3 WINNT 6.1 jaegermonkey opt test reftest-no-accel
+Rev3 WINNT 6.1 profiling opt test reftest-no-accel
+Rev3 WINNT 6.1 services-central opt test reftest-no-accel
+Rev3 WINNT 6.1 shadow-central opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-beta opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-beta pgo test reftest-no-accel
+Rev3 WINNT 6.1 release-mozilla-beta opt test reftest-no-accel
+Rev3 WINNT 6.1 release-mozilla-beta pgo test reftest-no-accel
+Rev3 WINNT 6.1 accessibility opt test reftest-no-accel
+Rev3 WINNT 6.1 cedar opt test reftest-no-accel
+Rev3 WINNT 6.1 devtools opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-aurora opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-aurora pgo test reftest-no-accel
+Rev3 WINNT 6.1 maple opt test reftest-no-accel
+Rev3 WINNT 6.1 ionmonkey opt test reftest-no-accel
+Rev3 WINNT 6.1 ash opt test reftest-no-accel
+Rev3 WINNT 6.1 electrolysis opt test reftest-no-accel
+Rev3 WINNT 6.1 ux opt test reftest-no-accel
+Rev3 WINNT 6.1 places opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-release opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-release pgo test reftest-no-accel
+Rev3 WINNT 6.1 release-mozilla-release opt test reftest-no-accel
+Rev3 WINNT 6.1 release-mozilla-release pgo test reftest-no-accel
+Rev3 WINNT 6.1 elm opt test reftest-no-accel
+Rev3 WINNT 6.1 oak opt test reftest-no-accel
+Rev3 WINNT 6.1 larch opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-central opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-central pgo test reftest-no-accel
+Rev3 WINNT 6.1 holly opt test reftest-no-accel
+Rev3 WINNT 6.1 fx-team opt test reftest-no-accel
+Rev3 WINNT 6.1 build-system opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-inbound opt test reftest-no-accel
+Rev3 WINNT 6.1 mozilla-inbound pgo test reftest-no-accel

Comment 3

[Phil Ringnalda (:philor)]

Mmm, release. Have to look at when it was I annotated them and how far down the train that is.

Comment 4

[Armen [:armenzg]]

(In reply to Phil Ringnalda (:philor) from comment #0)
> Because we have the attention span of a hey, a squirrel! a little over a
> year ago we got you to run Win7 reftest-no-accel on try, and then we'd get
> it green by ooh, shiny! and then run it everywhere.
> 
> Eventually I got tired of seeing it orange on try, and annotated the
> failures so it's green now. We should run it everywhere now, because
> otherwise we'd have to admit that even though we told Santa it was the most
> important thing to have ever and we'd just die without it, we were already
> tired of playing with it after 48 hours.

Does annotate mean to turn the oranges to be green without fixing them?

Could someone from the gfx team vouch in here to say that this is what we want to do?

On another note, does not-accelerated reftests still have value as it was thought originally?

I also would like to understand why accelerated reftests (which are not accelerated for the last 9 months) and non-accelerated reftests were yielding different results even though both were *not* accelerated (since we were running on a low screen resolution).

Comment 5

[Phil Ringnalda (:philor)]

Right, annotation is the system that reftest manifests use to say what results (fail or random) are expected in particular conditions. A test may fail in a condition because a bug hasn't been fixed yet, or because it isn't ever expected to pass in that condition.

Once we have these tests running everywhere, if a given patch causes a reftest to fail, then the right response to that might be to fix the patch, or it might be to land the patch, annotate the test as failing for these conditions, file a bug, and fix the problem later, or it might be to annotate the test as never again being expected to pass under these conditions.

For the existing tests, the right thing to do is to annotate them as failing, file bugs for the failures, and not hold the other 7000 tests hostage to those few, and that's why I did that.

Of course, the other possibility is that gfx doesn't actually care whether or not they or other people break the "gfx.direct2d.disabled=true && layers.accelerate-none=true" case, and they no longer want to have Win Ru run. It's difficult to say about that, because for some reason, they never ever seem to comment in bugs, or directly answer questions. Maybe it's the graphics thing, they don't think in text, only pictures, but they can't paste a picture of them smiling next to a Win Ru log into a bug to say that they would be happy to seem them run.

Comment 6

[Phil Ringnalda (:philor)]

My annotations are on Aurora right now, so I'll probably wait until after the end of the month. Or barbeque Joe's dog in his back yard. I haven't quite decided which.

Comment 7

[Joe Drew (not getting mail)]

Joke's on you: I don't even *have* a dog.

(In reply to Phil Ringnalda (:philor) from comment #5)
> Of course, the other possibility is that gfx doesn't actually care whether
> or not they or other people break the "gfx.direct2d.disabled=true &&
> layers.accelerate-none=true" case, and they no longer want to have Win Ru
> run.

We definitely want Ru to be run, on all platforms even.

Comment 8

[Phil Ringnalda (:philor)]

Comment on attachment 585672 [details] [diff] [review]
fix?

http://hg.mozilla.org/build/buildbot-configs/rev/ee08c137ecec

Comment 9

[Phil Ringnalda (:philor)]

Started running invisibly with yesterday's reconfig, and visibly with today's. Eventually, I'll have to deal with the way esr10 isn't sufficiently annotated to run them, but for now hidden works.