610406 - Grafx Bot failures: green rectangle is black, black bar in green rectangle (Grafx bot running stale reftests for up-to-date code)

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[JK]

User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20100101 Firefox/4.0b7
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101108 Firefox/4.0b8pre

http://brasstacks.mozilla.com/resultserv/data/results/ad80f262-f8c2-4fde-8ae0-9051ba30ec4c

http://brasstacks.mozilla.com/resultserv/data/results/10caa9fc-93ee-420d-bc9a-a4ba20b97412

Works in beta 7:
http://brasstacks.mozilla.com/resultserv/data/results/6c59c313-6a5a-4409-96ec-56c0d854b409

Reproducible: Always

Steps to Reproduce:
1. Run Grafx Bot

Comment 1

[JK]

Attachment: Black rectangle

Comment 2

[JK]

Attachment: Black bar

Comment 3

[Joe Drew (not getting mail)]

spammaaja, can you find a regression window for us, since it works in beta 7?

https://wiki.mozilla.org/QA/Triage#How_to_Help_with_Regressions_--_Finding_Regression_Windows

Comment 4

[Marcia Knous [:marcia]]

Confirming this, since I just ran the test using  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101109 Firefox/4.0b8pre and I see the same thing.

Comment 5

[JK]

Regression window:

2010-11-06-04-mozilla-central	WORKS
2010-11-07-04-mozilla-central	FAILS

Comment 6

[Joe Drew (not getting mail)]

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2b612890113b&tochange=8cbe83542596

There were several SVG-related commits in this window - folks I am CC-ing on this comment, do you have any ideas?

Comment 7

[Daniel Holbert [:dholbert]]

So the failures that occur in both logs are:
> == reset-4.svg[accelerated] green-box-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == reset-3.svg[accelerated] green-box-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-svg-preserveAspectRatio-01.svg[accelerated] lime.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-feTurbulence-numOctaves-01.svg[accelerated] anim-feTurbulence-numOctaves-01-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-discrete-to-2.svg[accelerated] anim-standard-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-discrete-to-1.svg[accelerated] anim-standard-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
random-if(MOZ_WIDGET_TOOLKIT> =="gtk2") > == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] 	TEST-UNEXPECTED-FAIL
> == marker-attribute-01.svg[accelerated] pass.svg[accelerated] 	TEST-UNEXPECTED-FAIL

and the first log also has:
> random-if(MOZ_WIDGET_TOOLKIT=="gtk2") == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] 	TEST-UNEXPECTED-FAIL

Comment 8

[Daniel Holbert [:dholbert]]

(sorry, disregard the two un-">"-prefixed "random-if"/"wordwrap" lines in the first chunk of comment 7)

Comment 9

[Daniel Holbert [:dholbert]]

spammaaja or Marcia: Which test(s) render with a black rectangle, and which render with the black bar? (apologies if this is already noted somewhere -- I'm unfamiliar with the Grafx tool)

Comment 10

[Joe Drew (not getting mail)]

Aw yeah, we do need that information, which is hard to get. Jonathan, any ideas on how for us to tell which test failed and how?

Comment 11

[Jonathan Griffin (:jgriffin)]

The failures are here:

== translate-pattern-1.svg[accelerated] lime.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5792130/

== objectBoundingBox-and-pattern-01c.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated] 
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791013/

== objectBoundingBox-and-pattern-01b.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791011/

== objectBoundingBox-and-pattern-01a.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791009/

Comment 12

[Daniel Holbert [:dholbert]]

Thanks for the links to testcase-snapshots, jgriffin. jgriffin also points out in IRC that the failure logs from Comment 0 are from 4.0b5 and 4.0b6pre, not 4.0b8pre -- so we think the submitter might have just included the wrong (old) logs when filing.

So, disregard Comment 7, as that was me excerpting the failures from those (old/incorrect) logs.  This bug is *actually* about the failures mentioned in Comment 11, not those failures.

So, to answer my question in Comment 9:
 - the first failure ("black rectangle", attachment 488891 [details]) is for all three  objectBoundingBox-and-pattern-01* tests
 - the second failure ("black bar", attachment 488891 [details]) is for translate-pattern-1.svg

Comment 13

[Daniel Holbert [:dholbert]]

...and per bug 608653 comment 7, the problem here is that these tests made broken assumptions -- and the commit that caused the bustage also fixed the tests so that they'd continue to pass.  (as noted in the final few comments on that bug.)

So this seems to be a bug in Grafx Bot -- it's running up-to-date code against stale reftests.

Comment 14

[Daniel Holbert [:dholbert]]

(canceling blocking request, as this isn't actually a regression)

jgriffin: is this something we can easily fix in Grafx bot? (to make sure the tests that we run are in sync with the build that they're testing?)

Comment 15

[Jonathan Griffin (:jgriffin)]

> jgriffin: is this something we can easily fix in Grafx bot? (to make sure the
> tests that we run are in sync with the build that they're testing?)

This is a little bit of a tough problem, since Grafx Bot can be run on arbitrary builds, and we don't have tests to match these except in hg.  But I can add some code so that Grafx Bot would download tests from the closest matching nightly bundle, which would eliminate most of these kinds of problems.

Comment 16

[Daniel Holbert [:dholbert]]

(In reply to comment #15)
> I can add some code so that Grafx Bot would download tests from the closest
> matching nightly bundle, which would eliminate most of these kinds of problems.

If these test-bundles can be more or less synced up with the generation of nightly builds (along with more formal release builds), then that would probably work well, since those are the builds that most Grafx users will be running.  (as opposed to, say, TryServer builds or hourlies or custom local builds).

Comment 17

[Daniel C]

Mozilla/5.0 (X11; Linux i686; rv:2.0b8pre) Gecko/20101121 Firefox-4.0/4.0b8pre

Some of the failures that I've just had on Linux are the same as these.

See http://brasstacks.mozilla.com/resultserv/data/results/a1fcc312-ca2f-44c8-9b4a-466bb1c758bd

Presumably the test changes were from bug 608653?

Can't you just release a new version of the extension when b8 ships that has the updated tests (and any new tests) and a min-version of 4.0b8? That still means that trunk users may be running out of date tests between betas but my guess is that the amount of tests that change is not worth the effort to make Grafx Bot download tests that correspond to the date of the nightly.

Comment 18

[Jonathan Griffin (:jgriffin)]

Tests refreshed for 4.0b8pre in http://hg.mozilla.org/automation/grafxbot/rev/73a0f867602f.  I'll work on downloadable tests if it appears that grafxbot will be used beyond FF4.0.