Grafx Bot failures: green rectangle is black, black bar in green rectangle (Grafx bot running stale reftests for up-to-date code)

RESOLVED FIXED

Status

()

Core
Graphics
--
major
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: JK, Unassigned)

Tracking

({regression})

Trunk
x86
All
regression
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

7 years ago
User-Agent:       Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20100101 Firefox/4.0b7
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101108 Firefox/4.0b8pre

http://brasstacks.mozilla.com/resultserv/data/results/ad80f262-f8c2-4fde-8ae0-9051ba30ec4c

http://brasstacks.mozilla.com/resultserv/data/results/10caa9fc-93ee-420d-bc9a-a4ba20b97412

Works in beta 7:
http://brasstacks.mozilla.com/resultserv/data/results/6c59c313-6a5a-4409-96ec-56c0d854b409

Reproducible: Always

Steps to Reproduce:
1. Run Grafx Bot
(Reporter)

Updated

7 years ago
blocking2.0: --- → ?
Version: unspecified → Trunk
(Reporter)

Comment 1

7 years ago
Created attachment 488891 [details]
Black rectangle
(Reporter)

Comment 2

7 years ago
Created attachment 488892 [details]
Black bar
(Reporter)

Updated

7 years ago
Summary: Grafx Bot failures: green rectangle is black, black line in green rectangle → Grafx Bot failures: green rectangle is black, black bar in green rectangle
(Reporter)

Updated

7 years ago
Keywords: regression
spammaaja, can you find a regression window for us, since it works in beta 7?

https://wiki.mozilla.org/QA/Triage#How_to_Help_with_Regressions_--_Finding_Regression_Windows
Keywords: regressionwindow-wanted
Confirming this, since I just ran the test using  Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101109 Firefox/4.0b8pre and I see the same thing.
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Reporter)

Comment 5

7 years ago
Regression window:

2010-11-06-04-mozilla-central	WORKS
2010-11-07-04-mozilla-central	FAILS
Keywords: regressionwindow-wanted
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2b612890113b&tochange=8cbe83542596

There were several SVG-related commits in this window - folks I am CC-ing on this comment, do you have any ideas?
So the failures that occur in both logs are:
> == reset-4.svg[accelerated] green-box-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == reset-3.svg[accelerated] green-box-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-svg-preserveAspectRatio-01.svg[accelerated] lime.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-feTurbulence-numOctaves-01.svg[accelerated] anim-feTurbulence-numOctaves-01-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-discrete-to-2.svg[accelerated] anim-standard-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
> == anim-discrete-to-1.svg[accelerated] anim-standard-ref.svg[accelerated] 	TEST-UNEXPECTED-FAIL
random-if(MOZ_WIDGET_TOOLKIT> =="gtk2") > == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] 	TEST-UNEXPECTED-FAIL
> == marker-attribute-01.svg[accelerated] pass.svg[accelerated] 	TEST-UNEXPECTED-FAIL

and the first log also has:
> random-if(MOZ_WIDGET_TOOLKIT=="gtk2") == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] 	TEST-UNEXPECTED-FAIL
(sorry, disregard the two un-">"-prefixed "random-if"/"wordwrap" lines in the first chunk of comment 7)
spammaaja or Marcia: Which test(s) render with a black rectangle, and which render with the black bar? (apologies if this is already noted somewhere -- I'm unfamiliar with the Grafx tool)
Aw yeah, we do need that information, which is hard to get. Jonathan, any ideas on how for us to tell which test failed and how?
Assignee: nobody → jgriffin
The failures are here:

== translate-pattern-1.svg[accelerated] lime.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5792130/

== objectBoundingBox-and-pattern-01c.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated] 
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791013/

== objectBoundingBox-and-pattern-01b.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791011/

== objectBoundingBox-and-pattern-01a.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated]
http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791009/
Thanks for the links to testcase-snapshots, jgriffin. jgriffin also points out in IRC that the failure logs from Comment 0 are from 4.0b5 and 4.0b6pre, not 4.0b8pre -- so we think the submitter might have just included the wrong (old) logs when filing.

So, disregard Comment 7, as that was me excerpting the failures from those (old/incorrect) logs.  This bug is *actually* about the failures mentioned in Comment 11, not those failures.

So, to answer my question in Comment 9:
 - the first failure ("black rectangle", attachment 488891 [details]) is for all three  objectBoundingBox-and-pattern-01* tests
 - the second failure ("black bar", attachment 488891 [details]) is for translate-pattern-1.svg
Assignee: jgriffin → nobody
...and per bug 608653 comment 7, the problem here is that these tests made broken assumptions -- and the commit that caused the bustage also fixed the tests so that they'd continue to pass.  (as noted in the final few comments on that bug.)

So this seems to be a bug in Grafx Bot -- it's running up-to-date code against stale reftests.
Assignee: nobody → jgriffin
Summary: Grafx Bot failures: green rectangle is black, black bar in green rectangle → Grafx Bot failures: green rectangle is black, black bar in green rectangle (Grafx bot running stale reftests for up-to-date code)
(canceling blocking request, as this isn't actually a regression)

jgriffin: is this something we can easily fix in Grafx bot? (to make sure the tests that we run are in sync with the build that they're testing?)
blocking2.0: ? → ---
> jgriffin: is this something we can easily fix in Grafx bot? (to make sure the
> tests that we run are in sync with the build that they're testing?)

This is a little bit of a tough problem, since Grafx Bot can be run on arbitrary builds, and we don't have tests to match these except in hg.  But I can add some code so that Grafx Bot would download tests from the closest matching nightly bundle, which would eliminate most of these kinds of problems.
(In reply to comment #15)
> I can add some code so that Grafx Bot would download tests from the closest
> matching nightly bundle, which would eliminate most of these kinds of problems.

If these test-bundles can be more or less synced up with the generation of nightly builds (along with more formal release builds), then that would probably work well, since those are the builds that most Grafx users will be running.  (as opposed to, say, TryServer builds or hourlies or custom local builds).

Comment 17

7 years ago
Mozilla/5.0 (X11; Linux i686; rv:2.0b8pre) Gecko/20101121 Firefox-4.0/4.0b8pre

Some of the failures that I've just had on Linux are the same as these.

See http://brasstacks.mozilla.com/resultserv/data/results/a1fcc312-ca2f-44c8-9b4a-466bb1c758bd

Presumably the test changes were from bug 608653?

Can't you just release a new version of the extension when b8 ships that has the updated tests (and any new tests) and a min-version of 4.0b8? That still means that trunk users may be running out of date tests between betas but my guess is that the amount of tests that change is not worth the effort to make Grafx Bot download tests that correspond to the date of the nightly.
OS: Windows 7 → All
Tests refreshed for 4.0b8pre in http://hg.mozilla.org/automation/grafxbot/rev/73a0f867602f.  I'll work on downloadable tests if it appears that grafxbot will be used beyond FF4.0.
Assignee: jgriffin → nobody
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.