User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20100101 Firefox/4.0b7 Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101108 Firefox/4.0b8pre http://brasstacks.mozilla.com/resultserv/data/results/ad80f262-f8c2-4fde-8ae0-9051ba30ec4c http://brasstacks.mozilla.com/resultserv/data/results/10caa9fc-93ee-420d-bc9a-a4ba20b97412 Works in beta 7: http://brasstacks.mozilla.com/resultserv/data/results/6c59c313-6a5a-4409-96ec-56c0d854b409 Reproducible: Always Steps to Reproduce: 1. Run Grafx Bot
spammaaja, can you find a regression window for us, since it works in beta 7? https://wiki.mozilla.org/QA/Triage#How_to_Help_with_Regressions_--_Finding_Regression_Windows
Confirming this, since I just ran the test using Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101109 Firefox/4.0b8pre and I see the same thing.
Regression window: 2010-11-06-04-mozilla-central WORKS 2010-11-07-04-mozilla-central FAILS
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2b612890113b&tochange=8cbe83542596 There were several SVG-related commits in this window - folks I am CC-ing on this comment, do you have any ideas?
So the failures that occur in both logs are: > == reset-4.svg[accelerated] green-box-ref.svg[accelerated] TEST-UNEXPECTED-FAIL > == reset-3.svg[accelerated] green-box-ref.svg[accelerated] TEST-UNEXPECTED-FAIL > == anim-svg-preserveAspectRatio-01.svg[accelerated] lime.svg[accelerated] TEST-UNEXPECTED-FAIL > == anim-feTurbulence-numOctaves-01.svg[accelerated] anim-feTurbulence-numOctaves-01-ref.svg[accelerated] TEST-UNEXPECTED-FAIL > == anim-discrete-to-2.svg[accelerated] anim-standard-ref.svg[accelerated] TEST-UNEXPECTED-FAIL > == anim-discrete-to-1.svg[accelerated] anim-standard-ref.svg[accelerated] TEST-UNEXPECTED-FAIL random-if(MOZ_WIDGET_TOOLKIT> =="gtk2") > == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] TEST-UNEXPECTED-FAIL > == marker-attribute-01.svg[accelerated] pass.svg[accelerated] TEST-UNEXPECTED-FAIL and the first log also has: > random-if(MOZ_WIDGET_TOOLKIT=="gtk2") == wordwrap-03.html[accelerated] wordwrap-03-ref.html[accelerated] TEST-UNEXPECTED-FAIL
(sorry, disregard the two un-">"-prefixed "random-if"/"wordwrap" lines in the first chunk of comment 7)
spammaaja or Marcia: Which test(s) render with a black rectangle, and which render with the black bar? (apologies if this is already noted somewhere -- I'm unfamiliar with the Grafx tool)
Aw yeah, we do need that information, which is hard to get. Jonathan, any ideas on how for us to tell which test failed and how?
The failures are here: == translate-pattern-1.svg[accelerated] lime.svg[accelerated] http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5792130/ == objectBoundingBox-and-pattern-01c.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated] http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791013/ == objectBoundingBox-and-pattern-01b.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated] http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791011/ == objectBoundingBox-and-pattern-01a.svg[accelerated] objectBoundingBox-and-pattern-01-ref.svg[accelerated] http://sm-brasstacks01.mozilla.org:8100/resultserv/data/testcase/5791009/
Thanks for the links to testcase-snapshots, jgriffin. jgriffin also points out in IRC that the failure logs from Comment 0 are from 4.0b5 and 4.0b6pre, not 4.0b8pre -- so we think the submitter might have just included the wrong (old) logs when filing. So, disregard Comment 7, as that was me excerpting the failures from those (old/incorrect) logs. This bug is *actually* about the failures mentioned in Comment 11, not those failures. So, to answer my question in Comment 9: - the first failure ("black rectangle", attachment 488891 [details]) is for all three objectBoundingBox-and-pattern-01* tests - the second failure ("black bar", attachment 488891 [details]) is for translate-pattern-1.svg
...and per bug 608653 comment 7, the problem here is that these tests made broken assumptions -- and the commit that caused the bustage also fixed the tests so that they'd continue to pass. (as noted in the final few comments on that bug.) So this seems to be a bug in Grafx Bot -- it's running up-to-date code against stale reftests.
(canceling blocking request, as this isn't actually a regression) jgriffin: is this something we can easily fix in Grafx bot? (to make sure the tests that we run are in sync with the build that they're testing?)
> jgriffin: is this something we can easily fix in Grafx bot? (to make sure the > tests that we run are in sync with the build that they're testing?) This is a little bit of a tough problem, since Grafx Bot can be run on arbitrary builds, and we don't have tests to match these except in hg. But I can add some code so that Grafx Bot would download tests from the closest matching nightly bundle, which would eliminate most of these kinds of problems.
(In reply to comment #15) > I can add some code so that Grafx Bot would download tests from the closest > matching nightly bundle, which would eliminate most of these kinds of problems. If these test-bundles can be more or less synced up with the generation of nightly builds (along with more formal release builds), then that would probably work well, since those are the builds that most Grafx users will be running. (as opposed to, say, TryServer builds or hourlies or custom local builds).
Mozilla/5.0 (X11; Linux i686; rv:2.0b8pre) Gecko/20101121 Firefox-4.0/4.0b8pre Some of the failures that I've just had on Linux are the same as these. See http://brasstacks.mozilla.com/resultserv/data/results/a1fcc312-ca2f-44c8-9b4a-466bb1c758bd Presumably the test changes were from bug 608653? Can't you just release a new version of the extension when b8 ships that has the updated tests (and any new tests) and a min-version of 4.0b8? That still means that trunk users may be running out of date tests between betas but my guess is that the amount of tests that change is not worth the effort to make Grafx Bot download tests that correspond to the date of the nightly.
Tests refreshed for 4.0b8pre in http://hg.mozilla.org/automation/grafxbot/rev/73a0f867602f. I'll work on downloadable tests if it appears that grafxbot will be used beyond FF4.0.