1311329 - ACID3 test 35 failing under aurora and nightly

Status: RESOLVED INVALID

(Web Compatibility :: Site Reports, defect, P1)

Description

[Bill Gianopoulos [:WG9s]]

Under both Aurora and nightly ACID3 test 35 is failing with this error:

Test 35 failed: expected '0' but got '1' - root element, with no parent node, claims to be a :first-child

Current relase and Beta are NOT affected.

Comment 1

[Bill Gianopoulos [:WG9s]]

This fails under WIndows as well.

Comment 2

[Mats Palmgren (inactive)]

Is this an intentional change from bug 1300374?

Comment 3

[Bill Gianopoulos [:WG9s]]

(In reply to Mats Palmgren (:mats) from comment #2)
> Is this an intentional change from bug 1300374?

If this is an intentional change, and the reasons for the change are legitimate, then we should ask to have the test changed.  That said, Google Chrome still gets 100% on this test.  I will try backing out that change to see how it impacts the test results.

Comment 4

[Bill Gianopoulos [:WG9s]]

I have verified that restoring this code:

if (!parent) {
    return false;
}

from edgeChildMatches

which was removed by the patch for bug 1300374 does restore the 100% result in the Acid3 test.

I also wonder why this was not known , if it was not, because I thought we included the Acid3 test in the our test suite.

Comment 5

[Emilio Cobos Álvarez (:emilio)]

Yes, this is intentional, since the spec changed. If it breaks stuff we can of course restore the old behavior (though I won't say Acid3 is representative of actual content, and this is the first report related to that bug AFAIK).

I think we should get the test changed, but I don't know how feasible is it.

Also, no, I don't think we include Acid3 in our test suite (otherwise that patch would've never landed).

Comment 6

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Fixing Acid3 has historically not been an issue. I asked Hixie <https://lists.w3.org/Archives/Public/www-archive/2016Oct/0005.html>.

Comment 7

[Astley Chen (inactive)]

(In reply to Emilio Cobos Álvarez [:emilio] from comment #5)
> I think we should get the test changed, but I don't know how feasible is it.
Will this change cause any webcompat issue here between Blink and us ? Following the spec is correct but still concerned about breakage of any websites due to compatibility issues.

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Hi Astley,

As I said before, I don't think Acid3 is representative of actual content, so I don't think sites rely on this. 

That being said, yes, it's a difference, but this is the first report related to that change in two months, and it's in a fairly artificial test.

Comment 9

[Astley Chen (inactive)]

So could we change to Tech Evangelism team to help the fix on ACID3 test cases?

Comment 10

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

No need. I'll try to poke again this week.

Comment 11

[Bill Gianopoulos [:WG9s]]

Problem is what to recommend doing with the test.  I would hate to have another test out of this suite end up being an everyone passes, yet if we change the test to expect the new behavior and make the old behavior fail, we end up needing to uplift bug 1300374 everywhere including ESR, otherwise we end up with release builds failing ACID3.  I am sure other browser vendors are int he same boat here.  Permitting both behaviors is kind of the same as making it an everyone passes test.  Can we suggest somehow putting in a current date check to permit old behavior as pass before some future date and make new behavior pass?

Comment 12

[Astley Chen (inactive)]

(In reply to Bill Gianopoulos [:WG9s] from comment #11)
> Can we suggest somehow putting
> in a current date check to permit old behavior as pass before some future
> date and make new behavior pass?

It's not possible to uplift the fix to all releases.
Instead, I think it all depends on how the test case would be updated in order to fit the spec change.

Comment 13

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://github.com/w3c/csswg-drafts/issues/695

Comment 14

[neb9]

I can confirm that this applies equally to the current aurora build of SeaMonkey 2.49a2.

Comment 15

[Astley Chen (inactive)]

Before there is any response or conclusion from WG, do we really want this behavior change being shipped along the way to release channel users ? What's the pros to keep this change even though there might be some webcompat issues hitting us in the future?

Comment 16

[Emilio Cobos Álvarez (:emilio)]

Tab seems to agree that this is the way forward to do this[1], and I'm trying to get the change landed on Blink too.

[1]: https://github.com/w3c/csswg-drafts/issues/695#issuecomment-263930658

Comment 17

[Boris Zbarsky [:bzbarsky]]

For what it's worth, we (and other browsers) used to have our current behavior until Acid3 was published, iirc.  Then we changed to make acid3 pass in this one weird edge case.

The right way to fix the test is to remove this part of it completely, imo.

Comment 18

[Anne (:annevk)]

As I mentioned in bug 1300374 comment 21, I replied to Ian: https://lists.w3.org/Archives/Public/www-archive/2017Jan/0014.html.

Comment 19

[Emilio Cobos Álvarez (:emilio)]

FYI, I landed the equivalent patch on Blink[1], so it shouldn't be considered a webcompat issue anymore.

[1]: https://codereview.chromium.org/2588643004/

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Sounds like this is wontfix for 52+ then, Boris?

Comment 21

[Boris Zbarsky [:bzbarsky]]

Yes, I think so.  We should keep pushing on Ian, but if it's not just us making the change that makes it much simpler to make the case the test is broken if he refuses to change it.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Calling this WONTFIX from a Firefox standpoint and moving the bug over to Tech Evangelism at this point, then. Thanks!

Comment 25

[Vader]

"Calling this WONTFIX from a Firefox standpoint and moving the bug over to TECH EVANGELISM at this point, then. Thanks!"
Well, this is embarrassing...

Comment 26

[Boris Zbarsky [:bzbarsky]]

What exactly is embarrassing?  We're following the spec.  The test is wrong per current spec text, but the test author refuses to change it.  Who is supposed to be embarrassed and why?

Comment 27

[Vader]

Where is bug in the test and where is this specification, that specifies 1 instead 0? If the test is wrond the title shall be "INVALID instead of tech evangelism. The test 72 is also a bug in test benchmark? 

What is embarrassing? Race Hazard in Firefox. Please see topic for test 72 and please multiple run the test. Why the result is 98+(rand()%2) instead of stable 99 or 98?

Comment 28

[Boris Zbarsky [:bzbarsky]]

> Where is bug in the test and where is this specification, that specifies 1 instead 0?

Did you read this bug report?  That's covered in the links in the first several comments.

> The test 72 is also a bug in test benchmark? 

This bug report is about test 35.  It has nothing to do with test 72.

That said, I just tried in Firefox 54 release and I can't reproduce any test 72 failures; I get a stable score of 99 over dozens of runs.  If you can reproduce a problem there, please file a separate bug with steps to reproduce and cc me.

Comment 29

[marius.spix]

How can I find out which test is failing? On my FF 53.0.3, the test result is 99%, the third bucket is gray. Hovering over the A converts the cursor to a help cursor, this state keeps permanently if you click on the A. No dialog appears, however and refreshing the page gives an 1% score. This is very weird.

Comment 30

[Boris Zbarsky [:bzbarsky]]

Please stop asking all possible Acid3 questions in this bug...  But in general, clicking the A is supposed to show an alert that explains which tests failed, and does for me on Firefox 53.0.3 with a clean profile.

Comment 31

[marius.spix]

Thank you. After resolving 1381336, clicking on the A works for me now, too.

Failed 1 tests.
Test 26 passed, but took 61ms (less than 30fps)
Test 35 failed: expected '0' but got '1' - root element, with no parent node, claims to be a :first-child
Total elapsed time: 0.35s

Comment 32

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Sent another email: http://lists.w3.org/Archives/Public/www-archive/2017Jul/0002.html

Comment 33

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Apparently the test is no longer being updated: <http://lists.w3.org/Archives/Public/www-archive/2017Jul/0003.html>. Not sure if there's much point in keeping this bug open.

Comment 34

[Mike Taylor [:miketaylr]]

(In reply to :Ms2ger (⌚ UTC+1/+2) from comment #33)
> Apparently the test is no longer being updated:
> <http://lists.w3.org/Archives/Public/www-archive/2017Jul/0003.html>. Not
> sure if there's much point in keeping this bug open.

Given that update, let's close. Thanks.

"Acid3, in particular, contains some controversial tests and no longer reflects the consensus of the Web standards it purports to test, especially when it comes to issues affecting mobile browsers. The tests remain available for historical purposes and for use by browser vendors."