1485668 - Unexpected reftest comparison when upstreaming the tests from bug 488725.

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Emilio Cobos Álvarez (:emilio)]

With a set of reftests like these:

  https://hg.mozilla.org/try/rev/21cc60a631998f0f72bc4a218a1b78cabfcae23b

./mach wpt css/CSS2/floats/float-nowrap-3.html fails, because it tries to compare:

  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html == http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3-ref.html
  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html != http://web-platform.test:8000/css/CSS2/floats/float-nowrap-4.html
  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html == http://web-platform.test:8000/css/CSS2/floats/float-nowrap-4-ref.html

I'm ~sure that the comparison should be:

  float-nowrap-3.html == float-nowrap-3-ref.html
  float-nowrap-3-ref.html != float-nowrap-4.html
  float-nowrap-4.html == float-nowrap-4-ref.html

So looks like we're not updating the left hand side of the expression.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1485668 - Fix reftest chain comparisons in marionette reftest implementation, r=ato

We were previously using the original test url as the lhs for each
comparison after the top level, rather than the previous rhs url as
expected.

Comment 2

[James Graham [:jgraham]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8f998725150451fbaa9e0577e0c5d280610bbbc3

Might need some metadata updates if this fixes/breaks some tests.

Comment 3

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 9003471 [details]
Bug 1485668 - Fix reftest chain comparisons in marionette reftest implementation, r=ato

Andreas Tolfsen ﹝:ato﹞ has approved the revision.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0cd882a506cf
Fix reftest chain comparisons in marionette reftest implementation, r=ato

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/12734 for changes under testing/web-platform/tests

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/12734
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/421980531?utm_source=github_status&utm_medium=notification)

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d325fc4bbfc2
followup: Update manifest. r=me

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0cd882a506cf
https://hg.mozilla.org/mozilla-central/rev/d325fc4bbfc2

Comment 9

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged