Closed Bug 1485668 Opened Last year Closed Last year

Unexpected reftest comparison when upstreaming the tests from bug 488725.

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: emilio, Assigned: jgraham)

References

Details

Attachments

(1 file)

With a set of reftests like these:

  https://hg.mozilla.org/try/rev/21cc60a631998f0f72bc4a218a1b78cabfcae23b

./mach wpt css/CSS2/floats/float-nowrap-3.html fails, because it tries to compare:

  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html == http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3-ref.html
  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html != http://web-platform.test:8000/css/CSS2/floats/float-nowrap-4.html
  Testing http://web-platform.test:8000/css/CSS2/floats/float-nowrap-3.html == http://web-platform.test:8000/css/CSS2/floats/float-nowrap-4-ref.html

I'm ~sure that the comparison should be:

  float-nowrap-3.html == float-nowrap-3-ref.html
  float-nowrap-3-ref.html != float-nowrap-4.html
  float-nowrap-4.html == float-nowrap-4-ref.html

So looks like we're not updating the left hand side of the expression.
Flags: needinfo?(james)
Blocks: 1485669
We were previously using the original test url as the lhs for each
comparison after the top level, rather than the previous rhs url as
expected.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8f998725150451fbaa9e0577e0c5d280610bbbc3

Might need some metadata updates if this fixes/breaks some tests.
Flags: needinfo?(james)
Comment on attachment 9003471 [details]
Bug 1485668 - Fix reftest chain comparisons in marionette reftest implementation, r=ato

Andreas Tolfsen ﹝:ato﹞ has approved the revision.
Attachment #9003471 - Flags: review+
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/0cd882a506cf
Fix reftest chain comparisons in marionette reftest implementation, r=ato
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/12734 for changes under testing/web-platform/tests
https://hg.mozilla.org/mozilla-central/rev/0cd882a506cf
https://hg.mozilla.org/mozilla-central/rev/d325fc4bbfc2
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Upstream PR merged
Assignee: nobody → james
You need to log in before you can comment on or make changes to this bug.