1501619 - [wpt-sync] Sync PR 13698 - [run_web_tests] Check for extra baselines

Status: RESOLVED FIXED

(Core :: Web Audio, enhancement, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 13698 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/13698
Details from upstream follow.

Xianzhu Wang <[e-mail]> wrote:
>  [run_web_tests] Check for extra baselines
>  
>  After https://chromium-review.googlesource.com/c/chromium/src/+/1277654,
>  we no longer generate text result of many tests, thus these tests no
>  longer need the text baselines.
>  
>  This CL checks for extra baselines in the following cases:
>  1. Extra -expected.png (existing logic) for ref tests;
>  2. Extra -expected.txt, -expected.png, -expected.wav if the test doesn't
>     generate the result;
>  3. Extra -expected.txt for all-pass testharness tests.
>  
>  For 2, the extra baseline will be treated as a mismatch failure (which
>  can be interpreted as that the test doesn't generate the result, but
>  the baseline expects the result).
>  
>  For 1 and 3, there are just warning messages because for now we don't
>  have a good way to present them in the layout test result page.
>  
>  We have already had a similar logic in pre-submit for 3 and the new
>  check in run_web_tests is more accurate. For other types of extra
>  baselines, it's hard to implement the check in presubmit because it
>  needs more knowledge than static pre-submit knows.
>  
>  Bug: 703899
>  Cq-Include-Trybots: luci.chromium.try:linux-blink-gen-property-trees;luci.chromium.try:linux_layout_tests_layout_ng;luci.chromium.try:linux_layout_tests_slimming_paint_v2;master.tryserver.blink:linux_trusty_blink_rel
>  Change-Id: I9016d5583791f179f1d51561245602af615d21de
>  Reviewed-on: https://chromium-review.googlesource.com/1286894
>  WPT-Export-Revision: 23f955341070ed3f7f92c588242674cc8ad2d713

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

PR 13698 applied with additional changes from upstream: 4e2ecda993fcfd636c09f61421c8b0a4244e8113

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=5df894c591624a14883ce3f86c4a21966e52cfe0

Comment 3

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=55f7e1e5204e9c249baec27cecadc9674bfc6755

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=befb1372f384891cbdd5c9f672611a29b1e59b39

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try (stability) https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7f9e09b24dc1038debe6038db3958d39038654f

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Ran 1 tests and 14 subtests
OK     : 1
PASS   : 14

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8e42fe25838c
[wpt PR 13698] - [run_web_tests] Check for extra baselines, a=testonly

Comment 8

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/8e42fe25838c