1469893 - Manifest updater should collapse conditions where possible

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

See for example [1]. This ini file has two conditions under which the test fails, but the two conditions are almost identical. The first has "not debug" and the second has "debug". In cases like this, where two conditions A and B are identical except for one term, and A has "term" while B has "!term", that term should just be dropped from the condition.

In this case, the collapsed condition would just be:

if webrender and e10s and (os == "linux") and (version == "Ubuntu 16.04") and (processor == "x86_64") and (bits == 64): FAIL

This makes it easier for humans to see what the conditions are without having to sort through the spurious things.

[1] https://searchfox.org/mozilla-central/rev/d0a41d2e7770fc00df7844d5f840067cc35ba26f/testing/web-platform/meta/css/css-backgrounds/background-rounded-image-clip.html.ini

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Attachment: collapser.rs

Here's a tool I wrote to postprocess the .ini files and do the collapsing.

cd $SRDIR/testing/web-platform/meta
find . -name "*.ini" -type f | while read file; do echo "Processing $file"; ~/collapser $file > collapsed; mv collapsed $file; done

Comment 2

[James Graham [:jgraham]]

Attachment: Bug 1469893 - Make the metadata update generate more compact conditionals,

Previously the wpt metadata update code generated rather conservative
conditionals, simply inclusing every possible condition unless all the
tests had the same outcome or similar. This isn't very nice to read
and makes the metadata rather fragile to changes in the configuration.

A better approach is to build a decision tree from the metadata,
choosing the properties to split on using a simple greedy algorithm,
as follows:

• Given a set of test results, associated run_info_properties and
  properties that we want to use, partition the test results by each
  property we are using in turn.

• For each partition, generate a score for how uniform the results
  are in each subset after partition. The score should be good if the
  partition results in small numbers of groups with mostly uniform
  test results and bad if it results in a large number of groups or
  groups with a mix of test results. For this we adopt a metric based
  on the Shannon entropy.

• Chose the partition resulting in the best score, and use that as a
  condition in the tree.

• Recursively reapply the algorithm to each group that's been
  created.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fc3a0502508
Make the metadata update generate more compact conditionals, r=maja_zf

Comment 4

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/1fc3a0502508

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18204 for changes under testing/web-platform/tests

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/18204
* Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/B5Hf6f70QbeZQ_kFn8zxiw)

Comment 7

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3337a23f2d41
[wpt PR 18204] - [Gecko Bug 1469893] Make the metadata update generate more compact conditionals, a=testonly

Comment 9

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/3337a23f2d41