Add a TC task that provides a json summary of wpt metadata

RESOLVED FIXED in Firefox 68

Status

enhancement
RESOLVED FIXED
3 months ago
6 days ago

People

(Reporter: jgraham, Assigned: jgraham)

Tracking

Version 3
mozilla68
Points:
---

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(4 attachments)

This can be used by the wpt dashboard to ingest that data without having to parse the wpt manifest files or similar.

The manifest compilers follow a mostly common design so it seems
reasonable to put the common parts into a base class, and allow
subclasses to override the functionaility where required.

This parses the metadata files without evaluating the conditional
expressions, producing a JSON summary file in which each possible
condition is represented as a string.

Not all data from the metadata files appears in the summary, only
exceptional cases likely to be of interest to engineers and unlikely
to be on wpt.fyi e.g. tests that have lsan leaks or tests that crash,
not just tests that fail. This is intended to keep the filesize down.

Depends on D24177

This task runs on wpt metadata changes and uploads an artifact
containing the summarised metadata.

Depends on D24178

Comment 4

3 months ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/9a680e886248
Factor out a base class for the wpt metadata compilers, r=ato
https://hg.mozilla.org/integration/autoland/rev/795287b1e059
Add a mach command to summarise wpt metadata, r=ato
https://hg.mozilla.org/integration/autoland/rev/e8758002d7d4
Add a task for summarising wpt metadata, r=ahal

Backed out 3 changesets (Bug 1536804) for /css/css-* failures

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=os%2Cx%2C10.10%2Cdebug%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-macosx64%2Fdebug-web-platform-tests-reftests-e10s-3%2Cw-e10s%28wr3%29&fromchange=563e8fb1be5db3fa6f984c34270a35c7cf1dd4e6&tochange=f755dfcfc421e45d63aef09ca95805c1cbd34a4d

https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=73a48619739e34609d11f5ed15f3fc1436ee154f&searchStr=linux%2Copt%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-linux32%2Fopt-web-platform-tests-reftests-e10s-3%2Cw-e10s%28wr3%29&tochange=f755dfcfc421e45d63aef09ca95805c1cbd34a4d

Backout link: https://hg.mozilla.org/integration/autoland/rev/f755dfcfc421e45d63aef09ca95805c1cbd34a4d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=236145971&repo=autoland&lineNumber=6122

[task 2019-03-26T16:36:39.743Z] 16:36:39 INFO - TEST-START | /css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht
[task 2019-03-26T16:36:39.751Z] 16:36:39 INFO - PID 7884 | 1553618199747 Marionette INFO Testing http://web-platform.test:8000/css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht == http://web-platform.test:8000/css/css-text/white-space/reference/text-space-collapse-preserve-breaks-001-ref.xht
[task 2019-03-26T16:36:39.758Z] 16:36:39 INFO - PID 7884 | JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 2383: TypeError: tabbrowser.getTabForBrowser is not a function
[task 2019-03-26T16:36:39.815Z] 16:36:39 INFO - PID 7884 | JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 2383: TypeError: tabbrowser.getTabForBrowser is not a function
[task 2019-03-26T16:36:39.856Z] 16:36:39 INFO - PID 7884 | 1553618199853 Marionette INFO No differences allowed
[task 2019-03-26T16:36:39.856Z] 16:36:39 INFO - PID 7884 | 1553618199853 Marionette INFO Found 1936 pixels different, maximum difference per channel 255
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - TEST-UNEXPECTED-FAIL | /css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht | Testing http://web-platform.test:8000/css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht == http://web-platform.test:8000/css/css-text/white-space/reference/text-space-collapse-preserve-breaks-001-ref.xht
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - REFTEST IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAJYCAYAAACadoJwAAAe3UlEQVR4nO3d2XLlOrIlwPz/nz79p
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - REFTEST IMAGE 2 (REFERENCE): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAJYCAYAAACadoJwAAAejE
[task 2019-03-26T16:36:39.917Z] 16:36:39 INFO - TEST-INFO took 147ms
[task 2019-03-26T16:36:39.934Z] 16:36:39 INFO - PID 7884 | 1553618199925 Marionette INFO Stopped listening on port 2828

Flags: needinfo?(james)
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/16129 for changes under testing/web-platform/tests
Upstream PR was closed without merging

Comment 9

3 months ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/7f7a2ccd9051
Factor out a base class for the wpt metadata compilers, r=ato
https://hg.mozilla.org/integration/autoland/rev/476e75d11578
Add a mach command to summarise wpt metadata, r=ato
https://hg.mozilla.org/integration/autoland/rev/1ce8b8034950
Add a task for summarising wpt metadata, r=ahal
Flags: needinfo?(james)

Comment 10

3 months ago
bugherder
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Upstream PR merged

Comment 12

7 days ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b3c0775078a
[wpt PR 16129] - [Gecko Bug 1536804] Factor out a base class for the wpt metadata compilers, a=testonly
You need to log in before you can comment on or make changes to this bug.