Closed Bug 1536804 Opened 7 years ago Closed 7 years ago

Add a TC task that provides a json summary of wpt metadata

Categories

(Testing :: web-platform-tests, enhancement)

Product:
Testing
Component:
web-platform-tests
Version:
Version 3
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox68 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla68
Tracking Flags:
Tracking Status
firefox68 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

Details

Attachments

(4 files)

Bug 1536804 - Factor out a base class for the wpt metadata compilers,
47 bytes, text/x-phabricator-request
Details | Review
Bug 1536804 - Add a mach command to summarise wpt metadata,
47 bytes, text/x-phabricator-request
Details | Review
Bug 1536804 - Add a task for summarising wpt metadata,
47 bytes, text/x-phabricator-request
Details | Review
Bug 1536804 - Fix metadata for test that mysteriously started (also) permafailing on Mac debug,
47 bytes, text/x-phabricator-request
Details | Review

This can be used by the wpt dashboard to ingest that data without having to parse the wpt manifest files or similar.

The manifest compilers follow a mostly common design so it seems
reasonable to put the common parts into a base class, and allow
subclasses to override the functionaility where required.

This parses the metadata files without evaluating the conditional
expressions, producing a JSON summary file in which each possible
condition is represented as a string.

Not all data from the metadata files appears in the summary, only
exceptional cases likely to be of interest to engineers and unlikely
to be on wpt.fyi e.g. tests that have lsan leaks or tests that crash,
not just tests that fail. This is intended to keep the filesize down.

Depends on D24177

This task runs on wpt metadata changes and uploads an artifact
containing the summarised metadata.

Depends on D24178

Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/9a680e886248 Factor out a base class for the wpt metadata compilers, r=ato https://hg.mozilla.org/integration/autoland/rev/795287b1e059 Add a mach command to summarise wpt metadata, r=ato https://hg.mozilla.org/integration/autoland/rev/e8758002d7d4 Add a task for summarising wpt metadata, r=ahal

Backed out 3 changesets (Bug 1536804) for /css/css-* failures

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=os%2Cx%2C10.10%2Cdebug%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-macosx64%2Fdebug-web-platform-tests-reftests-e10s-3%2Cw-e10s%28wr3%29&fromchange=563e8fb1be5db3fa6f984c34270a35c7cf1dd4e6&tochange=f755dfcfc421e45d63aef09ca95805c1cbd34a4d

https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=73a48619739e34609d11f5ed15f3fc1436ee154f&searchStr=linux%2Copt%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-linux32%2Fopt-web-platform-tests-reftests-e10s-3%2Cw-e10s%28wr3%29&tochange=f755dfcfc421e45d63aef09ca95805c1cbd34a4d

Backout link: https://hg.mozilla.org/integration/autoland/rev/f755dfcfc421e45d63aef09ca95805c1cbd34a4d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=236145971&repo=autoland&lineNumber=6122

[task 2019-03-26T16:36:39.743Z] 16:36:39 INFO - TEST-START | /css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht
[task 2019-03-26T16:36:39.751Z] 16:36:39 INFO - PID 7884 | 1553618199747 Marionette INFO Testing http://web-platform.test:8000/css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht == http://web-platform.test:8000/css/css-text/white-space/reference/text-space-collapse-preserve-breaks-001-ref.xht
[task 2019-03-26T16:36:39.758Z] 16:36:39 INFO - PID 7884 | JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 2383: TypeError: tabbrowser.getTabForBrowser is not a function
[task 2019-03-26T16:36:39.815Z] 16:36:39 INFO - PID 7884 | JavaScript error: resource:///modules/sessionstore/SessionStore.jsm, line 2383: TypeError: tabbrowser.getTabForBrowser is not a function
[task 2019-03-26T16:36:39.856Z] 16:36:39 INFO - PID 7884 | 1553618199853 Marionette INFO No differences allowed
[task 2019-03-26T16:36:39.856Z] 16:36:39 INFO - PID 7884 | 1553618199853 Marionette INFO Found 1936 pixels different, maximum difference per channel 255
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - TEST-UNEXPECTED-FAIL | /css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht | Testing http://web-platform.test:8000/css/css-text/white-space/text-space-collapse-preserve-breaks-001.xht == http://web-platform.test:8000/css/css-text/white-space/reference/text-space-collapse-preserve-breaks-001-ref.xht
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - REFTEST IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAJYCAYAAACadoJwAAAe3UlEQVR4nO3d2XLlOrIlwPz/nz79p
[task 2019-03-26T16:36:39.913Z] 16:36:39 INFO - REFTEST IMAGE 2 (REFERENCE): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAJYCAYAAACadoJwAAAejE
[task 2019-03-26T16:36:39.917Z] 16:36:39 INFO - TEST-INFO took 147ms
[task 2019-03-26T16:36:39.934Z] 16:36:39 INFO - PID 7884 | 1553618199925 Marionette INFO Stopped listening on port 2828

Flags: needinfo?(james)
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/16129 for changes under testing/web-platform/tests
Upstream PR was closed without merging
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/7f7a2ccd9051 Factor out a base class for the wpt metadata compilers, r=ato https://hg.mozilla.org/integration/autoland/rev/476e75d11578 Add a mach command to summarise wpt metadata, r=ato https://hg.mozilla.org/integration/autoland/rev/1ce8b8034950 Add a task for summarising wpt metadata, r=ahal
Flags: needinfo?(james)
Assignee: nobody → james
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/mozilla-inbound/rev/8b3c0775078a [wpt PR 16129] - [Gecko Bug 1536804] Factor out a base class for the wpt metadata compilers, a=testonly
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: