1637463 - Split writeruntimes data by suite

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Currently, all results from the writeruntimes script gets dumped into one big dictionary. One file is generated per platform, of which there are three:

['unix', 'win', 'android']

There are 24 distinct 'suites' reported as defined by testing/mozbase/moztest/moztest/resolve.py.

With the addition of web-platform-tests and its associated subsuites into the mix, this introduces two problems:

1. dictionary is getting large, which is not a reason to refactor in itself, but more importantly;
2. web-platform-tests and its subsuites share the same directory structure, often with reftests and vanilla web-platform-tests mixed into the same directory.

Point 2 means that if runtime data is obtained for both web-platform-tests and web-platform-tests-reftest, the normalization that occurs in the writeruntimes script means results for both of these get mixed together, skewing the final runtime value that is used in chunking.py.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Scope of work:

• do not modify the definition of suites from resolver.py.
• splitting out to multiple files is acceptable.

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1637463 - writeruntimes script to output runtime information into nested dictionary

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1637463 - update manifest runtime information with new schema for all platforms

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1637463 - support loading of runtime information for a specific suite

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a5fc1b604238
writeruntimes script to output runtime information into nested dictionary r=ahal
https://hg.mozilla.org/integration/autoland/rev/5c8ccda5f1b1
update manifest runtime information with new schema for all platforms r=ahal
https://hg.mozilla.org/integration/autoland/rev/93efb53380c2
support loading of runtime information for a specific suite r=ahal

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c83c52541dd2
Backed out 3 changesets for causing bc jobs to timeout.

Comment 8

[Alexandru Michis [:malexandru]]

Backed out 3 changesets for causing bc jobs to timeout.

Backout link: https://hg.mozilla.org/integration/autoland/rev/c83c52541dd21556eb3afabf64a0960891e17f0d

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Cx64%2Casan%2Cmochitests%2Ctest-windows10-64-asan%2Fopt-mochitest-browser-chrome-e10s-1%2Cm%28bc1%29&tochange=c83c52541dd21556eb3afabf64a0960891e17f0d&fromchange=adde6fa9b4d67f17599f6edd4b7de39a5e6abaab&selectedTaskRun=NRCsY4AhR9y5chl0YT70sQ-0

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=303426395&repo=autoland&lineNumber=5781

In this case the bc1 and bc16 jobs from Windows 10 x64 asan and OS X 10.14 debug respectively are timing out.

Comment 9

[Edwin Takahashi (:egao | infrequent contributor)]

Taking a look. The test paths scheduled for windows10-64-asan/opt bc1 chunk seems to differ after the halfway point pre- and post-patch, which I would expect since I updated the runtime information file.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/588155fb515a
writeruntimes script to output runtime information into nested dictionary r=ahal
https://hg.mozilla.org/integration/autoland/rev/ffa30311305c
update manifest runtime information with new schema for all platforms r=ahal
https://hg.mozilla.org/integration/autoland/rev/7ce1fc0e8368
support loading of runtime information for a specific suite r=ahal

Comment 11

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/588155fb515a
https://hg.mozilla.org/mozilla-central/rev/ffa30311305c
https://hg.mozilla.org/mozilla-central/rev/7ce1fc0e8368

Comment 12

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1637463 - update manifest runtime information with web-platform-tests-reftest information

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e31d66f10e6d
update manifest runtime information with web-platform-tests-reftest information r=jmaher

Comment 14

[Edwin Takahashi (:egao | infrequent contributor)]

Reopening since my previous push to update the manifest-runtime data for each platform didn't include the web-platform-tests-reftest for some reason.

Comment 15

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/e31d66f10e6d