1614640 - Some reftest-family suites have missing manifest information in errorsummary.log

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Marco Castelluccio [:marco]]

E.g. https://firefox-ci-tc.services.mozilla.com/tasks/CCTTJOumQn-d7yENLAt7Qw:

{"action": "test_groups", "line": 114, "groups": ["default"]}

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I don't see this failing on treeherder:
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=286570846&tier=1%2C2%2C3&revision=c0fa6d007c58437398cc06a97d221c42d41dcf9e&searchStr=crashtest%2Cwindows%2Cx64

maybe the errorsummary is generated as a stub and is output even if there is no failure.

:marco, is there a specific error you are looking for in the log, or just an errorsummary file that is not blank?

Comment 2

[Marco Castelluccio [:marco]]

The errorsummary.log usually contains the manifest names that were run too, not only the failures. ActiveData (and mozci) use it to get the manifests which were run in a task.

For example, from the same push, https://firefoxci.taskcluster-artifacts.net/a-5YI_GlRR2Csz27vZT_4w/0/public/test_info//mochitest-browser-chrome_errorsummary.log:

{"action": "test_groups", "line": 4, "groups": ["browser/modules/test/browser/formValidation/browser.ini", "dom/security/test/cors/browser.ini", "testing/modules/tests/browser/browser.ini", "browser/base/content/test/performance/browser.ini", "toolkit/components/startup/tests/browser/browser.ini", "browser/components/resistfingerprinting/test/browser/browser.ini", "browser/base/content/test/plugins/browser.ini", "browser/components/urlbar/tests/ext/browser/browser.ini", "browser/components/search/test/browser/google_codes/browser.ini", "toolkit/components/aboutperformance/tests/browser/browser.ini", "browser/components/urlbar/tests/ext/browser/update/browser.ini", "toolkit/components/windowwatcher/test/browser.ini", "toolkit/mozapps/extensions/test/browser/browser.ini", "dom/crypto/test/browser/browser.ini", "accessible/tests/browser/states/browser.ini", "dom/plugins/test/mochitest/browser.ini", "widget/tests/browser/browser.ini", "browser/base/content/test/referrer/browser.ini", "browser/extensions/screenshots/test/browser/browser.ini", "browser/base/content/test/static/browser.ini", "toolkit/mozapps/extensions/test/xpinstall/browser.ini", "toolkit/content/tests/browser/browser.ini", "browser/base/content/test/metaTags/browser.ini", "dom/fetch/tests/browser.ini", "toolkit/components/reader/test/browser.ini", "browser/components/protocolhandler/test/browser/browser.ini", "browser/base/content/test/performance/hidpi/browser.ini", "dom/security/test/csp/browser.ini", "accessible/tests/browser/browser.ini", "toolkit/components/pictureinpicture/tests/browser.ini", "browser/components/safebrowsing/content/test/browser.ini", "browser/base/content/test/popupNotifications/browser.ini", "dom/ipc/tests/JSWindowActor/browser.ini"]}

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:gbrown, can you help get this bug resolved when you have some cycles?

Comment 4

[Geoff Brown [:gbrown]]

reftest-plain errorsummary files are good; crashtests and jsreftests are not.

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #4)

reftest-plain errorsummary files are good; crashtests and jsreftests are not.

desktop reftest-plain errorsummary files are good; android reftest-plain errorsummary files are not; crashtests and jsreftests are not.

Comment 6

[Geoff Brown [:gbrown]]

Some reftest-family suites have self.run_by_manifest:

https://searchfox.org/mozilla-central/rev/c1e3d3edd4a9b784971555dc74a5de23d768b2e1/layout/tools/reftest/runreftest.py#266
https://searchfox.org/mozilla-central/rev/c1e3d3edd4a9b784971555dc74a5de23d768b2e1/layout/tools/reftest/remotereftest.py#145

Those that are not run_by_manifest log suite_start from reftest.jsm, rather than from runreftest.py (js vs python structured log implementations):

https://searchfox.org/mozilla-central/rev/c1e3d3edd4a9b784971555dc74a5de23d768b2e1/layout/tools/reftest/runreftest.py#948

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1614640 - Improve reftest suite_start data to provide consistent group/manifest names even when not run-by-manifest; r=

Reftest-family test suites can be "run-by-manifest" (reftest-plain, on desktop), or not
(all reftest-family suites on Android, crashtest and jsreftest on desktop). When not
run-by-manifest, code in reftest.jsm is used to format the suite_start data; when
run-by-manifest, code in runreftest.py is used. Currently, the reftest.jsm implementation
submits a simple array of test IDs -- no associated manifests -- possibly consistent with
the concept that tests are not being run/grouped by manifest.

This patch updates the reftest.jsm implementation to generate the same format of
suite_start data as the python implementation: An array of dictionaries mapping
manifest IDs to arrays of associated test IDs.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0480c061117c
Improve reftest suite_start data to provide consistent group/manifest names even when not run-by-manifest; r=ahal

Comment 9

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/0480c061117c