1476574 - Reset gcov counters before every test and dump them before shutdown in per-test coverage mode (for web-platform-tests)

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Comment 1

[Marco Castelluccio [:marco]]

Attachment: Support resetting/dumping code coverage counters before/after web-platform-tests

Comment 2

[James Graham [:jgraham]]

Comment on attachment 8993281 [details] [diff] [review]
Support resetting/dumping code coverage counters before/after web-platform-tests

Review of attachment 8993281 [details] [diff] [review]:
-----------------------------------------------------------------

This mostly loosk great, just a few minor pieces of cleanup required.

::: testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py
@@ +391,5 @@
> +            """
> +        with self.marionette.using_context(self.marionette.CONTEXT_CHROME):
> +            error = self.marionette.execute_async_script(script)
> +            if error is not None:
> +                raise Exception('Failure while resetting counters: {}'.format(error))

FWIW we don't use .format elsewhere in this file but prefer %

@@ +407,5 @@
> +            """
> +        with self.marionette.using_context(self.marionette.CONTEXT_CHROME):
> +            error = self.marionette.execute_async_script(script)
> +            if error is not None:
> +                raise Exception('Failure while dumping counters: {}'.format(error))

So I think if you have a test that crashes this will raise an exception because you can't communicate with marionette, but we want to record the status as "CRASH" rather than "ERROR". So I would catch exceptions here similarly to the way that it's done in MarionetteAssertsProtocolPart to avoid that (similarly in reset, although that's less likely).

@@ +742,4 @@
>          result = self.implementation.run_test(test)
>  
> +        if self.protocol.coverage.is_enabled:
> +            self.protocol.coverage.dump()

This will cover testharness and reftest tests; if you also care about wdspec tests you'll need to override do_test in the MarionetteWdspecExecutor, or add in some before_test and after_test methods that do nothing by default, or something.

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Enable reset/dump for wpt and cleanup harness code for supporting reset/dump now that it is supported by all test suites

Comment 4

[Marco Castelluccio [:marco]]

Attachment: Support resetting/dumping code coverage counters before/after web-platform-tests

(In reply to James Graham [:jgraham] from comment #2)
> @@ +742,4 @@
> >          result = self.implementation.run_test(test)
> >  
> > +        if self.protocol.coverage.is_enabled:
> > +            self.protocol.coverage.dump()
> 
> This will cover testharness and reftest tests; if you also care about wdspec
> tests you'll need to override do_test in the MarionetteWdspecExecutor, or
> add in some before_test and after_test methods that do nothing by default,
> or something.

wdspec will require more changes, as it's not using MarionetteProtocol but GeckoDriverProtocol (so I'd need to implement something like MarionetteCoverageProtocolPart for GeckoDriver I guess). I'm thinking I'll take care of it in a follow-up.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8993366 [details] [diff] [review]
Enable reset/dump for wpt and cleanup harness code for supporting reset/dump now that it is supported by all test suites

Review of attachment 8993366 [details] [diff] [review]:
-----------------------------------------------------------------

nice cleanup

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ec37892ce390
Support resetting/dumping code coverage counters before/after web-platform-tests. r=jgraham
https://hg.mozilla.org/integration/mozilla-inbound/rev/4ea7997194e9
Enable reset/dump for wpt and cleanup harness code for supporting reset/dump now that it is supported by all test suites. r=jmaher

Comment 7

[Cristina Coroiu [:ccoroiu]]

Backed out 2 changesets (bug 1476574) for browser-chrome failures at browser/base/content/test/static/browser_all_files_referenced.js

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/b79c5a3c4e8ac73632177c018f52bb08d669df00

Failure push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4ea7997194e9f995bb0b1e434524a0ea5596d758

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=189014438&repo=mozilla-inbound&lineNumber=1836

[task 2018-07-19T20:13:23.439Z] 20:13:23     INFO - TEST-START | browser/base/content/test/static/browser_all_files_referenced.js
[task 2018-07-19T20:13:35.131Z] 20:13:35     INFO - TEST-INFO | started process screentopng
[task 2018-07-19T20:13:35.603Z] 20:13:35     INFO - TEST-INFO | screentopng: exit 0
[task 2018-07-19T20:13:35.603Z] 20:13:35     INFO - Buffered messages logged at 20:13:23
[task 2018-07-19T20:13:35.603Z] 20:13:35     INFO - Entering test bound checkAllTheFiles
[task 2018-07-19T20:13:35.605Z] 20:13:35     INFO - Console message: [JavaScript Warning: "Use of nsIFile in content process is deprecated." {file: "resource://gre/modules/FileUtils.jsm" line: 170}]
[task 2018-07-19T20:13:35.605Z] 20:13:35     INFO - Buffered messages logged at 20:13:35
[task 2018-07-19T20:13:35.606Z] 20:13:35     INFO - indirectly whitelisted file: chrome://marionette/content/test_dialog.dtd used from chrome://marionette/content/test_dialog.xul
[task 2018-07-19T20:13:35.606Z] 20:13:35     INFO - indirectly whitelisted file: chrome://global-platform/locale/intl.properties used from resource://gre/greprefs.js
[task 2018-07-19T20:13:35.610Z] 20:13:35     INFO - indirectly whitelisted file: chrome://marionette/content/test.xul used from chrome://marionette/content/test_anonymous_content.xul
[task 2018-07-19T20:13:35.612Z] 20:13:35     INFO - indirectly whitelisted file: chrome://marionette/content/test_nested_iframe.xul used from chrome://marionette/content/test.xul
[task 2018-07-19T20:13:35.614Z] 20:13:35     INFO - indirectly whitelisted file: chrome://marionette/content/test2.xul used from chrome://marionette/content/test.xul,chrome://marionette/content/test_nested_iframe.xul
[task 2018-07-19T20:13:35.617Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/addonutils.js used from resource://services-sync/engines/addons.js
[task 2018-07-19T20:13:35.619Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/Promise-backend.js used from resource://gre/modules/Promise.jsm,resource://devtools/shared/worker/loader.js,resource://devtools/shared/Loader.jsm
[task 2018-07-19T20:13:35.619Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/engines/clients.js used from resource://services-sync/service.js
[task 2018-07-19T20:13:35.625Z] 20:13:35     INFO - indirectly whitelisted file: chrome://pippki/content/resetpassword.js used from chrome://pippki/content/resetpassword.xul
[task 2018-07-19T20:13:35.625Z] 20:13:35     INFO - indirectly whitelisted file: chrome://global/content/accessibility/content-script.js used from resource://gre/modules/accessibility/AccessFu.jsm
[task 2018-07-19T20:13:35.626Z] 20:13:35     INFO - indirectly whitelisted file: chrome://global/content/remote-test-ipc.js used from chrome://global/content/test-ipc.xul
[task 2018-07-19T20:13:35.628Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/telemetry.js used from resource://services-sync/bookmark_repair.js,resource://services-sync/service.js
[task 2018-07-19T20:13:35.630Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/stages/enginesync.js used from resource://services-sync/service.js
[task 2018-07-19T20:13:35.632Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/addonsreconciler.js used from resource://services-sync/engines/addons.js
[task 2018-07-19T20:13:35.633Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/collection_validator.js used from resource://services-sync/engines/addons.js,resource://services-sync/engines/forms.js,resource://services-sync/engines/passwords.js
[task 2018-07-19T20:13:35.635Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/policies.js used from resource://services-sync/service.js
[task 2018-07-19T20:13:35.645Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/bookmark_validator.js used from resource://services-sync/engines/bookmarks.js
[task 2018-07-19T20:13:35.646Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/browserid_identity.js used from resource://services-sync/telemetry.js,resource://services-sync/status.js
[task 2018-07-19T20:13:35.647Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-common/tokenserverclient.js used from resource://services-sync/browserid_identity.js
[task 2018-07-19T20:13:35.648Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/status.js used from resource://services-sync/telemetry.js,resource://services-sync/policies.js,resource://services-sync/service.js
[task 2018-07-19T20:13:35.649Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/collection_repair.js used from resource://services-sync/engines/clients.js,resource://services-sync/bookmark_repair.js,resource://services-sync/doctor.js
[task 2018-07-19T20:13:35.650Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/stages/declined.js used from resource://services-sync/service.js
[task 2018-07-19T20:13:35.651Z] 20:13:35     INFO - indirectly whitelisted file: chrome://pippki/content/load_device.js used from chrome://pippki/content/load_device.xul
[task 2018-07-19T20:13:35.652Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/engines.js used from resource://services-sync/engines/clients.js,resource://services-sync/engines/addons.js,resource://services-sync/engines/forms.js,resource://services-sync/engines/history.js,resource://services-sync/engines/tabs.js,resource://services-sync/engines/bookmarks.js,resource://services-sync/engines/extension-storage.js,resource://services-sync/engines/passwords.js,resource://services-sync/engines/prefs.js,resource://services-sync/service.js,resource://formautofill/FormAutofillSync.jsm
[task 2018-07-19T20:13:35.654Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-common/logmanager.js used from resource://services-sync/policies.js
[task 2018-07-19T20:13:35.656Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/EventManager.jsm used from chrome://global/content/accessibility/content-script.js
[task 2018-07-19T20:13:35.657Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Utils.jsm used from chrome://global/content/accessibility/content-script.js,resource://gre/modules/accessibility/Traversal.jsm,resource://gre/modules/accessibility/EventManager.jsm,resource://gre/modules/accessibility/Presentation.jsm,resource://gre/modules/accessibility/AccessFu.jsm,resource://gre/modules/accessibility/OutputGenerator.jsm,resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-07-19T20:13:35.659Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/SyncedBookmarksMirror.jsm used from resource://services-sync/engines/bookmarks.js
[task 2018-07-19T20:13:35.661Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Presentation.jsm used from chrome://global/content/accessibility/content-script.js,resource://gre/modules/accessibility/EventManager.jsm,resource://gre/modules/accessibility/AccessFu.jsm,resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-07-19T20:13:35.662Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/OutputGenerator.jsm used from resource://gre/modules/accessibility/Presentation.jsm
[task 2018-07-19T20:13:35.667Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Constants.jsm used from chrome://global/content/accessibility/content-script.js,resource://gre/modules/accessibility/Traversal.jsm,resource://gre/modules/accessibility/EventManager.jsm,resource://gre/modules/accessibility/Utils.jsm,resource://gre/modules/accessibility/Presentation.jsm,resource://gre/modules/accessibility/OutputGenerator.jsm,resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-07-19T20:13:35.668Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/ContentControl.jsm used from chrome://global/content/accessibility/content-script.js
[task 2018-07-19T20:13:35.669Z] 20:13:35     INFO - indirectly whitelisted file: chrome://passwordmgr/content/recipes.json used from resource://gre/greprefs.js
[task 2018-07-19T20:13:35.669Z] 20:13:35     INFO - indirectly whitelisted file: resource://formautofill/FormAutofillSync.jsm used from resource://services-sync/service.js
[task 2018-07-19T20:13:35.672Z] 20:13:35     INFO - indirectly whitelisted file: chrome://global/locale/AccessFu.properties used from resource://gre/modules/accessibility/Utils.jsm
[task 2018-07-19T20:13:35.673Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/bookmark_repair.js used from resource://services-sync/collection_repair.js
[task 2018-07-19T20:13:35.675Z] 20:13:35     INFO - indirectly whitelisted file: resource://services-sync/doctor.js used from resource://services-sync/bookmark_repair.js,resource://services-sync/stages/enginesync.js
[task 2018-07-19T20:13:35.677Z] 20:13:35     INFO - indirectly whitelisted file: resource://gre/modules/accessibility/Traversal.jsm used from resource://gre/modules/accessibility/ContentControl.jsm
[task 2018-07-19T20:13:35.678Z] 20:13:35     INFO - Buffered messages finished
[task 2018-07-19T20:13:35.680Z] 20:13:35     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | there should be no unreferenced files - Got 1, expected 0
[task 2018-07-19T20:13:35.681Z] 20:13:35     INFO - Stack trace:
[task 2018-07-19T20:13:35.683Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:test_is:1305
[task 2018-07-19T20:13:35.685Z] 20:13:35     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:684
[task 2018-07-19T20:13:35.686Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1103
[task 2018-07-19T20:13:35.687Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1094
[task 2018-07-19T20:13:35.688Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:996
[task 2018-07-19T20:13:35.689Z] 20:13:35     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-07-19T20:13:35.691Z] 20:13:35     INFO - Not taking screenshot here: see the one that was previously logged
[task 2018-07-19T20:13:35.693Z] 20:13:35     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | unreferenced file: chrome://marionette/content/PerTestCoverageUtils.jsm - 
[task 2018-07-19T20:13:35.695Z] 20:13:35     INFO - Stack trace:
[task 2018-07-19T20:13:35.696Z] 20:13:35     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:688
[task 2018-07-19T20:13:35.698Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1103
[task 2018-07-19T20:13:35.700Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1094
[task 2018-07-19T20:13:35.701Z] 20:13:35     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:996
[task 2018-07-19T20:13:35.702Z] 20:13:35     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-07-19T20:13:35.704Z] 20:13:35     INFO - ignored unused whitelist entry: chrome://global/locale/crashes.dtd
[task 2018-07-19T20:13:35.706Z] 20:13:35     INFO - ignored unused whitelist entry: chrome://global/locale/crashes.properties
[task 2018-07-19T20:13:35.708Z] 20:13:35     INFO - ignored unused whitelist entry: resource://app/blocklist.xml

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5e961986f536
Support resetting/dumping code coverage counters before/after web-platform-tests. r=jgraham
https://hg.mozilla.org/integration/mozilla-inbound/rev/7afb8c53da3a
Enable reset/dump for wpt and cleanup harness code for supporting reset/dump now that it is supported by all test suites. r=jmaher

Comment 9

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/12084 for changes under testing/web-platform/tests

Comment 10

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 11

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/12084
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/406089632?utm_source=github_status&utm_medium=notification)

Comment 12

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/5e961986f536
https://hg.mozilla.org/mozilla-central/rev/7afb8c53da3a

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/44b123467fc3
Detect earlier if per-test coverage mode is disabled by detecting if we're in a ccov build using mozinfo. r=jgraham

Comment 14

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/44b123467fc3

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged