1471573 - Reset gcov counters before every test and dump them before shutdown in per-test coverage mode (for Mochitest)

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Comment 1

[Marco Castelluccio [:marco]]

Attachment: Part 1: Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen

Comment 2

[Marco Castelluccio [:marco]]

Attachment: Part 2: Support resetting/dumping coverage counters for mochitest

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988443 [details] [diff] [review]
Part 1: Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen

Review of attachment 8988443 [details] [diff] [review]:
-----------------------------------------------------------------

looks good

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988466 [details] [diff] [review]
Part 2: Support resetting/dumping coverage counters for mochitest

Review of attachment 8988466 [details] [diff] [review]:
-----------------------------------------------------------------

really nice

Comment 5

[Marco Castelluccio [:marco]]

Attachment: Part 1: Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen

I've had to change a few things to fix a failure on try and to rebase on top of recently landed changes to reset/dump functionality.

The patch is almost the same as before, but I'm re-requesting review just in case you want to take a look again.

Comment 6

[Marco Castelluccio [:marco]]

Attachment: Part 2: Support resetting/dumping coverage counters for mochitest

Same as Part 1.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8989597 [details] [diff] [review]
Part 1: Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen

Review of attachment 8989597 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8989598 [details] [diff] [review]
Part 2: Support resetting/dumping coverage counters for mochitest

Review of attachment 8989598 [details] [diff] [review]:
-----------------------------------------------------------------

++

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/eb34fd6b8ce3
Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen. r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/0903f7dc6c7f
Support resetting/dumping coverage counters for mochitest. r=jmaher

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 2 changesets (bug 1471573) for mochitests failures eg. dom/tests/mochitest/beacon/test_beaconContentPolicy.html

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0903f7dc6c7f3d7f10fec403d2ebee470e887876

Backout link: https://hg.mozilla.org/integration/mozilla-inbound/rev/7a80adc796d7e53033c668630655def2874c4ce3

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=186421859&repo=mozilla-inbound&lineNumber=19229

Comment 11

[Marco Castelluccio [:marco]]

Most test failures were due to the usage of SpinEventLoopUntil. I've removed it (which also makes the patch smaller) and they are now passing.
The last remaining failure is bug 1473443, which I think is actually unrelated from this bug The patch makes it more frequent though, as it can slow down the shutdown of the tests.

Anyway, since the requestDumpCoverageCounters and requestResetCoverageCounters can slow down the shutdown of the tests, I'll make them completely no-ops when we are not in per-test coverage mode.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e7272b60f48f
Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen. r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/cdfcca543396
Support resetting/dumping coverage counters for mochitest. r=jmaher

Comment 13

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cff5a58124f6d4c33390b7348dc60acbcf1ef718&tochange=f3ee6caf13e906dc2f5558f7221ce609d6fa53b0&filter-searchStr=android%204.3%20api16%2B%20opt%20mochitests%20test-android-em-4.3-arm7-api-16%2Fopt-robocop&selectedJob=186749654

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=186749654&repo=mozilla-inbound&lineNumber=1824

Backout link: https://hg.mozilla.org/integration/mozilla-inbound/rev/f3ee6caf13e906dc2f5558f7221ce609d6fa53b0

[task 2018-07-06T00:35:40.314Z] 00:35:40     INFO -  - play media -
[task 2018-07-06T00:35:40.314Z] 00:35:40     INFO -  TEST-PASS | testAudioFocus | JSBridege existence check - Should connect JSBridge before using JS!
[task 2018-07-06T00:35:40.314Z] 00:35:40     INFO -  Buffered messages finished
[task 2018-07-06T00:35:40.314Z] 00:35:40  WARNING -  TEST-UNEXPECTED-FAIL | testAudioFocus | GeckoEventExpecter - blockForEvent timeout: Robocop:Java
[task 2018-07-06T00:35:40.315Z] 00:35:40     INFO -  0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAudioFocus | GeckoEventExpecter - blockForEvent timeout: Robocop:Java
[task 2018-07-06T00:35:40.315Z] 00:35:40     INFO -  	at junit.framework.Assert.fail(Assert.java:50)
[task 2018-07-06T00:35:40.316Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:124)
[task 2018-07-06T00:35:40.316Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:145)
[task 2018-07-06T00:35:40.316Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:113)
[task 2018-07-06T00:35:40.317Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:97)
[task 2018-07-06T00:35:40.317Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForBundle(FennecNativeActions.java:171)
[task 2018-07-06T00:35:40.317Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.helpers.JavascriptBridge.processPendingMessage(JavascriptBridge.java:186)
[task 2018-07-06T00:35:40.318Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.helpers.JavascriptBridge.ensureJavaBridgeLoaded(JavascriptBridge.java:222)
[task 2018-07-06T00:35:40.318Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.helpers.JavascriptBridge.sendMessage(JavascriptBridge.java:227)
[task 2018-07-06T00:35:40.319Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.helpers.JavascriptBridge.syncCall(JavascriptBridge.java:138)
[task 2018-07-06T00:35:40.319Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.testAudioFocus.testAdjustMediaVolumeOrMuted(testAudioFocus.java:166)
[task 2018-07-06T00:35:40.319Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.testAudioFocus.testAudioFocus(testAudioFocus.java:29)
[task 2018-07-06T00:35:40.320Z] 00:35:40     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
[task 2018-07-06T00:35:40.320Z] 00:35:40     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
[task 2018-07-06T00:35:40.320Z] 00:35:40     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
[task 2018-07-06T00:35:40.321Z] 00:35:40     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
[task 2018-07-06T00:35:40.321Z] 00:35:40     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
[task 2018-07-06T00:35:40.321Z] 00:35:40     INFO -  	at org.mozilla.gecko.tests.BaseRobocopTest.runTest(BaseRobocopTest.java:187)
[task 2018-07-06T00:35:40.322Z] 00:35:40     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:134)
[task 2018-07-06T00:35:40.322Z] 00:35:40     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:115)
[task 2018-07-06T00:35:40.322Z] 00:35:40     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:133)
[task 2018-07-06T00:35:40.323Z] 00:35:40     INFO -  	at junit.framework.TestResult.run(TestResult.java:118)
[task 2018-07-06T00:35:40.323Z] 00:35:40     INFO -  	at junit.framework.TestCase.run(TestCase.java:124)
[task 2018-07-06T00:35:40.323Z] 00:35:40     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
[task 2018-07-06T00:35:40.324Z] 00:35:40     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
[task 2018-07-06T00:35:40.324Z] 00:35:40     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
[task 2018-07-06T00:35:40.324Z] 00:35:40     INFO -  	at org.mozilla.gecko.FennecInstrumentationTestRunner.onStart(FennecInstrumentationTestRunner.java:64)
[task 2018-07-06T00:35:40.325Z] 00:35:40     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
[task 2018-07-06T00:35:40.325Z] 00:35:40  WARNING -  TEST-UNEXPECTED-FAIL | testAudioFocus | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testAudioFocus | GeckoEventExpecter - blockForEvent timeout: Robocop:Java
[task 2018-07-06T00:35:40.325Z] 00:35:40     INFO -  TEST-OK | testAudioFocus | took 273254ms
[task 2018-07-06T00:35:40.326Z] 00:35:40     INFO -  TEST-START | Shutdown
[task 2018-07-06T00:35:40.326Z] 00:35:40     INFO -  Passed: 33
[task 2018-07-06T00:35:40.327Z] 00:35:40     INFO -  Failed: 2
[task 2018-07-06T00:35:40.327Z] 00:35:40     INFO -  Todo: 0
[task 2018-07-06T00:35:40.327Z] 00:35:40     INFO -  SimpleTest FINISHED

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/88540abd9e9f
Make SpecialPowers RequestDumpCoverageCounters and RequestResetCoverageCounters async messages that wait on the dumping/resetting to actually happen. r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/d2c052a21730
Support resetting/dumping coverage counters for mochitest. r=jmaher

Comment 15

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/88540abd9e9f
https://hg.mozilla.org/mozilla-central/rev/d2c052a21730