1471769 - Perma-failing tier2 ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkip TEST-UNEXPECTED-FAIL

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=185253172&repo=mozilla-central

https://queue.taskcluster.net/v1/task/SlWnaLtORTSzwOxrWWxmGA/runs/0/artifacts/public/logs/live_backing.log

[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - /builds/worker/workspace/build/src/testing/xpcshell/selftest.py
[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - ============================= test session starts ==============================
[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - platform linux2 -- Python 2.7.9, pytest-3.2.5, py-1.5.3, pluggy-0.4.0 -- /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/src-UL-dti-o/bin/python
[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - rootdir: /builds/worker/workspace/build/src/testing/xpcshell, inifile: /builds/worker/workspace/build/src/config/mozunit/mozunit/pytest.ini
[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - collecting ... collected 55 items
[task 2018-06-27T22:57:57.159Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskRunNextTest PASSED
[task 2018-06-27T22:57:57.159Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkip TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.160Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskSkipAll TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.160Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskStackTrace TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.160Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestFailureInside PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestMultiple TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestRejected PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTaskTestSingle TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestFailing PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestSimple TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejection PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAddTestUncaughtRejectionJSM PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAssertStack TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testAsyncCleanup TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChild TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildFail TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildHang TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildMozinfo TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testChildPass TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseExplicit TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseInManifest TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoPrintWhenVerboseNotExplicit TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportForeignObject TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportNonSyntaxError PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportRefError TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoReportSyntaxError PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowForeignObject TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testDoThrowString TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testFail PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testHangingTimeout <- ../../../../../../../usr/lib/python2.7/unittest/case.py SKIPPED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testKnownFail TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testLogCorrectFileName TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMissingHeadFile PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testMozinfo TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTask TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskFail PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTaskMultiple TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTest TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestAddTask TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestAddTestFail PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNoRunTestEmptyTest TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testNotSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPass TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testPassFail TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testRandomExecution TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testReturnNonzero PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkip PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTask TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSkipForAddTest TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testSyntaxError PASSED
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejection TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUncaughtRejectionJSM TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnexpectedPass TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57  WARNING - ../testing/xpcshell/selftest.py::XPCShellTestsTests::testUnicodeInAssertMethods TEST-UNEXPECTED-FAIL
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - =================================== FAILURES ===================================
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ______________________ XPCShellTestsTests.testAddTaskSkip ______________________
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - self = <selftest.XPCShellTestsTests testMethod=testAddTaskSkip>
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO -     def testAddTaskSkip(self):
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO -         self.writeFile("test_tasks_skip.js", ADD_TASK_SKIP)
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO -         self.writeManifest(["test_tasks_skip.js"])
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - >       self.assertTestResult(True)
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py:1067:
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - ../testing/xpcshell/selftest.py:527: in assertTestResult
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO -     """ % ("passed" if expected else "failed", self.log.getvalue()))
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - E   AssertionError: Tests should have passed, log:
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - E   ========
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - E   MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail.
[task 2018-06-27T22:57:57.170Z] 22:57:57     INFO - E   Running tests sequentially.
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   SUITE-START | Running 1 tests
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   TEST-START | test_tasks_skip.js
[task 2018-06-27T22:57:57.171Z] 22:57:57  WARNING - E   TEST-UNEXPECTED-FAIL | test_tasks_skip.js | xpcshell return code: -11
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   TEST-INFO took 371ms
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   >>>>>>>
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | [CodeCoverage] Setting handlers for process 6546.
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | [6546, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2682
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | JavaScript error: /builds/worker/workspace/build/src/testing/xpcshell/head.js, line 527: TypeError: PerTestCoverageUtils is null
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | Couldn't convert chrome URL: chrome://branding/locale/brand.properties
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | [6546, Main Thread] WARNING: Could not get the program name for a cubeb stream.: 'NS_SUCCEEDED(rv)', file /builds/worker/workspace/build/src/dom/media/CubebUtils.cpp, line 351
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | ERROR: GC found live Cell 0x7f95504e2040 of kind FUNCTION at shutdown
[task 2018-06-27T22:57:57.171Z] 22:57:57     INFO - E   PID 6546 | ERROR: GC found live Cell 0x7f95504e2080 of kind FUNCTION at shutdown

Comment 1

[Marco Castelluccio [:marco]]

This is my fault, I'll fix it tomorrow morning.

Comment 2

[Andreea Pavel [:apavel]]

(In reply to Marco Castelluccio [:marco] from comment #1)
> This is my fault, I'll fix it tomorrow morning.

This has a tier1 failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=185270367

Can you please take a look?

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Make sure PerTestCoverageUtils doesn't do anything unless per-test mode is enabled

This makes PerTestCoverageUtils.jsm always used, but be a no-op when per-test coverage mode is not enabled.

Comment 4

[Marco Castelluccio [:marco]]

Attachment: Cleanup both GCOV_PREFIX and GCOV_RESULTS_DIR directories at the end of a test

Turns out this was actually needed (unfortunately), otherwise we'd cleanup only one of the two directories.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988446 [details] [diff] [review]
Make sure PerTestCoverageUtils doesn't do anything unless per-test mode is enabled

Review of attachment 8988446 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/xpcshell/head.js
@@ +515,2 @@
>  
> +  PerTestCoverageUtils.beforeTestSync();

how does this work if it defaults to {} ?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988447 [details] [diff] [review]
Cleanup both GCOV_PREFIX and GCOV_RESULTS_DIR directories at the end of a test

Review of attachment 8988447 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/desktop_unittest.py
@@ +910,3 @@
>                          )
> +                        if 'GCOV_RESULTS_DIR' in env:
> +                            shutil.rmtree(gcov_dir)

wouldn't you want to:
shutil.rmtree(env['GCOV_RESULTS_DIR'])

Comment 7

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #6)
> Comment on attachment 8988447 [details] [diff] [review]
> Cleanup both GCOV_PREFIX and GCOV_RESULTS_DIR directories at the end of a
> test
> 
> Review of attachment 8988447 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/mozharness/scripts/desktop_unittest.py
> @@ +910,3 @@
> >                          )
> > +                        if 'GCOV_RESULTS_DIR' in env:
> > +                            shutil.rmtree(gcov_dir)
> 
> wouldn't you want to:
> shutil.rmtree(env['GCOV_RESULTS_DIR'])

add_per_test_coverage_report calls parse_coverage_artifacts, which removes it, so here I need to remove the GCOV_PREFIX dir.

Comment 8

[Marco Castelluccio [:marco]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)
> Comment on attachment 8988446 [details] [diff] [review]
> Make sure PerTestCoverageUtils doesn't do anything unless per-test mode is
> enabled
> 
> Review of attachment 8988446 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/xpcshell/head.js
> @@ +515,2 @@
> >  
> > +  PerTestCoverageUtils.beforeTestSync();
> 
> how does this work if it defaults to {} ?

The function itself performs the check to see if per-test mode is enabled. If it is enabled, it will reset the counters; if it is not enabled, it will be a no-op.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988446 [details] [diff] [review]
Make sure PerTestCoverageUtils doesn't do anything unless per-test mode is enabled

Review of attachment 8988446 [details] [diff] [review]:
-----------------------------------------------------------------

after irc chat, this is now always enabled (not just present in coverage builds) and will require the env var to activate it.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8988447 [details] [diff] [review]
Cleanup both GCOV_PREFIX and GCOV_RESULTS_DIR directories at the end of a test

Review of attachment 8988447 [details] [diff] [review]:
-----------------------------------------------------------------

I see where we clean up properly.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1658ec07186e
Make sure PerTestCoverageUtils doesn't do anything unless per-test mode is enabled. r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/9f5863545b24
Cleanup both GCOV_PREFIX and GCOV_RESULTS_DIR directories at the end of a test. r=jmaher

Comment 12

[Intermittent Failures Robot]

24 failures in 125 pushes (0.192 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-central: 13
* try: 11

Platform breakdown:
* linux64-ccov: 16
* linux64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1471769&startday=2018-06-28&endday=2018-06-28&tree=trunk

Comment 13

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/1658ec07186e
https://hg.mozilla.org/mozilla-central/rev/9f5863545b24

Comment 14

[Intermittent Failures Robot]

41 failures in 718 pushes (0.057 failures/push) were associated with this bug in the last 7 days. 

This is the #17 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-central: 27
* try: 14

Platform breakdown:
* linux64-ccov: 26
* linux64: 13
* windows2012-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1471769&startday=2018-06-25&endday=2018-07-01&tree=trunk