1447616 - Massive Xpcshell test failure 2018-03-21: 29 failures

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Jorg K (CEST = GMT+2)]

TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_offlinePlayback.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_offlinePlayback.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_mailstoreConverter.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_mailstoreConverter.js | testMaildirConversion - [testMaildirConversion : 85] 0 == 1000
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_mailstoreConverter.js | testMaildirConversion - [testMaildirConversion : 131] false == true
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_mailstoreConverter.js | testMaildirConversion - [testMaildirConversion : 257] A promise chain failed to handle a rejection: [Exception... "Abort" nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame :: /Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js :: _abort_failed_test :: line 723" data: no] - stack: _do_main@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:211:3
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_copyThenMoveManual.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_copyThenMoveManual.js | verifyFolders1 - [verifyFolders1 : 58] 0 == 1
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_converterDeferredAccount.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_converterDeferredAccount.js | testMaildirConversion - [testMaildirConversion : 90] 0 == 1000
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_converterDeferredAccount.js | testMaildirConversion - [testMaildirConversion : 177] false == true
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_converterDeferredAccount.js | testMaildirConversion - [testMaildirConversion : 257] A promise chain failed to handle a rejection: [Exception... "Abort" nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame :: /Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js :: _abort_failed_test :: line 723" data: no] - stack: _do_main@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:211:3
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_folderCompact.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_folderCompact.js | lastTestCheck - [lastTestCheck : 290] "undefined" == 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_quarantineFilterMove.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/base/test/unit/test_quarantineFilterMove.js | verifyFolders1 - [verifyFolders1 : 45] 0 == 2
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_downloadOffline.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_imapID.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_offlineCopy.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_subfolderLocation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_cacheParts.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_downloadOffline.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_imapID.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_offlineCopy.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/imap/test/unit/test_subfolderLocation.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/local/test/unit/test_pop3FilterActions.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/local/test/unit/test_pop3MoveFilter.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/local/test/unit/test_pop3MoveFilter2.js | xpcshell return code: 0
TEST-UNEXPECTED-FAIL | comm/mailnews/local/test/unit/test_pop3MoveFilter2.js | verifyFolders2 - [verifyFolders2 : 56] 0 == 1

M-C last good: 6d0f84afc194e287b6f144d17cd88371a5
M-C first bad: e636edf00e6fbdc3206c9df4a1548ae38b
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6d0f84afc194e287b6f144d17cd88371a5&tochange=e636edf00e6fbdc3206c9df4a1548ae38b
which is the merge this morning.

Log
https://taskcluster-artifacts.net/NIQGjCpNR8CSIGLb_nMW3g/0/public/logs/live_backing.log
shows:
test_converterDeferredAccount.js, line 176: ReferenceError: reference to undefined property "error"
which is from:
log.error("Conversion failed: " + reason.error);
but that looks just like a secondary problem where something failed and the error reporting failed as well :-(

Looking for TypeError I see 22 matches, for example:
TypeError: IMAPPump.daemon is null at /Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/tests/comm/mailnews/imap/test/unit/test_imapID.js:31
TypeError: IMAPPump.inbox is null at /Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/tests/comm/mailnews/imap/test/unit/test_offlineCopy.js:125
TypeError: gUidValidity is undefined at /Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/tests/comm/mailnews/imap/test/unit/test_cacheParts.js:145

Looks like this comes from
https://hg.mozilla.org/mozilla-central/rev/e2a78fa784c6
Bug 1446833: Part 2 - Stop using Task.jsm in xpcshell/head.js. r=florian

Or maybe not since test_cacheParts.js already uses
function run_test() {
  tests.forEach(x => add_task(x));
  run_next_test();
}

Aceman, could you assist? Florian, any hints?

Mozmill is not affected and this looks like test failures and not functional failures.

Comment 1

[Florian Quèze [:florian]]

These tests use generators as tasks. We removed all occurrences of that in m-c in bug 1353542 and bug 1374282.

You need to replace all the 'yield' with 'await', and all the 'function*' with 'async function' and that should work. I have scripts in the bugs mentioned above that could do that automatically. They may have bitrotted by now...

I couldn't just run them in all of Thunderbird's code because gloda uses plenty of generators :-(. Some of the work will need to happen by hand. Or someone needs to convert gloda to use async functions instead of its own magic based on generators.

Comment 2

[:aceman]

comm/mailnews/imap/test/unit/test_offlinePlayback.js:
04:21:35     INFO -  Unexpected exception NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.getChildNamed]
04:21:35     INFO -  prepareToGoOffline@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/tests/comm/mailnews/imap/test/unit/test_offlinePlayback.js:28:21
04:21:35     INFO -  run_next_test/_run_next_test/<@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:1418:22
04:21:35     INFO -  async*_run_next_test@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:1418:10
04:21:35     INFO -  run@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:669:9
04:21:35     INFO -  _do_main@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:211:3
04:21:35     INFO -  _execute_test@/Users/cltbld/tasks/task_1521630803/build/tests/xpcshell/head.js:517:5

Comment 3

[Jorg K (CEST = GMT+2)]

(In reply to Florian Quèze [:florian] from comment #1)
> These tests use generators as tasks. We removed all occurrences of that in
> m-c in bug 1353542 and bug 1374282.
Hmm, but these landed long ago.

> Or
> someone needs to convert gloda to use async functions instead of its own
> magic based on generators.
That's been on the books for ages.

I'm still interested to know what caused the bustage we're seeing now.

Comment 4

[Florian Quèze [:florian]]

(In reply to Jorg K (GMT+1) from comment #3)

> I'm still interested to know what caused the bustage we're seeing now.

You said it in comment 0:
(In reply to Jorg K (GMT+1) from comment #0)

> Looks like this comes from
> https://hg.mozilla.org/mozilla-central/rev/e2a78fa784c6
> Bug 1446833: Part 2 - Stop using Task.jsm in xpcshell/head.js. r=florian


Task.jsm was compatible with both generators and async functions. Yesterday we made xpcshell tests stop using Task.jsm, so now only async functions are supported.

Comment 5

[Jorg K (CEST = GMT+2)]

In case anyone is following, I'm doing the manual conversion, starting with IMAP. Two passed so far.

Comment 6

[Jorg K (CEST = GMT+2)]

Attachment: 1447616-task.patch

This fixes
IMAP:  mozilla/mach xpcshell-test mailnews/imap/test/unit/

Comment 7

[Jorg K (CEST = GMT+2)]

Attachment: 1447616-task.patch (v2)

This should do it.
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=e2d21cba6e0653e5c7f48ee2a0921f50cc514fc6

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0a725681fe43
Port bug 1446833 (Part 2): don't rely on Task.jsm. rs=bustage-fix

Comment 9

[Jorg K (CEST = GMT+2)]

Attachment: 1447616-task-follow-up.patch

Missed some, how strange. test_autoReply.js doesn't run on Mac, but the other one passed on try. Strange.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/e2d00a308beb
Port bug 1446833 (Part 2): don't rely on Task.jsm (take 2). rs=bustage-fix DONTBUILD

Comment 11

[Intermittent Failures Robot]

5 failures in 822 pushes (0.006 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* comm-central: 5

Platform breakdown:
* osx-10-10: 2
* windows10-64: 1
* linux64: 1
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1447616&startday=2018-03-20&endday=2018-03-26&tree=trunk