462269 - [MacOSX] test_mailboxes.js (and test_mailtoURL.js, test_nntpPassword.js and test_nntpPassword2.js) fails intermittently

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Serge Gautherie (:sgautherie)]

Examples:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225214219.1225220426.14133.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/28 10:16:59
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225220060.1225226418.714.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/28 11:54:20
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225238040.1225245465.21714.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/28 16:54:00
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225251107.1225257641.23349.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/28 20:31:47
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225292698.1225299092.3954.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/29 08:04:58

{
NEXT ERROR TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_imap/unit/test_mailboxes.js | test failed, see log
../../../mozilla/_tests/xpcshell-simple/test_imap/unit/test_mailboxes.js.log:
>>>>>>>
uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: /builds/slave/comm-central-macosx/build/mozilla/../mailnews/test/resources/mailDirService.js :: <TOP_LEVEL> :: line 91"  data: no]
*** test pending
Wants directory: IMapMD
Wants directory: MFCaF
[Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIMsgIncomingServer.performExpand]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: ../../../mozilla/_tests/xpcshell-simple/test_imap/unit/test_mailboxes.js :: run_test :: line 12"  data: no]
*** FAIL ***

<<<<<<<
}

Comment 1

[Serge Gautherie (:sgautherie)]

(First time I notice this one:)

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225341488.1225343173.4070.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/29 21:38:08

{
NEXT ERROR TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword.js | test failed, see log
../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword.js.log:
>>>>>>>
uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: /builds/slave/comm-central-macosx/build/mozilla/../mailnews/test/resources/mailDirService.js :: <TOP_LEVEL> :: line 91"  data: no]
../../../mozilla/_tests/xpcshell-simple/test_news/unit/head_server_setup.js:21: TypeError: groups is undefined
*** test pending
NNTP Protocol test news:* failed for type RFC 977:
Commands called: undefined
*** exiting
NEXT ERROR *** TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword.js | TypeError: URLCreator is undefined
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/head.js :: do_throw :: line 101
JS frame :: ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword.js :: run_test :: line 47
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/tail.js :: _execute_test :: line 41
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38
2147500036
*** FAIL ***

<<<<<<<
}

Comment 2

[Serge Gautherie (:sgautherie)]

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225398115.1225404362.4935.gz
MacOSX 10.4 comm-central dep unit test on 2008/10/30 13:21:55
TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_imap/unit/test_mailboxes.js | test failed, see log

Comment 3

[Serge Gautherie (:sgautherie)]

(First time I notice this one:)

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225640040.1225646528.6699.gz
MacOSX 10.4 comm-central dep unit test on 2008/11/02 07:34:00

{
NEXT ERROR TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_compose/unit/test_mailtoURL.js | test failed, see log
../../../mozilla/_tests/xpcshell-simple/test_compose/unit/test_mailtoURL.js.log:
>>>>>>>
uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: /builds/slave/comm-central-macosx/build/mozilla/../mailnews/test/resources/mailDirService.js :: <TOP_LEVEL> :: line 91"  data: no]
*** test pending
*** test finished
*** exiting
*** PASS ***

<<<<<<<
}

Comment 4

[Robert Kaiser]

Mark, any idea what could be up there? It seems like this has intensified now and we're failing most of the time now :(

Comment 5

[Mark Banner (:standard8)]

Strange, all the errors imply that the recursive remove of the mailtest directory (that is used as the profile) is failing occasionally.

I think I've seen this once on the thunderbird tinderboxes as well.

The best thing is to probably extend the debug and see if we can catch where/why it is failing. I'll see if I can knock something up soon.

Comment 6

[Mark Banner (:standard8)]

Attachment: [checked in] Add some debugging info

If nsIFile.remove fails, this sets up a manual recursive remove, adds some debugging and hopefully we'll find out what isn't being removed, and maybe why.

This should give us some information as to what is going on and give us enough to file a real bug on the actual issue.

SM is seeing this very frequently, TB only occasionally, so I'd like to get it in and give it a run to see what happens.

Comment 7

[David :Bienvenu]

Comment on attachment 352301 [details] [diff] [review]
[checked in] Add some debugging info

someone's probably still got the file open - I guess this will tell us which file(s).

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 352301 [details] [diff] [review]
[checked in] Add some debugging info

Checked in: http://hg.mozilla.org/comm-central/rev/9fded7cdb0a8

Comment 9

[Mark Banner (:standard8)]

At last we've just had a case of this with the patch in:

NEXT ERROR TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_server.js | test failed, see log
../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_server.js.log:
>>>>>>>
Couldn't recursive remove directory: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: /builds/slave/comm-central-macosx/build/mozilla/../mailnews/test/resources/mailDirService.js :: <TOP_LEVEL> :: line 99"  data: no]
Trying manually...
Processing Directory /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest
Processing Directory /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News
Processing Directory /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/localhost
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/localhost/test.filter.msf
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/localhost/test.subscribe.empty.msf
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/localhost/test.subscribe.simple.msf
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/localhost
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News/newsrc-localhost
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/News
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/panacea.dat
Removing /builds/slave/comm-central-macosx/build/objdir/mozilla/_tests/mailtest/signons.txt
...manual done
uncaught exception: 2147500037
../../../mozilla/_tests/xpcshell-simple/test_news/unit/head_server_setup.js:21: TypeError: groups is undefined
*** test pending
NNTP Protocol test news:* failed for type RFC 977:
Commands called: undefined
*** exiting
NEXT ERROR *** TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_server.js | TypeError: URLCreator is undefined
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/head.js :: do_throw :: line 101
JS frame :: ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_server.js :: testRFC977 :: line 101
JS frame :: ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_server.js :: run_test :: line 111
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/tail.js :: _execute_test :: line 41
JS frame :: /builds/slave/comm-central-macosx/build/mozilla/tools/test-harness/xpcshell-simple/execute_test.js :: <TOP_LEVEL> :: line 38
2147500036
*** FAIL ***

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1229117258.1229122533.11862.gz#err0

I've not got time to look at it in detail, just dumping it here now for the record.

Comment 10

[Serge Gautherie (:sgautherie)]

(In reply to comment #9)

Another, same:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1229183881.1229189131.3444.gz
MacOSX 10.4 comm-central dep unit test on 2008/12/13 07:58:01

Comment 11

[Serge Gautherie (:sgautherie)]

(In reply to comment #9)

"same", but with test_nntpPassword.js:
http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird3.0/1229799642.1229801219.4028.gz
MacOSX 10.4 comm-central check on 2008/12/20 11:00:42

Comment 12

[Serge Gautherie (:sgautherie)]

(In reply to comment #11)

A log a little different:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1229909998.1229915265.3655.gz
MacOSX 10.4 comm-central dep unit test on 2008/12/21 17:39:58

Comment 13

[Serge Gautherie (:sgautherie)]

Another occurrence:
http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1230254740.1230259982.25129.gz
MacOSX 10.4 comm-central dep unit test on 2008/12/25 17:25:40

Comment 14

[Serge Gautherie (:sgautherie)]

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1230460255.1230465445.9296.gz
MacOSX 10.4 comm-central dep unit test on 2008/12/28 02:30:55

Comment 15

[Serge Gautherie (:sgautherie)]

New TB/m-c has this too, on test_nntpPassword2.js;
{
http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1230745274.1230746523.14329.gz
MacOSX 10.4 comm-central check on 2008/12/31 09:41:14

TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword2.js | test failed, see log
*** TEST-UNEXPECTED-FAIL | ../../../mozilla/_tests/xpcshell-simple/test_news/unit/test_nntpPassword2.js | TypeError: groups is undefined

http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1230758916.1230760035.9745.gz
MacOSX 10.4 comm-central check on 2008/12/31 13:28:36
}

Comment 16

[Mark Banner (:standard8)]

Yep, they seem like they have got worse. I'll try and take a look next week.

Don't worry about any more log references 1 or 2 are enough.

Comment 17

[Mark Banner (:standard8)]

Attachment: [checked in] Rework debugging info

Ok, the previous try didn't give enough info, apparently the recursive remove just fails without removing any files.

Since we've got check boxes on trunk as well, we're failing even more on mac, I'd like to swap around the debugging to see if that leads to anything - i.e. we'll use our manual function to iterate the directory and see if there's a specific file that is failing, the recursive remove will be the fallback, though we'll still re-throw the exception if we fail on the manual remove.

Comment 18

[Mark Banner (:standard8)]

Comment on attachment 355400 [details] [diff] [review]
[checked in] Rework debugging info

Checked in: http://hg.mozilla.org/comm-central/rev/cce5b0c4d780

I'll be watching the tree over the next couple of days to see what the effect is.

Comment 19

[Mark Banner (:standard8)]

(In reply to comment #18)
> (From update of attachment 355400 [details] [diff] [review])
> Checked in: http://hg.mozilla.org/comm-central/rev/cce5b0c4d780

and a small bustage fix: http://hg.mozilla.org/comm-central/rev/4931e158775b

Comment 20

[Mark Banner (:standard8)]

That last change seems to have "fixed" the issue - the boxes have been green now for a day or so at least. I'm going to let them run for this week, and if they are still passing on Friday, then I'll raise a core issue on the nsIFile.remove function as I think that's where the real issue lies.

Comment 21

[Mark Banner (:standard8)]

Attachment: [checked in] Update the comments, remove some print statements

I've come to the conclusion this is a core issue that I'm not going to be able to debug here. I've therefore raised bug 473385 to cover the problem.

Rather than back out the debugging info I think we need to keep the tinderboxes stable, so I'd like to keep it in, but update the comments (and remove a couple of print statements). I'll close this bug once this is in, and bug 473385 can cover the real problem.

Comment 22

[Mark Banner (:standard8)]

Comment on attachment 356723 [details] [diff] [review]
[checked in] Update the comments, remove some print statements

Checked in: http://hg.mozilla.org/comm-central/rev/cb9492232f17

Comment 23

[Mark Banner (:standard8)]

That last patch is now checked in. Therefore closing this bug and hopefully bug 473385 will come up with the real fix.

Comment 24

[Legacy TBPL/Treeherder Robot]

Standard8
https://tbpl.mozilla.org/php/getParsedLog.php?id=16269244&tree=Thunderbird-Trunk
TB Rev4 MacOSX Snow Leopard 10.6 comm-central debug test xpcshell on 2012-10-19 08:43:07
slave: talos-r4-snow-073

TEST-UNEXPECTED-FAIL | /Users/cltbld/talos-slave/test/build/xpcshell/tests/mailnews/news/test/unit/test_nntpPassword2.js | test failed (with xpcshell return code: 1), see following log:
PROCESS-CRASH | /Users/cltbld/talos-slave/test/build/xpcshell/tests/mailnews/news/test/unit/test_nntpPassword2.js | application crashed (minidump found)
Thread 0 (crashed)