872357 - Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpmddsgp\mailtest\Mail\Local Folders\Inbox, error 32

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[Joshua Cranmer [:jcranmer]]

Looks like bug 640371 is the likely culprit for causing this failure. Log failure:
<https://tbpl.mozilla.org/php/getParsedLog.php?id=22909839&tree=Thunderbird-Trunk>


TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:
JS frame :: mark_file_region_sparse@../../../resources/mailTestUtils.js:458
JS frame :: growInbox@C:/talos-slave/test/build/xpcshell/tests/mailnews/local/test/unit/test_over4GBMailboxes.js:82
JS frame :: run_test@C:/talos-slave/test/build/xpcshell/tests/mailnews/local/test/unit/test_over4GBMailboxes.js:135
JS frame :: _execute_test@C:\talos-slave\test\build\xpcshell\head.js:325
JS frame :: @-e:1

This landed just before the Aurora merge, so these failures are also present on comm-aurora

Comment 1

[:aceman]

Fails only on Windows. I'll check it out. Maybe Windows is more sensitive to the size of the sparse block.
(error 0 - very helpful...)

Comment 2

[:aceman]

Attachment: experiment 1

Comment 3

[:aceman]

Attachment: experiment 2

Comment 4

[:aceman]

Aryx, I'd need a try run from each of these patches, Windows build only, xpcshell tests only. Thanks.

Comment 5

[:aceman]

Ok, so according to https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=de743cdd38a8 the experiment 1 worked (which removes sparse marking on the file) but got killed by xpcshell due to timeout. So we have an easy workaround if I find no better solution (the timeout can be lifted, there is a patch already in bug 794303).

But I try some more experiments.

Comment 6

[:aceman]

Attachment: experiment 3

Comment 7

[:aceman]

Aryx, could you please make a try build (windows only and xpcshell tests only) with the patch "experiment 3"?

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Sorry for taking so long: https://tbpl.mozilla.org/?tree=Thunderbird-Try&showall=1&rev=0e2685cab82b

Comment 9

[:aceman]

Attachment: experiment 4

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Pushed to Thunderbird-Try: https://tbpl.mozilla.org/?tree=Thunderbird-Try&showall=1&rev=ff51077c09bc

Comment 11

[:aceman]

Thanks Aryx, but the build was busted at that time. Can you start it again with current trunk?

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://tbpl.mozilla.org/?tree=Thunderbird-Try&showall=1&rev=240e68c1346c

Comment 13

[:aceman]

OK, great, the original problem of the test seems to be solved, CreateFile does no longer fail. But now it timeouts. Could anybody run the test locally and see how long it really takes and if the file (something like c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox) is actually properly marked sparse (can probably seen in Windows explorer in file properties so that Size will be 4GB but Size on disk will be much lower). If that works, and we just need to increase the timeout then there is a pending patch in bug 794303 for that.

Can anybody please apply the 3 patches locally (as one is for mozilla-central I am not sure we could send it to Thunderbird-try) in this order:
https://bugzilla.mozilla.org/attachment.cgi?id=754234
https://bugzilla.mozilla.org/attachment.cgi?id=749005
https://bugzilla.mozilla.org/attachment.cgi?id=749004
and see how it works?

Comment 14

[:aceman]

So that doesn't help either.

It looks like the test now times out hanging at pop3d.js regardless of the timeout length:

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [ParseListener_run_test.OnStopRunningUrl : 337] [xpconnect wrapped nsIMsgDatabase] != null

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [ParseListener_run_test.OnStopRunningUrl : 338] true == true

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [downloadUnder4GiB : 169] [nsIMsgIncomingServer: server2] != null

TEST-INFO | (xpcshell/head.js) | test pending (2)

TEST-PASS | ../../../fakeserver/pop3d.js | [readFile : 22] [xpconnect wrapped nsIFile] != null

TEST-PASS | ../../../fakeserver/pop3d.js | [readFile : 23] true == true
<<<<<<<

PROCESS-CRASH | thunderbird-binary\mozilla\_tests\xpcshell\mailnews\local\test\unit\test_over4GBMailboxes.js | application crashed [Unknown top frame]
Crash dump filename: thunderbird-binary\mozilla\_tests\xpcshell\mailnews\local\test\unit\dba680ae-cb08-44f9-80b6-93bfd24974d3.dmp
MINIDUMP_STACKWALK not set, can't process dump.


Any ideas about that?

Comment 15

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23865403&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central debug test xpcshell on 2013-06-06 09:22:48
slave: talos-r3-w7-113

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpqgj02q\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 16

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23836793&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central debug test xpcshell on 2013-06-05 16:16:48
slave: talos-r3-w7-111

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpa5za81\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 17

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23766215&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central debug test xpcshell on 2013-06-04 06:34:47
slave: talos-r3-w7-115

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpezn1g7\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 18

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23864978&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2013-06-06 09:22:50
slave: talos-r3-xp-082

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpkg2c8h\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 19

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23864129&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central opt test xpcshell on 2013-06-06 09:09:09
slave: talos-r3-xp-075

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpecum6h\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 20

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23916732&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2013-06-07 12:54:53
slave: talos-r3-xp-100

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmp3bm_cu\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 21

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23917541&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central debug test xpcshell on 2013-06-07 12:54:44
slave: talos-r3-w7-113

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpsend5x\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 22

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23936224&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central opt test xpcshell on 2013-06-08 04:44:25
slave: talos-r3-w7-117

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmp8jgr2j\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 23

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23914987&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central opt test xpcshell on 2013-06-07 12:35:57
slave: talos-r3-xp-082

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpz4ac_h\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:
TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\toolkit\mozapps\update\test_svc\unit\test_0114_general_svc.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | C:/talos-slave/test/build/xpcshell/tests/toolkit/mozapps/update/test_svc/unit/head_update.js |  == applied - See following stack:

Comment 24

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23915772&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2013-06-07 12:39:08
slave: talos-r3-xp-070

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpn5mzga\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 25

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23916619&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central debug test xpcshell on 2013-06-07 12:38:47
slave: talos-r3-w7-107

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmptwgzdp\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 26

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=23915686&tree=Thunderbird-Trunk
TB Rev3 WINNT 6.1 comm-central opt test xpcshell on 2013-06-07 12:35:45
slave: talos-r3-w7-110

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpudsdor\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 27

[Masatoshi Kimura [:emk]]

The actual error code was 32 (ERROR_SHARING_VIOLATION).
https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=f76bd265c3bc

Comment 28

[:aceman]

Thanks. You seem to have some interesting fix to show the real error there. Can you check that in? But that ERROR_SHARING_VIOLATION should already been solved in experiment 4. But there now a new error (hand in pop3d.js).

Comment 29

[Masatoshi Kimura [:emk]]

Attachment: [checked in in comment 32] Use ctypes.winLastError

Direct GetLastError() call will not work as expected.

Comment 30

[Mark Banner (:standard8)]

Comment on attachment 761501 [details] [diff] [review]
[checked in in comment 32] Use ctypes.winLastError

Looks good to me as an improvement to the error codes.

Comment 31

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=24065676&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central debug test xpcshell on 2013-06-12 10:05:44
revision: 188ea3049daf
slave: talos-r3-xp-113

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmp6os0jj\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 32

[Masatoshi Kimura [:emk]]

https://hg.mozilla.org/comm-central/rev/a458adca8045

Comment 33

[Legacy TBPL/Treeherder Robot]

mkmelin
https://tbpl.mozilla.org/php/getParsedLog.php?id=24103698&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central opt test xpcshell on 2013-06-13 05:05:03
revision: 47d08a243bde
slave: talos-r3-xp-118

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpfafogu\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:

Comment 34

[:aceman]

Masatoshi, you seem to know something about the windows internals, are you able to see why pop3d.js is hanging in the test (with patch "experiment 4" applied)? Is it waiting for something? Locked file or connection blocked by firewall?

Comment 35

[Legacy TBPL/Treeherder Robot]

jcranmer
https://tbpl.mozilla.org/php/getParsedLog.php?id=25131972&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central opt test xpcshell on 2013-07-10 10:35:48
revision: d6e582202f1e
slave: talos-r3-xp-125

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpu1s1es\mailtest\Mail\Local Folders\Inbox, error 32 - See following stack:

Comment 36

[Legacy TBPL/Treeherder Robot]

jcranmer
https://tbpl.mozilla.org/php/getParsedLog.php?id=25134170&tree=Thunderbird-Trunk
TB Rev3 WINNT 5.1 comm-central opt test xpcshell on 2013-07-10 11:20:57
revision: ec27e2fa203e
slave: talos-r3-xp-127

TEST-UNEXPECTED-FAIL | C:\talos-slave\test\build\xpcshell\tests\mailnews\local\test\unit\test_over4GBMailboxes.js | test failed (with xpcshell return code: 0), see following log:
TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpmhay1-\mailtest\Mail\Local Folders\Inbox, error 32 - See following stack:

Comment 37

[Mark Banner (:standard8)]

Attachment: [checked in] Disable test on Windows for now

I think that given we haven't worked out where the issue is caused yet, plus we have coverage of this test on Mac and Linux, we should disable it on windows - unless we think there's something really critical here.

This patch just adds skip-if to the manifest to disable it just on windows.

Comment 38

[:aceman]

Tweak summary to make it matchable on tbpl test result log.

Comment 39

[:aceman]

Comment on attachment 773497 [details] [diff] [review]
[checked in] Disable test on Windows for now

I think this is an important test for our quest to fix problems around 4GB mailboxes. But if it irritates people to have the test failure always coming up I am OK with disabling the test.
As long as you do not close the bug so that it can stay on my radar.
Hopefully the real fix for the test comes int he future. It may be some simple thing like trying to read a file that is still (unnecessarily) open for writing (which I think Windows disallows, but other OSes may allow).

Comment 40

[Mark Banner (:standard8)]

(In reply to :aceman from comment #39)
> I think this is an important test for our quest to fix problems around 4GB
> mailboxes. But if it irritates people to have the test failure always coming
> up I am OK with disabling the test.

Agreed, but with permanent orange we either need to back out the original patch, or disable the test. Having things permanently orange too easily hides additional failures that may creep in.

> As long as you do not close the bug so that it can stay on my radar.

Definitely not closing the bug.

Comment 41

[Mark Banner (:standard8)]

Comment on attachment 773497 [details] [diff] [review]
[checked in] Disable test on Windows for now

https://hg.mozilla.org/comm-central/rev/9377331eb262

(wrong commit comment, but I've pointed that bug at this one).

Comment 42

[Mark Banner (:standard8)]

Taken onto aurora as well https://hg.mozilla.org/releases/comm-aurora/rev/dde128e46dde (not flagging due to bug flag complexities).

Comment 43

[:aceman]

rkent, do you by chance have a Windows machine with dev environment? Would you be able to look into this? This would be nice to have for the 4GB work.

Comment 44

[Kent James (:rkent)]

Yes this is an important test, and yes I do the vast majority of my dev on Windows. But it is hard to see where I will find time to work on this anytime soon.

Comment 45

[Kent James (:rkent)]

Attachment: win4GBtest.patch

Here's my fix for this.

Comment 46

[:aceman]

Nice. Suyash, can you test the patch?

Comment 47

[Suyash Agarwal (:sshagarwal)]

Attachment: testlog.log

Sorry for taking so long, this is the test result on Windows.

Comment 48

[Kent James (:rkent)]

Attachment: combine rkent and aceman changes

Hi aceman,

I looked again at the changes that you proposed earlier and realized they were good, but there we just a few more tweaks needed. See this patch and tell me what you think. See try run:

https://treeherder.mozilla.org/ui/#/jobs?repo=try-comm-central&revision=d63a86ff10b6

Comment 49

[:aceman]

Comment on attachment 8547101 [details] [diff] [review]
combine rkent and aceman changes

Review of attachment 8547101 [details] [diff] [review]:
-----------------------------------------------------------------

As it reuses my code and if it solves the problem on Windows, then I am fine with the changes. It works on Linux.

Comment 50

[Kent James (:rkent)]

I'll go ahead and land this. The worst that could go wrong is that the test starts to fail. Also, I really need this so that I can help with the 4GB folder issues and reviews.

Comment 51

[Kent James (:rkent)]

Comment on attachment 8547101 [details] [diff] [review]
combine rkent and aceman changes

Checked in https://hg.mozilla.org/comm-central/rev/cdc3b6b7b4af