Closed Bug 872357 Opened 6 years ago Closed 5 years ago

Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpmddsgp\mailtest\Mail\Local Folders\Inbox, error 32

Categories

(MailNews Core :: Backend, defect)

x86_64
Windows 7
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 37.0

People

(Reporter: jcranmer, Assigned: rkent)

References

Details

(Keywords: intermittent-failure)

Attachments

(9 files)

Looks like bug 640371 is the likely culprit for causing this failure. Log failure:
<https://tbpl.mozilla.org/php/getParsedLog.php?id=22909839&tree=Thunderbird-Trunk>


TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 0 - See following stack:
JS frame :: mark_file_region_sparse@../../../resources/mailTestUtils.js:458
JS frame :: growInbox@C:/talos-slave/test/build/xpcshell/tests/mailnews/local/test/unit/test_over4GBMailboxes.js:82
JS frame :: run_test@C:/talos-slave/test/build/xpcshell/tests/mailnews/local/test/unit/test_over4GBMailboxes.js:135
JS frame :: _execute_test@C:\talos-slave\test\build\xpcshell\head.js:325
JS frame :: @-e:1

This landed just before the Aurora merge, so these failures are also present on comm-aurora
Fails only on Windows. I'll check it out. Maybe Windows is more sensitive to the size of the sparse block.
(error 0 - very helpful...)
Assignee: nobody → acelists
Attached patch experiment 1Splinter Review
Attached patch experiment 2Splinter Review
Aryx, I'd need a try run from each of these patches, Windows build only, xpcshell tests only. Thanks.
Ok, so according to https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=de743cdd38a8 the experiment 1 worked (which removes sparse marking on the file) but got killed by xpcshell due to timeout. So we have an easy workaround if I find no better solution (the timeout can be lifted, there is a patch already in bug 794303).

But I try some more experiments.
Attached patch experiment 3Splinter Review
Aryx, could you please make a try build (windows only and xpcshell tests only) with the patch "experiment 3"?
Status: NEW → ASSIGNED
Attached patch experiment 4Splinter Review
Thanks Aryx, but the build was busted at that time. Can you start it again with current trunk?
OK, great, the original problem of the test seems to be solved, CreateFile does no longer fail. But now it timeouts. Could anybody run the test locally and see how long it really takes and if the file (something like c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox) is actually properly marked sparse (can probably seen in Windows explorer in file properties so that Size will be 4GB but Size on disk will be much lower). If that works, and we just need to increase the timeout then there is a pending patch in bug 794303 for that.

Can anybody please apply the 3 patches locally (as one is for mozilla-central I am not sure we could send it to Thunderbird-try) in this order:
https://bugzilla.mozilla.org/attachment.cgi?id=754234
https://bugzilla.mozilla.org/attachment.cgi?id=749005
https://bugzilla.mozilla.org/attachment.cgi?id=749004
and see how it works?
So that doesn't help either.

It looks like the test now times out hanging at pop3d.js regardless of the timeout length:

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [ParseListener_run_test.OnStopRunningUrl : 337] [xpconnect wrapped nsIMsgDatabase] != null

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [ParseListener_run_test.OnStopRunningUrl : 338] true == true

TEST-PASS | thunderbird-binary/mozilla/_tests/xpcshell/mailnews/local/test/unit/test_over4GBMailboxes.js | [downloadUnder4GiB : 169] [nsIMsgIncomingServer: server2] != null

TEST-INFO | (xpcshell/head.js) | test pending (2)

TEST-PASS | ../../../fakeserver/pop3d.js | [readFile : 22] [xpconnect wrapped nsIFile] != null

TEST-PASS | ../../../fakeserver/pop3d.js | [readFile : 23] true == true
<<<<<<<

PROCESS-CRASH | thunderbird-binary\mozilla\_tests\xpcshell\mailnews\local\test\unit\test_over4GBMailboxes.js | application crashed [Unknown top frame]
Crash dump filename: thunderbird-binary\mozilla\_tests\xpcshell\mailnews\local\test\unit\dba680ae-cb08-44f9-80b6-93bfd24974d3.dmp
MINIDUMP_STACKWALK not set, can't process dump.


Any ideas about that?
The actual error code was 32 (ERROR_SHARING_VIOLATION).
https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=f76bd265c3bc
Thanks. You seem to have some interesting fix to show the real error there. Can you check that in? But that ERROR_SHARING_VIOLATION should already been solved in experiment 4. But there now a new error (hand in pop3d.js).
Direct GetLastError() call will not work as expected.
Attachment #761501 - Flags: review?(mbanner)
Comment on attachment 761501 [details] [diff] [review]
[checked in in comment 32] Use ctypes.winLastError

Looks good to me as an improvement to the error codes.
Attachment #761501 - Flags: review?(mbanner) → review+
Summary: Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 0 → Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 32
Masatoshi, you seem to know something about the windows internals, are you able to see why pop3d.js is hanging in the test (with patch "experiment 4" applied)? Is it waiting for something? Locked file or connection blocked by firewall?
Summary: Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:458 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 32 → Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 32
I think that given we haven't worked out where the issue is caused yet, plus we have coverage of this test on Mac and Linux, we should disable it on windows - unless we think there's something really critical here.

This patch just adds skip-if to the manifest to disable it just on windows.
Attachment #773497 - Flags: review?(acelists)
Tweak summary to make it matchable on tbpl test result log.
Summary: Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\docume~1\cltbld\locals~1\temp\tmpp2tfyl\mailtest\Mail\Local Folders\Inbox, error 32 → Perma-orange on Windows: TEST-UNEXPECTED-FAIL | ../../../resources/mailTestUtils.js:444 | Error: CreateFile failed for c:\users\cltbld\appdata\local\temp\tmpmddsgp\mailtest\Mail\Local Folders\Inbox, error 32
Comment on attachment 773497 [details] [diff] [review]
[checked in] Disable test on Windows for now

I think this is an important test for our quest to fix problems around 4GB mailboxes. But if it irritates people to have the test failure always coming up I am OK with disabling the test.
As long as you do not close the bug so that it can stay on my radar.
Hopefully the real fix for the test comes int he future. It may be some simple thing like trying to read a file that is still (unnecessarily) open for writing (which I think Windows disallows, but other OSes may allow).
Attachment #773497 - Flags: review?(acelists) → review+
(In reply to :aceman from comment #39)
> I think this is an important test for our quest to fix problems around 4GB
> mailboxes. But if it irritates people to have the test failure always coming
> up I am OK with disabling the test.

Agreed, but with permanent orange we either need to back out the original patch, or disable the test. Having things permanently orange too easily hides additional failures that may creep in.

> As long as you do not close the bug so that it can stay on my radar.

Definitely not closing the bug.
Comment on attachment 773497 [details] [diff] [review]
[checked in] Disable test on Windows for now

https://hg.mozilla.org/comm-central/rev/9377331eb262

(wrong commit comment, but I've pointed that bug at this one).
Attachment #773497 - Attachment description: Disable test on Windows for now → [checked in] Disable test on Windows for now
Taken onto aurora as well https://hg.mozilla.org/releases/comm-aurora/rev/dde128e46dde (not flagging due to bug flag complexities).
Keywords: checkin-needed
Whiteboard: [check in only patch "Use ctypes.winLastError"]
Keywords: checkin-needed
Whiteboard: [check in only patch "Use ctypes.winLastError"]
Attachment #761501 - Attachment description: Use ctypes.winLastError → [checked in in comment 32] Use ctypes.winLastError
Whiteboard: [test disabled on Windows]
rkent, do you by chance have a Windows machine with dev environment? Would you be able to look into this? This would be nice to have for the 4GB work.
Flags: needinfo?(kent)
Yes this is an important test, and yes I do the vast majority of my dev on Windows. But it is hard to see where I will find time to work on this anytime soon.
Flags: needinfo?(kent)
QA Contact: kent
Attached patch win4GBtest.patchSplinter Review
Here's my fix for this.
Nice. Suyash, can you test the patch?
Assignee: acelists → kent
Flags: needinfo?(syshagarwal)
Attached file testlog.log
Sorry for taking so long, this is the test result on Windows.
Flags: needinfo?(syshagarwal)
Hi aceman,

I looked again at the changes that you proposed earlier and realized they were good, but there we just a few more tweaks needed. See this patch and tell me what you think. See try run:

https://treeherder.mozilla.org/ui/#/jobs?repo=try-comm-central&revision=d63a86ff10b6
Attachment #8547101 - Flags: review?(acelists)
Comment on attachment 8547101 [details] [diff] [review]
combine rkent and aceman changes

Review of attachment 8547101 [details] [diff] [review]:
-----------------------------------------------------------------

As it reuses my code and if it solves the problem on Windows, then I am fine with the changes. It works on Linux.
Attachment #8547101 - Flags: review?(acelists) → review+
I'll go ahead and land this. The worst that could go wrong is that the test starts to fail. Also, I really need this so that I can help with the 4GB folder issues and reviews.
Comment on attachment 8547101 [details] [diff] [review]
combine rkent and aceman changes

Checked in https://hg.mozilla.org/comm-central/rev/cdc3b6b7b4af
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Whiteboard: [test disabled on Windows]
Target Milestone: --- → Thunderbird 37.0
You need to log in before you can comment on or make changes to this bug.