Closed Bug 361730 Opened 13 years ago Closed 8 years ago

Seamonkey _deletes_ mail file if the mail file's size is greater than 2GiB and the .msf file has zero length

Categories

(MailNews Core :: Backend, defect, critical)

x86
Linux
defect
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: baldauf--2015--bugzilla.mozilla.org, Assigned: Bienvenu)

Details

(Keywords: dataloss, fixed1.8.0.9, fixed1.8.1.1, Whiteboard: needs landing)

Attachments

(1 file, 1 obsolete file)

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.8) Gecko/20060911 SUSE/1.0.6-0.1 SeaMonkey/1.0.6
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.8) Gecko/20060911 SUSE/1.0.6-0.1 SeaMonkey/1.0.6

At a friends computer, I could reproduce severe data loss. It seems that Seamonkey Mail cannot handle files with length >2GiB, at least on i686 (32 bit machine), and goes wild :-(

I could verify this using strace:

[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Inbox", {st_mode=S_IFREG|0644, st_size=596951668, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Trash", {st_mode=S_IFREG|0600, st_size=199594990, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", {st_mode=S_IFREG|0600, st_size=2148532224, ...}) = 0
[pid  9402] open("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666 <unfinished ...>
[pid  9404] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  9404] gettimeofday({1164334227, 629206}, NULL) = 0
[pid  9404] futex(0x80a5368, FUTEX_WAKE, 1) = 0
[pid  9404] gettimeofday({1164334227, 629358}, NULL) = 0
[pid  9404] gettimeofday({1164334227, 629414}, NULL) = 0
[pid  9404] clock_gettime(CLOCK_REALTIME, {1164334227, 629469000}) = 0
[pid  9404] futex(0x80a53d0, FUTEX_WAIT, 251, {1, 187945000} <unfinished ...>
[pid  9402] <... open resumed> )        = 47
[pid  9402] lseek(47, 0, SEEK_CUR)      = 0
[pid  9402] lseek(47, 0, SEEK_END)      = 0
[pid  9402] lseek(47, 0, SEEK_SET)      = 0
[pid  9402] close(47)                   = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Drafts", {st_mode=S_IFREG|0600, st_size=50566887, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Templates", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
[pid  9402] open("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 47
[pid  9402] fstat64(47, {st_mode=S_IFDIR|0755, st_size=744, ...}) = 0
[pid  9402] fcntl64(47, F_SETFD, FD_CLOEXEC) = 0
[pid  9402] mmap2(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb1cbd000
[pid  9402] getdents64(47, /* 24 entries */, 131072) = 824
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop", {st_mode=S_IFDIR|0755, st_size=744, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent.msf", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop", {st_mode=S_IFDIR|0755, st_size=744, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Drafts.msf", {st_mode=S_IFREG|0644, st_size=3863, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Drafts", {st_mode=S_IFREG|0600, st_size=50566887, ...}) = 0
[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop", {st_mode=S_IFDIR|0755, st_size=744, ...}) = 0

At first, seamonkey gets stat data for that file "Mail/pop/Sent". You see, the file has size 2148532224, as "st_size=2148532224" tells.

[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", {st_mode=S_IFREG|0600, st_size=2148532224, ...}) = 0

Then seamonkey opens the file for writing, using the O_TRUNC flag, which means that the file is to be truncated (its length set to zero)

[pid  9402] open("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666 <unfinished ...>

We can even verify this result, as now the file size is 0, as "st_size=0" tells.

[pid  9402] stat64("/home/heike/.mozilla/heike/zqf8bswn.slt/Mail/pop/Sent", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0


Reproducible: Always

Steps to Reproduce:
1. Create a very large mail file in the appropriate directory. In my specific case, this was "Mail/pop/Sent" file. This file should be greater than 2GiB. Using "dd if=/dev/zero of=Sent bs=1M count=2049" is fine.
2. Create the corresponding .msf file (e.g. "Mail/pop/Sent,msf"). This file should have zero length. Using "rm Sent.msf", "touch Sent.msf" is fine.
3. Start Seamonkey Mail
4. Look at the size of the Mail file.

Actual Results:  
The size of "Sent" is 0

Expected Results:  
The size of "Sent" should remain greater than 2GiB.

This problem has been verified both on Seamonkey 1.0.2 and Seamonkey 1.0.6.
I can confirm this for SeaMonkey 1.1b [Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061105 SeaMonkey/1.1b].
Status: UNCONFIRMED → NEW
Ever confirmed: true
So what is this http://lxr.mozilla.org/seamonkey/source/xpcom/io/nsLocalFileUnix.cpp#1110 ?

The source code line

  LL_UI2L(*aFileSize, (PRUint32)mCachedStat.st_size);

resolves to

  (*aFileSize) = (PRInt64) (PRUint32) mCachedStat.st_size;

This may not be the reason for this bug, but this is still silly! First down-convert to 32 bits and then up-convert to 64 bits again.
JFTR: SeaMonkey 1.1b [Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.1) Gecko/20061102 SeaMonkey/1.1b] is *NOT* affected, it already seems to be using 64 bit stuff.

I assume that Thunderbird is also affected?
I've been looking at this on the mac, which I think has the same bug. I believe the bug is in the xpcom/obsolete/filestream code, but I'm still debugging it. It's also a core bug, not specific to seamonkey.
Hello,

I was able to obtain a stack trace to where the bug occurs:

(gdb) bt
#0  *__GI___xstat (vers=3, name=0xa5ae800 "/home/testie/.mozilla/testie/seletqfv.slt/Mail/Local Folders/Testfolder", buf=0x7fd4be84)
    at ../sysdeps/unix/sysv/linux/i386/xstat.c:92
#1  0x7731ea58 in stat (__path=0xa5ae800 "/home/testie/.mozilla/testie/seletqfv.slt/Mail/Local Folders/Testfolder", __statbuf=0x7fd4be84) at /usr/include/sys/stat.h:436
#2  0x7731a22d in nsFileSpec::Exists (this=0xa5acf8c) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/obsolete/nsFileSpecUnix.cpp:162
#3  0x77324efb in nsFileSpecImpl::Exists (this=0xa5acf80, _retval=0x7fd4bf2c) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/obsolete/nsFileSpecImpl.cpp:376
#4  0x08f69d2a in nsMsgLocalMailFolder::AddSubfolder (this=0xa56d000, name=@0x7fd4bf64, child=0x7fd4c040)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/local/src/nsLocalMailFolder.cpp:281
#5  0x08f6a790 in nsMsgLocalMailFolder::CreateSubFolders (this=0xa56d000, path=@0x7fd4c0dc)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/local/src/nsLocalMailFolder.cpp:258
#6  0x08f6ad18 in nsMsgLocalMailFolder::GetSubFolders (this=0xa56d000, result=0x7fd4c160)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/local/src/nsLocalMailFolder.cpp:430
#7  0x08e4f380 in nsMsgFolderDataSource::createFolderOpenNode (this=0xa3e58b0, folder=0xa56d028, target=0x7fd4c208)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgFolderDataSource.cpp:1460
#8  0x08e54f39 in nsMsgFolderDataSource::createFolderNode (this=0xa3e58b0, folder=0xa56d028, property=0x9839210, target=0x7fd4c208)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgFolderDataSource.cpp:1035
#9  0x08e5554c in nsMsgFolderDataSource::GetTarget (this=0xa3e58b0, source=0xa56d000, property=0x9839210, tv=1, target=0x7fd4c208)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgFolderDataSource.cpp:439
#10 0x08e63623 in GetTargetHasAssertion (dataSource=0xa3e58b0, folderResource=0xa56d000, property=0x9839210, tv=1, target=0x9839270, hasAssertion=0x7fd4c498)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgRDFUtils.cpp:100
#11 0x08e5138e in nsMsgFolderDataSource::DoFolderHasAssertion (this=0xa3e58b0, folder=0xa56d028, property=0x9839210, target=0x9839270, tv=1, hasAssertion=0x7fd4c498)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgFolderDataSource.cpp:2258
#12 0x08e516ae in nsMsgFolderDataSource::HasAssertion (this=0xa3e58b0, source=0xa56d000, property=0x9839210, target=0x9839270, tv=1, hasAssertion=0x7fd4c498)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/mailnews/base/src/nsMsgFolderDataSource.cpp:593
#13 0x08841e7c in nsXULTreeBuilder::IsContainerOpen (this=0xa3e1500, aResource=0xa56d000, aOpen=0x7fd4c498)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1794
#14 0x08842aab in nsXULTreeBuilder::IsContainerOpen (this=0xa3e1500, aResult=0xa5a9d98, aOpen=0x7fd4c498)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1783
#15 0x08843f8c in nsXULTreeBuilder::OpenSubtreeForQuerySet (this=0xa3e1500, aSubtree=0xa3e1594, aIndex=-1, aResult=0xa5a9bc8, aQuerySet=0xa563720, aDelta=0x7fd4c65c,
    open=@0x7fd4c62c) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1681
#16 0x088441d4 in nsXULTreeBuilder::OpenSubtreeOf (this=0xa3e1500, aSubtree=0xa3e1594, aIndex=-1, aResult=0xa5a9bc8, aDelta=0x7fd4c7b4)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1538
#17 0x0884449c in nsXULTreeBuilder::OpenContainer (this=0xa3e1500, aIndex=-1, aResult=0xa5a9bc8)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1511
#18 0x0884517b in nsXULTreeBuilder::RebuildAll (this=0xa3e1500) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTreeBuilder.cpp:1382
#19 0x0884c2b2 in nsXULTemplateBuilder::Rebuild (this=0xa3e1500)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTemplateBuilder.cpp:309
#20 0x08851409 in nsXULTemplateBuilder::AttributeChanged (this=0xa3e1500, aDocument=0x9b3b688, aContent=0xa2c6e60, aNameSpaceID=0, aAttribute=0x9842c08, aModType=2)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/templates/src/nsXULTemplateBuilder.cpp:890
#21 0x08758c5a in nsBindingManager::AttributeChanged (this=0x9b3b930, aDocument=0x9b3b688, aContent=0xa2c6e60, aNameSpaceID=0, aAttribute=0x9842c08, aModType=2)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xbl/src/nsBindingManager.cpp:1173
#22 0x08694620 in nsNodeUtils::AttributeChanged (aContent=0xa2c6e60, aNameSpaceID=0, aAttribute=0x9842c08, aModType=2)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/base/src/nsNodeUtils.cpp:98
#23 0x08680629 in nsGenericElement::SetAttrAndNotify (this=0xa2c6e60, aNamespaceID=0, aName=0x9842c08, aPrefix=0x0, aOldValue=@0x7fd4cb88, aParsedValue=@0x7fd4cc28,
    aModification=0, aFireMutation=0, aNotify=1) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/base/src/nsGenericElement.cpp:3253
#24 0x086809c7 in nsGenericElement::SetAttr (this=0xa2c6e60, aNamespaceID=0, aName=0x9842c08, aPrefix=0x0, aValue=@0xa5a9b88, aNotify=1)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/base/src/nsGenericElement.cpp:3177
#25 0x0868785c in nsGenericElement::SetAttr (this=0xa2c6e60, aNameSpaceID=0, aName=0x9842c08, aValue=@0xa5a9b88, aNotify=1)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/html/content/src/../../../base/src/nsGenericElement.h:392
#26 0x0868504f in nsGenericElement::SetAttribute (this=0xa2c6e60, aName=@0xa5a9b70, aValue=@0xa5a9b88)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/base/src/nsGenericElement.cpp:1386
#27 0x08836003 in nsXULElement::SetAttribute (this=0xa2c6e60, name=@0xa5a9b70, value=@0xa5a9b88)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/xul/content/src/nsXULElement.h:580
#28 0x77eca139 in NS_InvokeByIndex () at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/reflect/xptinfo/src/xptiInterfaceInfo.cpp:73
#29 0x081faf21 in XPCWrappedNative::CallMethod (ccx=@0x7fd4d030, mode=XPCWrappedNative::CALL_METHOD)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/xpconnect/src/xpcwrappednative.cpp:2162
#30 0x0820858e in XPC_WN_CallMethod (cx=0x9dfea40, obj=0xa3ca1f0, argc=2, argv=0xa567c08, vp=0x7fd4d174)
---Type <return> to continue, or q <return> to quit---
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/xpconnect/src/xpcwrappednativejsops.cpp:1455
#31 0x77d4ec5f in js_Invoke (cx=0x9dfea40, argc=2, flags=0) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/jsinterp.c:1396
#32 0x77d6284a in js_Interpret (cx=0x9dfea40, pc=0xa261983 ":", result=0x7fd4de70) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/jsinterp.c:3948
#33 0x77d4ece9 in js_Invoke (cx=0x9dfea40, argc=1, flags=2) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/jsinterp.c:1415
#34 0x77d4f07b in js_InternalInvoke (cx=0x9dfea40, obj=0x9e70a08, fval=170339752, flags=0, argc=1, argv=0xa567a38, rval=0x7fd4e08c)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/jsinterp.c:1490
#35 0x77d0ecdd in JS_CallFunctionValue (cx=0x9dfea40, obj=0x9e70a08, fval=170339752, argc=1, argv=0xa567a38, rval=0x7fd4e08c)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/js/src/jsapi.c:4357
#36 0x0878c2ec in nsJSContext::CallEventHandler (this=0x9e9d008, aTarget=0x9e4c144, aScope=0x9e70a08, aHandler=0xa272da8, aargv=0xa4e0b68, arv=0x7fd4e1f8)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/dom/src/base/nsJSEnvironment.cpp:1750
#37 0x087ec64c in nsJSEventListener::HandleEvent (this=0xa2a4678, aEvent=0x9d9ec68)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/dom/src/events/nsJSEventListener.cpp:212
#38 0x086c5c57 in nsEventListenerManager::HandleEventSubType (this=0x9e090f8, aListenerStruct=0xa2a46a0, aListener=0xa2a4678, aDOMEvent=0x9d9ec68, aCurrentTarget=0x9fe6ba0,
    aPhaseFlags=6) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/events/src/nsEventListenerManager.cpp:1266
#39 0x086c6193 in nsEventListenerManager::HandleEvent (this=0x9e090f8, aPresContext=0x9b3c598, aEvent=0x7fd4e5e4, aDOMEvent=0x7fd4e560, aCurrentTarget=0x9fe6ba0, aFlags=6,
    aEventStatus=0x7fd4e564) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/events/src/nsEventListenerManager.cpp:1382
#40 0x086e58f0 in nsEventTargetChainItem::HandleEvent (this=0x9c0b828, aVisitor=@0x7fd4e558, aFlags=6)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/events/src/nsEventDispatcher.cpp:355
#41 0x086e5cc5 in nsEventTargetChainItem::HandleEventTargetChain (this=0x9c0b840, aVisitor=@0x7fd4e558, aFlags=6, aCallback=0x0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/events/src/nsEventDispatcher.cpp:431
#42 0x086e63f0 in nsEventDispatcher::Dispatch (aTarget=0x9fe6ba0, aPresContext=0x9b3c598, aEvent=0x7fd4e5e4, aDOMEvent=0x0, aEventStatus=0x7fd4e63c, aCallback=0x0,
    aTargetIsChromeHandler=0) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/content/events/src/nsEventDispatcher.cpp:639
#43 0x08443983 in DocumentViewerImpl::LoadComplete (this=0x9b3bc88, aStatus=0) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/layout/base/nsDocumentViewer.cpp:1068
#44 0x08a86ba7 in nsDocShell::EndPageLoad (this=0x9dfe828, aProgress=0x9dfe83c, aChannel=0x9e090a0, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/docshell/base/nsDocShell.cpp:4740
#45 0x08a48b2d in nsWebShell::EndPageLoad (this=0x9dfe828, aProgress=0x9dfe83c, channel=0x9e090a0, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/docshell/base/nsWebShell.cpp:972
#46 0x08a8ee2e in nsDocShell::OnStateChange (this=0x9dfe828, aProgress=0x9dfe83c, aRequest=0x9e090a0, aStateFlags=131088, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/docshell/base/nsDocShell.cpp:4655
#47 0x08a59bde in nsDocLoader::FireOnStateChange (this=0x9dfe828, aProgress=0x9dfe83c, aRequest=0x9e090a0, aStateFlags=131088, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/uriloader/base/nsDocLoader.cpp:1235
#48 0x08a5a7ba in nsDocLoader::doStopDocumentLoad (this=0x9dfe828, request=0x9e090a0, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/uriloader/base/nsDocLoader.cpp:858
#49 0x08a5a996 in nsDocLoader::DocLoaderIsEmpty (this=0x9dfe828) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/uriloader/base/nsDocLoader.cpp:763
#50 0x08a5b057 in nsDocLoader::OnStopRequest (this=0x9dfe828, aRequest=0xa43ef78, aCtxt=0x0, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/uriloader/base/nsDocLoader.cpp:679
#51 0x080eaab7 in nsLoadGroup::RemoveRequest (this=0x9e89710, request=0xa43ef78, ctxt=0x0, aStatus=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/netwerk/base/src/nsLoadGroup.cpp:685
#52 0x083792f0 in imgRequestProxy::RemoveFromLoadGroup (this=0xa43ef78, releaseLoadGroup=1)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/modules/libpr0n/src/imgRequestProxy.cpp:162
#53 0x08379453 in imgRequestProxy::OnStopRequest (this=0xa43ef78, request=0xa43f260, ctxt=0x0, statusCode=0, lastPart=1)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/modules/libpr0n/src/imgRequestProxy.cpp:502
#54 0x08375e2b in imgRequest::OnStopRequest (this=0xa43f300, aRequest=0xa43f260, ctxt=0x0, status=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/modules/libpr0n/src/imgRequest.cpp:773
#55 0x08370e87 in ProxyListener::OnStopRequest (this=0xa43f658, aRequest=0xa43f260, ctxt=0x0, status=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/modules/libpr0n/src/imgLoader.cpp:866
#56 0x761279e7 in nsJARChannel::OnStopRequest (this=0xa43f260, req=0xa43f948, ctx=0x0, status=0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/modules/libjar/nsJARChannel.cpp:751
#57 0x080e0f9b in nsInputStreamPump::OnStateStop (this=0xa43f948) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/netwerk/base/src/nsInputStreamPump.cpp:571
#58 0x080e10ae in nsInputStreamPump::OnInputStreamReady (this=0xa43f948, stream=0xa43f9ac)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/netwerk/base/src/nsInputStreamPump.cpp:396
#59 0x77e81dd2 in nsInputStreamReadyEvent::Run (this=0xa3b8548) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/io/nsStreamUtils.cpp:111
#60 0x77eb33cc in nsThread::ProcessNextEvent (this=0x9741330, mayWait=1, result=0x7fd4efa4)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/threads/nsThread.cpp:482
#61 0x77e3f1b7 in NS_ProcessNextEvent_P (thread=0x9741330, mayWait=1) at nsThreadUtils.cpp:225
#62 0x77eb3850 in nsThread::Shutdown (this=0xa4d3fd8) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/threads/nsThread.cpp:442
---Type <return> to continue, or q <return> to quit---
#63 0x77eca139 in NS_InvokeByIndex () at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/reflect/xptinfo/src/xptiInterfaceInfo.cpp:73
#64 0x77ebcd07 in nsProxyObjectCallInfo::Run (this=0xa4deef8) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/proxy/src/nsProxyEvent.cpp:181
#65 0x77eb33cc in nsThread::ProcessNextEvent (this=0x9741330, mayWait=1, result=0x7fd4f0c0)
    at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpcom/threads/nsThread.cpp:482
#66 0x77e3f1b7 in NS_ProcessNextEvent_P (thread=0x9741330, mayWait=1) at nsThreadUtils.cpp:225
#67 0x083cbe2c in nsBaseAppShell::Run (this=0x983bc40) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/widget/src/xpwidgets/nsBaseAppShell.cpp:153
#68 0x08d59466 in nsAppStartup::Run (this=0x9842328) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpfe/components/startup/src/nsAppStartup.cpp:218
#69 0x0807a87c in main1 (argc=1, argv=0x7fd4f3b4, nativeApp=0x9746b50) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpfe/bootstrap/nsAppRunner.cpp:1245
#70 0x0807ad4b in main (argc=1, argv=0x7fd4f3b4) at /root/install/distro/unpackaged/cvs/mozilla/mozilla/xpfe/bootstrap/nsAppRunner.cpp:1747


Actually, it is a severe design bug. The method nsFileSpec::Exists() returns either "true" or "false", but it may not return the state "unknown" or "error happened during this operation". However, the underlying calls may return something different from both "yes, the file exists" and "no, the file does not exist", like ENOMEM, EACCES, EOVERFLOW. (In Java, we would throw and catch an Exception in these conditions.) Thus, the method nsFileSpec::Exists() erroneously replaces these "I do not know" conditions to "I know that the file does not exist". This, in turn leads seamonkey to try to create the file again, which actually results in overwriting (truncating) the file.

The correct way to fix this bug is to introduce at least another possible result  (like "unknown") additionally to "yes, the file exists" and "no, the file does not exist". However, this is an API change, implying that all callers of this method have to be changed. As all callers need a review anyway (checking what they do if stat(3) returns an error condition and thus Exists() errorneously returns false), changing all callers is also the correct (but possibly painful) solution to this severe bug.

thx for the investigation and analysis - WHY is there an error checking if the file exists? The stack trace is on folder discovery, presumably after a restart - my guess is something happened before you restarted that caused an error...
We know for sure that if EOVERFLOW happened, the file existed, just the stat data could not be filled nicely without an overflow. Thus, returning "Yes, the file exists" if stat(3) returns -1, but errno is EOVERFLOW, is the desired behaviour.

However, if errno is one of

EACCES
EIO
ELOOP
ENAMETOOLONG
ENOTDIR
EOVERFLOW
EFAULT
ENOMEM

the result is not "unknown" but remains to be "No, the file does not exist", which is clearly wrong in most cases (though, in some cases like EACCESS, it might not matter much). In cases like EIO, a partly unreadable file might (due to this bug) get fully deleted (truncated) unintentionally, depending on the filesystem.

Only if errno==ENOENT, the method should return "No, the file does not exist".
Attachment #246505 - Flags: review?
Hello David,

if the file exists, you cannot asume that stat() will return 0. Even if the file exists, stat() might return -1 and set errno to something like EACCES, EIO, ENAMETOOLONG, EOVERFLOW, EFAULT, ENOMEM. Conversely, this is not bijective in every case. So if you get ENOMEM, EIO or EACCES, you do not know wether the file existed or not.

Why seamonkey runs into folder discovery, I do not know. Presumably because the corresponding ".msf" file is empty. But this may happen, e.g. if seamonkey or the machine crashes during the generation of the ".msf" file. Whatever leads to this situation, no mozilla application shall ever overwrite or delete any user's file without the user's intention and confirmation.
Comment on attachment 246505 [details] [diff] [review]
This is a specific workaround to the specific problem which triggered the bug. However, some (rarer) cases are still unhandled

access(mPath, F_OK); might be an alternative that doesn't have the EOVERFLOW problem for large files (or stat64, of course).

I also want to note that this API (nsFileSpec) is deprecated and that the non-deprecated version (nsIFile/nsILocalFile) does have a way to report errors.
> Presumably because the corresponding ".msf" file is empty

No, we don't care about that - we'll just regenerate it.

So stat() is overflowing because it can't fit the file size in a signed 32 bit int - stat64 sounds like a good choice, and being more discriminating with the stat error codes...  
I tried the following:

Index: nsFileSpecUnix.cpp
===================================================================
RCS file: /cvsroot/mozilla/xpcom/obsolete/nsFileSpecUnix.cpp,v
retrieving revision 1.9
diff -u -w -p -r1.9 nsFileSpecUnix.cpp
--- nsFileSpecUnix.cpp  16 Mar 2005 20:07:34 -0000      1.9
+++ nsFileSpecUnix.cpp  27 Nov 2006 18:25:43 -0000
@@ -158,7 +158,7 @@ PRBool nsFileSpec::Exists() const
 //----------------------------------------------------------------------------------------
 {
     struct stat st;
-    return !mPath.IsEmpty() && 0 == stat(mPath, &st); 
+    return !mPath.IsEmpty() && 0 == access(mPath, F_OK); 
 } // nsFileSpec::Exists


and I still lost my folder copying messages into a 2GB+ folder. I'm not sure if the access call failed, or if some other code went asunder. I'll debug it further.
switching to core component
Component: MailNews: Main Mail Window → MailNews: Backend
Product: Mozilla Application Suite → Core
Version: unspecified → Trunk
requesting blocking...I'm still investigating this, however...
Flags: blocking1.8.0.9?
Hello David,

-    return !mPath.IsEmpty() && 0 == stat(mPath, &st); 
+    return !mPath.IsEmpty() && ((0 == stat(mPath, &st)) || (errno==EOVERFLOW)); 

works for me. Does it work for you, too?
Component: MailNews: Backend → Build Config
Flags: blocking1.8.0.9?
Product: Core → Mozilla Application Suite
Version: Trunk → 1.0 Branch
this really is a core bug - my changes got crunched, apparently. I'll try that specific change, though I suspect that only helps with special folders like Inbox, Sent, etc.
Product: Mozilla Application Suite → Core
Assignee: mail → bienvenu
Component: Build Config → MailNews: Backend
Version: 1.0 Branch → Trunk
 Xuân, can you try the "access" version that I put in a previous comment? Thx!
Status: NEW → ASSIGNED
Flags: blocking1.8.0.9?
(In reply to comment #16)
> this really is a core bug - my changes got crunched, apparently.

David,

I'm sorry. I did not intend to revert your changes and, at submission time, it was not apparent to me that I actually triggered this revert. This just happened because we together did a kind of parallel submission (we both started from the state of comment #12 and then submitted intependently) and bugzilla did not complain about the risk of overwriting new state by old state due to parallel submission. This is another bugzilla bug.

> I'll try that
> specific change, though I suspect that only helps with special folders like
> Inbox, Sent, etc.

I tested this on non-special (i.e. user defined) mail folders of >2GiB size, and it worked fine there.
(In reply to comment #17)
>  Xuân, can you try the "access" version that I put in a previous comment? Thx!
> 

David,

I did test your "access" version, and it works well. :-) Maybe you tested this using a wrong (old) binary?
I'm not sure - I haven't been able to reproduce the problem since, with the access() version of the patch (I never got around to trying your patch, since I prefer the access() version...) - so I think your idea is a big improvement, and I'll try to get it landed.

We still shouldn't be limited to 2GB - up to 4GB works on Windows, so I'm going to look at that next...
Attached patch biesi's proposalSplinter Review
Attachment #246709 - Flags: superreview?(mscott)
(In reply to comment #21)
> Created an attachment (id=246709)

Couldn't the line "struct stat st;" be removed with this patch, since the struct istn't used anymore?
Attachment #246709 - Flags: superreview?(mscott) → superreview+
Comment on attachment 246709 [details] [diff] [review]
biesi's proposal

requesting 1.8.0.9 approval - this affects mac and linux. I think it's fairly safe, and fixes a nasty bug.
Attachment #246709 - Flags: approval1.8.0.9?
fixed on trunk and 2.0 branch.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Keywords: fixed1.8.1.1
Resolution: --- → FIXED
the 2GB problem has to do with unix's that don't support 64 bit seeks, or indeed, any seeks > 2GBs. xpcom/obsolete/nsIFileStream.cpp has this problem, but so does netwerk/base/src/nsFileStreams.cpp.

This can be detected with an #ifdef _PR_NO_LARGE_FILES, so I could change the 4GB warning to a 2GB warning on those platforms. But I can't believe Mac and various linuxes don't have some way around this problem that NSPR could use
Component: MailNews: Backend → Build Config
Flags: blocking1.8.0.9?
Product: Core → Mozilla Application Suite
Version: Trunk → 1.0 Branch
argh, I've done it to myself somehow - resetting components again...
Product: Mozilla Application Suite → Core
Component: Build Config → MailNews: Backend
Flags: blocking1.8.0.9?
(In reply to comment #25)
> the 2GB problem has to do with unix's that don't support 64 bit seeks, or
> indeed, any seeks > 2GBs. xpcom/obsolete/nsIFileStream.cpp has this problem,
> but so does netwerk/base/src/nsFileStreams.cpp.
> 
> This can be detected with an #ifdef _PR_NO_LARGE_FILES, so I could change the
> 4GB warning to a 2GB warning on those platforms. But I can't believe Mac and
> various linuxes don't have some way around this problem that NSPR could use
> 

Maybe you can do relative seeks (using SEEK_CUR) in multiple steps, each of these steps in the range of -2GiB..+2GiB ?

Or if we do absolute position, we first seek to the start (using SEEK_END) and then repeatedly advance (using SEEK_CUR)?
yeah, I was thinking of relative seeks as well, if nothing else will work...
Comment on attachment 246709 [details] [diff] [review]
biesi's proposal

NSPR also uses access(..., F_OK) to implement
PR_Access(..., PR_Access PR_ACCESS_EXISTS), which checks
the existence of a file.

I don't understand the comments about seeking.  There is
no seeking in this patch.
Attachment #246709 - Flags: review+
wtc - the reference to seeking has to do with a related issue, having to do with mailboxes > 2GB. Basically, on _PR_NO_LARGE_FILES platforms, we can't support mailboxes larger than 2GB because we can't do the needed seeking. It's surprising to me that some unix platforms, including OS/X 10.4, we don't support seeks of > 2GB. As Xuân points out, we could perhaps fake this by breaking up the seeks into offsets < 2GB. We could either do this in various parts of the client code (I know of at least two pieces of client code that would need to change so far) or do it in NSPR. I'm guessing that doing it in NSPR is unlikely, but I thought I'd at least mention it.
Version: 1.0 Branch → Trunk
Comment on attachment 246709 [details] [diff] [review]
biesi's proposal

approved for 1.8.0 branch, a=dveditz for drivers
Attachment #246709 - Flags: approval1.8.0.9? → approval1.8.0.9+
Flags: blocking1.8.0.9? → blocking1.8.0.9+
Whiteboard: needs landing
fixed on the 1.8.0.x branch.
Keywords: fixed1.8.0.9
We should fix NSPR so that it supports large files on all
major platforms.  For Mac OS X, this may just require defining
_PR_HAVE_LARGE_OFF_T instead of _PR_NO_LARGE_FILES in
mozilla/nsprpub/pr/include/md/_darwin.h.  (off_t is 8 bytes
on Mac OS X.)  Then you can use PR_GetFileInfo64, PR_GetOpenFileInfo64,
and PR_Seek64.
oh, even better - we'd love that for TB 2.0, if possible.
Please still look into changing the client code just in case I don't
have time to work on NSPR before TB 2.0 is released.

Here are the supported Unix platforms on which _PR_NO_LARGE_FILES is defined:

FreeBSD
Mac OS X
NetBSD
OpenBSD
OpenVMS
QNX / NTO (Neutrino)
RISC OS

Which of these matter to you?

I don't know the status of OS/2 and BeOS because the NSPR code for these
two platforms doesn't use the _PR_NO_LARGE_FILES macro.  Does anyone know
if NSPR supports large files on OS/2 and BeOS?
Mac OS/X matters the most - I don't really know about the rest - I'd say the BSD's are probably next in importance, but that's just a guess.
Keywords: dataloss
(In reply to comment #35)
> Please still look into changing the client code just in case I don't
> have time to work on NSPR before TB 2.0 is released.
> 
> Here are the supported Unix platforms on which _PR_NO_LARGE_FILES is defined:
> 
> FreeBSD
> Mac OS X
> NetBSD
> OpenBSD
> OpenVMS
> QNX / NTO (Neutrino)
> RISC OS
> 
> Which of these matter to you?
> 
> I don't know the status of OS/2 and BeOS because the NSPR code for these
> two platforms doesn't use the _PR_NO_LARGE_FILES macro.  Does anyone know
> if NSPR supports large files on OS/2 and BeOS?
> 
Are we still talking about this reported bug? Because this bug happened under Linux, which has been supporting stat64() and other 64-bit file operations for years. So, as _PR_NO_LARGE_FILES is not defined for Linux, why could this bug then happen? Or is _PR_NO_LARGE_FILES also defined for Linux?
I opened NSPR bug 362244 for the NSPR's lack of large file support
on some platforms.  Let's stay focused on the original problem in
this bug report.
for QA/testing, this change affects most of the code that checks to see if a file exists in mail/news. But, if it didn't work, mail pretty much wouldn't work. To recreate the bug, you need to create a very large folder, > 2GB, and see if it survives restart on a platform like OS/X.
Unfortunately I don't have a folder that large to test - Reporter, can you help verify the bug using the candidate build: ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2.0.0.0-candidates.
Attachment #246505 - Attachment is obsolete: true
Attachment #246505 - Flags: review?
Product: Core → MailNews Core
This bug still persists in seamonkey 1.1.12. :-( My sister just lost 6 years worth of e-mail due to this bug. :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
David, is there a unit test for this?  Xuan, are you perhaps not seeing the exact symptoms of this bug, but rather bug 362244?

(resetting QA contact, was empty)
QA Contact: backend
(In reply to comment #42)
> Xuan, are you perhaps not seeing the
> exact symptoms of this bug, but rather bug 362244?
I do see completely lost Inbox folders, repeatedly. :-( The data loss one happened to a friend about 1 month ago.
no activity for a long time, please close
Closed per Comment 44
Status: REOPENED → RESOLVED
Closed: 13 years ago8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.