180227 - large mail downloads corrupt mail database

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Brant Gurganus]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2b) Gecko/20021108
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2b) Gecko/20021108

Whenever I receive large amounts of messages via POP, the mail database seems to
become corrupt for messages I had before I downloaded the new mail.

Reproducible: Sometimes

Steps to Reproduce:
1. Have some messages in your inbox.
2. Receive a large amount of messages (about 150) at the same time.
3. Start reading some of your new messages while they download.
4. Look back at your old messages.

Actual Results:  
The old messages seem to have merged their headers or something.  For example,
one of my messages looks like the lower half on one and the upper half of another:

Change the sentence to "The content associated with an element in the
source document.  Some elements have no content, in which case they are
called empty."



[e-mail]>
Received: from openoffice.org (unverified [64.125.133.202]) by
UV-SERVER02.ultravision.net
 (Vircom SMTPRS 5.1.202) with SMTP id <[e-mail]>
for <[e-mail]>;
 Wed, 13 Nov 2002 20:44:55 -0600
Received: (qmail 27477 invoked by uid 5302); 14 Nov 2002 02:48:20 -0000
Mailing-List: contact [e-mail]; run by ezmlm
Precedence: bulk
X-No-Archive: yes
list-help: <mailto:[e-mail]>
list-unsubscribe: <mailto:[e-mail]>
list-post: <mailto:[e-mail]>
Reply-To: [e-mail]
Delivered-To: mailing list [e-mail]
Received: (qmail 27454 invoked from network); 14 Nov 2002 02:48:15 -0000
Message-ID: <[e-mail]>
Date: Wed, 13 Nov 2002 21:44:18 -0500
From: Brant Langer Gurganus <[e-mail]>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.2b) Gecko/20021108
X-Accept-Language: en-us, en
MIME-Version: 1.0
To:  [e-mail]
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
micalg=sha1; boundary="------------ms000404080205030504050708"
Subject: [qa-dev] Usage of oooqa keyword

--------------ms000404080205030504050708
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

I am a little confused on the usage of the oooqa keyword.  Do I set it 
on any bug that I have triaged?  That is what it says at 
<http://qa.openoffice.org/helping.html>.  Do I set it when I confirm a 
bug?  That is what it says (or seems to say) at 
<http://qa.openoffice.org/issues/describekeywords.cgi>.  If that second 
description is unclear or incorrect, I will include that in the bug I 
will soon submit about the bad, inconsistent grammar used in the keyword 
descriptions.
-- Brant Langer Gurganus http://troop545.cjb.net/brant.htm
--------------ms000404080205030504050708 Content-Type:
application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding:
base64 Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKejCC
AzgwggKhoAMCAQICEGZFcrfMdPXPY3ZFhNAukQEwDQYJKoZIhvcNAQEEBQAwgdExCzAJBgNV
BAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG
A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2Vydmlj
ZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkG
CSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMDA4MzAwMDAw
MDBaFw0wNDA4MjcyMzU5NTlaMIGSMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBD
YXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xDzANBgNVBAoTBlRoYXd0ZTEdMBsGA1UECxMUQ2Vy
dGlmaWNhdGUgU2VydmljZXMxKDAmBgNVBAMTH1BlcnNvbmFsIEZyZWVtYWlsIFJTQSAyMDAw
LjguMzAwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAN4zMqZjxwklRT7SbngnZ4HF2ogZ
gpcO40QpimM1Km1wPPrcrvfudG8wvDOQf/k0caCjbZjxw0+iZdsN+kvx1t1hpfmFzVWaNRqd
knWoJ67Ycvm6AvbXsJHeHOmr4BgDqHxDQlBRh4M88Dm0m1SKE4f/s5udSWYALQmJ7JRr6aFp
AgMBAAGjTjBMMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwxLTI5NzAS
BgNVHRMBAf8ECDAGAQH/AgEAMAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQQFAAOBgQAxsUtH
XfkBceX1U2xdedY9mMAmE2KBIqcS+CKV6BtJtyd7BDm6/ObyJOuR+r3sDSo491BVqGz3Da1M
G7wD9LXrokefbKIMWI0xQgkRbLAaadErErJAXWr5edDqLiXdiuT82w0fnQLzWtvKPPZE6iZp
h39Ins6ln+eE2MliYq0FxjCCA5swggMEoAMCAQICAwf5AzANBgkqhkiG9w0BAQQFADCBkjEL
MAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3du
MQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZpY2VzMSgwJgYD
VQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwMB4XDTAyMDcyNjAyMjIwNFoX
DTAzMDcyNjAyMjIwNFowgYAxHzAdBgNVBAMTFlRoYXd0ZSBGcmVlbWFpbCBNZW1iZXIxNTAz
BgkqhkiG9w0BCQEWJmJyYW50Z3VyZ2FudXMyMDAxQGNoZXJva2Vlc2NvdXRpbmcub3JnMSYw
JAYJKoZIhvcNAQkBFhdicmFudGd1cmdhQHByb3RvbmljLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAM7+6uXDFIh/27PIkQegDGgeaFsKG4HE4lVFoiUKp9XVHFLslTjr
A/bKgL3OEv98hshwkklnYvIhX1J3ihoTic5NipFabVZB/1Zr1vsdsKl4eBZ1x88xh4zLa3nI
A590mUcrWFMsnghDMPTZnqU9cMGgpxcwxA8q/HF6FHBOqorIKPYkzuMYXTUwk+Ac/ilvxGL4
rsaV6h139p5mul2igEFtA0Cz2woFkKT4SrMpKsh9uz4quXxVu4W8HXc6psPWrSEmxv88iQtV
9OqeQvLhg7lGzOVPmuNaexbKCgAzut7yPyGDkVbW1Ku0j5rI7naABc3HTPN+Yj3HJyYAWf+n
OdsCAwEAAaOBijCBhzArBgUrZQEEAQQiMCACAQAwGzAZAgEEBBRFS3Z3VHBZZTY0VXhSTkFh
N3ZzWTBKBgNVHREEQzBBgSZicmFudGd1cmdhbnVzMjAwMUBjaGVyb2tlZXNjb3V0aW5nLm9y
Z4EXYnJhbnRndXJnYUBwcm90b25pYy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQQF
AAOBgQBnm+3oSX60aEhwXY64GaADLnkaXEU68wogEJ3udNIywY3Wy8a+S0Xu2GzTONYvPAr/
RTX0ZAtzgGbI02qGwdHNyvLFNwiQRVP9L7XKDCSCPXPftXyH/FDsYtz7iiyoY9iNx5vGKTH9
1JlCtYx3wtSNcjFWcvhYBaBW31nhnkSVcTCCA5swggMEoAMCAQICAwf5AzANBgkqhkiG9w0B
AQQFADCBkjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ
Q2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZp
Y2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwMB4XDTAyMDcy
NjAyMjIwNFoXDTAzMDcyNjAyMjIwNFowgYAxHzAdBgNVBAMTFlRoYXd0ZSBGcmVlbWFpbCBN
ZW1iZXIxNTAzBgkqhkiG9w0BCQEWJmJyYW50Z3VyZ2FudXMyMDAxQGNoZXJva2Vlc2NvdXRp
bmcub3JnMSYwJAYJKoZIhvcNAQkBFhdicmFudGd1cmdhQHByb3RvbmljLmNvbTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAM7+6uXDFIh/27PIkQegDGgeaFsKG4HE4lVFoiUK
p9XVHFLslTjrA/bKgL3OEv98hshwkklnYvIhX1J3ihoTic5NipFabVZB/1Zr1vsdsKl4eBZ1
x88xh4zLa3nIA590mUcrWFMsnghDMPTZnqU9cMGgpxcwxA8q/HF6FHBOqorIKPYkzuMYXTUw
k+Ac/ilvxGL4rsaV6h139p5mul2igEFtA0Cz2woFkKT4SrMpKsh9uz4quXxVu4W8HXc6psPW
rSEmxv88iQtV9OqeQvLhg7lGzOVPmuNaexbKCgAzut7yPyGDkVbW1Ku0j5rI7naABc3HTPN+
Yj3HJyYAWf+nOdsCAwEAAaOBijCBhzArBgUrZQEEAQQiMCACAQAwGzAZAgEEBBRFS3Z3VHBZ
ZTY0VXhSTkFhN3ZzWTBKBgNVHREEQzBBgSZicmFudGd1cmdhbnVzMjAwMUBjaGVyb2tlZXNj
b3V0aW5nLm9yZ4EXYnJhbnRndXJnYUBwcm90b25pYy5jb20wDAYDVR0TAQH/BAIwADANBgkq
hkiG9w0BAQQFAAOBgQBnm+3oSX60aEhwXY64GaADLnkaXEU68wogEJ3udNIywY3Wy8a+S0Xu
2GzTONYvPAr/RTX0ZAtzgGbI02qGwdHNyvLFNwiQRVP9L7XKDCSCPXPftXyH/FDsYtz7iiyo
Y9iNx5vGKTH91JlCtYx3wtSNcjFWcvhYBaBW31nhnkSVcTGCA9UwggPRAgEBMIGaMIGSMQsw
CQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24x
DzANBgNVBAoTBlRoYXd0ZTEdMBsGA1UECxMUQ2VydGlmaWNhdGUgU2VydmljZXMxKDAmBgNV
BAMTH1BlcnNvbmFsIEZyZWVtYWlsIFJTQSAyMDAwLjguMzACAwf5AzAJBgUrDgMCGgUAoIIC
DzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wMjExMTQwMjQ0
MThaMCMGCSqGSIb3DQEJBDEWBBSR+bDXajJDK217QPG+YDPEErJbRjBSBgkqhkiG9w0BCQ8x
RTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMC
BzANBggqhkiG9w0DAgIBKDCBqwYJKwYBBAGCNxAEMYGdMIGaMIGSMQswCQYDVQQGEwJaQTEV
MBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xDzANBgNVBAoTBlRo
YXd0ZTEdMBsGA1UECxMUQ2VydGlmaWNhdGUgU2VydmljZXMxKDAmBgNVBAMTH1BlcnNvbmFs
IEZyZWVtYWlsIFJTQSAyMDAwLjguMzACAwf5AzCBrQYLKoZIhvcNAQkQAgsxgZ2ggZowgZIx
CzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93
bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZpY2F0ZSBTZXJ2aWNlczEoMCYG
A1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4zMAIDB/kDMA0GCSqGSIb3DQEB
AQUABIIBAMcW4zmqZnENX4b/p65vWV25z8toL02gqXBIBnSJ0EWCkcYDZublLfatV1xyU7OW
n4E0MaabU7JqIoThYUb39He7PN1SnhxqAvv5tvbX5xcAQw9oWNH9pmRnY2nEMMzhx3jL/NL8
Ahm3rBs+Fwj8kOUEGoKLYJlFWybu/5/53EP4J6rLKzcOcQduTT7wm07nH39U/DtLuuK8sRND
69nMb6dnPEr10ClQ4MSjNPwZZR4BEEAwQ+vqm18McyHeIWdVLvx9eZ+eMU64Cug//spU6Tio
JyYxIMxJFPgUySYicgqcYqM3AbjfWqXjEDSj5L7itcErN9v1IUff0BTW+4x7nGsAAAAAAAA=
--------------ms000404080205030504050708-- 

Expected Results:  
This corruption should not occur.

I searched for duplicates using "corrupt mail".  Candidates from those include:
Bug 178842.  However, I don't think it is a duplicate since I don't use any
programs that would access the mail files except Mozilla itself.

When I restart Mozilla, I will see what happens when I delete the *.msf file
which seems to clear up similar problems sometimes.

Comment 1

[Brant Gurganus]

I deleted the *.msf files and the problem seemed to go away.  In fact, some
e-mails appeared that I hadn't seen before.  It looks as if there is something
interfering with teh *.msf writing/updating process.

Comment 2

[David :Bienvenu]

Navin worked on a fix for this. It sounds like it might not be fixed in all
situations because the fix went in before 1.2 final. It would be great to know
if you can still reproduce this with 1.2 final or 1.3

Comment 3

[Brant Gurganus]

I don't have specific reproduction steps, but this does still happen for me?  I
think my Internet connection sometimes becomes disconnected at the time the
corruption occurs.  I am not sure though.

Comment 4

[Brant Gurganus]

Okay, I think I can consistently reproduce this by downloading mail and then
disconnecting the phone line since it happened again the last time the phone
line disconnected due to Call Waiting interferring with the connection.  I don't
disable call waiting when I am on the computer since it will usually cause the
connection to disconnect and the phone will then ring which is what I want.

Comment 5

[Brant Gurganus]

Adding keyword since I think I can consistently reproduce this now.

Comment 6

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 7

[Ken Schneider]

Simular problem here, occasionally cxn drops halfway through mail download on
primary pop account.  Only I can't open the INBOX after this happens.  I can
open subfolders okay though.  Rebooting and updating from stable to 200030908
didn't help.  I'm not sure that it is dependent upon the size of the messages.

Temp fix, create another account with the same username/password to d/l your
messages.  Fist you must rename the "account name" in the old one as you can't
duplicate account names.

Comment 8

[David :Bienvenu]

taking

Comment 9

[Pádraig Brady]

Me too. Mozilla 1.4, redhat 9.

I was downloading 283 of mails from a POP to my inbox
which contained about 50 read messages. I was deleting
(loads of 150K M$ viruses) as they were being downloaded.
Then the download seemed to stop after about 150, so I just 
restarted mozilla and downloaded the rest, and deleted all
but one. However all of the existing mails were gone from the mbox,
though the msf seemed fine, so I could see the messages but
not their contents. Looking at the mbox directly showed
a couple of the M$ virus mails left (even though I deleted
them, and they weren't in the msf) and the one message I
didn't delete (which I can view fine).

Comment 10

[odi]

I have been seeing this on W2k from Mozilla 1.4 onwards. The inbox.msf file of a
POP3 GMX SSL account with "leave messages on server" option gets corrupted
frequently when I delete any message that is not the last on the list (sorted by
descending date). That never happens with IMAP accounts.

I am attaching a zip file with Inbox and Inbox.msf after I pressed delete on the
first message (not in inbox any more).

Comment 11

[odi]

Attachment: Corrupted Inbox.msf

This is the Inbox.msf just after I deleted the first message and closed
Mozilla.

Comment 12

[David :Bienvenu]

the INBOX.msf file w/o the corresponding INBOX file isn't that useful (it looks
fine as it is) - can you e-mail me the zip of the INBOX and the corresponding
INBOX.msf file and I can maybe see how it's out of sync. thx.

Comment 13

[odi]

Comment on attachment 133735 [details]
Corrupted Inbox.msf

useless without the Inbox file

Comment 14

[odi]

Hint:
I have a message filter setup to mark certain incoming mail as read and delete
certain incoming mail at once. When my Inbox broke this morning I noticed one
Email with status 'read'. After I deleted the msf file that mail was gone. So I
guess the mail filter may be the one messing up the msf file. I have sent my
Inbox to David Bienvenue.

Comment 15

[David :Bienvenu]

Ortwin, can you send me your filter file so I can have a better idea what's
going on there? I looked through your INBOX and INBOX.msf, and the INBOX.msf
definitely has the wrong offsets for messages, and the wrong messages sizes -
I'm trying to figure out how it could have gotten that way. Also, could you try
running a later build, like 1.6, or 1.7b, which should be coming out soon. Thx.

Comment 16

[odi]

Filter file sent to David. I have been running 1.6 since it is out (Jan 15th
2004) and I have not encountered the problem recently. So maybe this has already
been fixed. Feel free to mark as WFM if you do not hear anything in the next few
months.

Comment 17

[David :Bienvenu]

I'm going to mark this wfm for now - running for 2 months w/o a problem is
pretty good and I might forget in a few months. Please re-open if it happens to
you again. Thx for you help in diagnosing this.