206473 - messages corrupted after moving between folders

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Stefano Caselli]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030507
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030507

i've successfully imported from outlook express more than 900 messages.
then using drag and drop i moved all the messages from the "inbox" folder
created by the import tool ("outlook express mail") to another folder (i've used
the "inbox folder" for the mail account).
the process terminated correctly. then i tried reading some random messages, to
see if all was ok... and it wasn't. many messages were corrupted: some parts of
the messages were missing or the text was intermixed with parts of other
messages. it seems that the "move messages" feature is broken. maybe this
happens only  when moving  many messages at the same time.


Reproducible: Always

Steps to Reproduce:
1. import messages from outlook express (tried only using more that 900 messages)
2. select all imported messages using ctrl + a.
3. drag the messages to the inbox folder
4. check the messages: many are corrupted

Comment 1

[David :Bienvenu]

do you have check for new mail turned on, and automatically download new mail?
In general, the move message feature works fine, afaik. I don't know about the
outlook express import feature, though. Do the messages look fine in the outlook
express folder?

Comment 2

[Stefano Caselli]

>do you have check for new mail turned on, and automatically download new mail?
I moved the messages when i was offline, so i don't think that's a problem.

>I don't know about the outlook express import feature, though.
the messages immediately after the import were ok

>Do the messages look fine in the outlook express folder?
yes

I can try removing all personal messages and send you the remaining ones (most
of the email are from mailing lists and similar, so I don't have problems
sharing them). I don't know if the problem will be present after moving the
files, but maybe you'll be able to reproduce it. Feel free to contact me
directly by email, if you want.
Thanks.

Comment 3

[David :Bienvenu]

Attachment: proposed fix

this patch makes copying local messages tolerant of embedded nulls in messages.

Comment 4

[(not reading, please use [e-mail] instead)]

Comment on attachment 124464 [details] [diff] [review]
proposed fix

r/sr=sspitzer

david, is something we want for 1.4 final?

maybe after it gets tested and verified on the trunk, I could land on the 1.4
branch.

Comment 5

[David :Bienvenu]

you might - it's definitely data loss. Also, remember that compact uses this
code, so compacting a folder like this will lose data as well.

Comment 6

[(not reading, please use [e-mail] instead)]

I've logged bug #207522 on the import code issue.

I'd like to take this on the 1.4 branch, once verified on the trunk.

I'll go query for dups, david thinks we might have a few.

Comment 7

[David :Bienvenu]

Attachment: sample outlook express inbox that will demonstrate the problem

Comment 8

[David :Bienvenu]

Attachment: correct fix

previous fix was wrong (but worked when I tested it) since it was using end
uninitialized!

Comment 9

[(not reading, please use [e-mail] instead)]

Comment on attachment 124507 [details] [diff] [review]
correct fix

r/sr=sspitzer for the trunk, once verified, we'd want on the branch.

Comment 10

[David :Bienvenu]

fix checked in.

Comment 11

[David :Bienvenu]

Attachment: fix w/o regression :-(

for the branch, this is the fix that should be considered. The previous patch
introduced a regression that this cumulative patch fixes.

Comment 12

[esther]

David, when did you check this in?  I took the 6-2 trunk and I still see a
problem with the 1st message in the attached Inbox.  

I saved the dbx file to my desktop on winxp, created a new profile, added a pop
account then imported the dbx file (note: this OS profile doesn't have outlook
express installed so I'm asked for the location of the dbx file).  Both messages
look OK within in the Outlook imported folder.

I then dragged both messages from the Outlook imported folder to a new folder on
my pop account.  The 1st message is displayed with raw text in the beige area of
the message.  The 2nd message displays OK.
If I drag each message individually, the 1st one displays no text in the beige
area. The 2nd message displays OK.

Comment 13

[David :Bienvenu]

the 6.2 trunk build has this fix. But I don't know that pretending it's an
Outlook file is a valid test. I tried again with a trunk build and the messages
moved OK.

Comment 14

[esther]

So how did you use the sample dbx file?  

Comment 15

[David :Bienvenu]

I have outlook express installed - maybe when you said Outlook, you meant
Outlook Express?

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

Where do we stand on this one?  Do we need re-reviews?  Is the no-regression
patch in the trunk?  Do we have verification so we can request for 1.4 approval?

Comment 17

[David :Bienvenu]

yes, the cumulative fix is on the trunk. We apparently don't have verification
though it does fix the problem for me. Seth and I have agreed that we're not
going to push this for 1.4.

Comment 18

[esther]

Yes, I meant Outlook Express throughout my comment 12.  I decided to test this
using my Outlook Express profile located in a different Winxp profile.  That
profile had qatest04 account set up in Outlook Express.  I launched the 6-2
trunk and imported my qatest04 messages from Outlook Express and then moved all
the message into a local folder.  I opened all of the messages that had been
moved and they all look fine.  If there was something special about those those
2 messages, in particular the app originally used to send them, let me know and
I'll try to recreate them.

Comment 19

[David :Bienvenu]

yes, those messages had embedded nulls in them, which caused the problem when
moving them (though not so much when importing them). So in that other winxp
profile, you can just drop the attached Inbox.dbx file on top of the existing
inbox.dbx and try these steps again.

Comment 20

[esther]

Well I'm not sure my Outlook Express is quite right because I searched for
inbox.dbx files throughout my system and only found the one I saved to my hard
drive from the bug.  That's why I didn't do this first.  In any case one of the
messages in my imported qatest04 account exhibited the problem in a build before
the fix and did not have the problem in the 6-2 using the menu to move or 6-3
trunk build using drag n drop so I can verify this based on my one message.  

Comment 21

[esther]

reopening, need to verify for trunk only and leave open for branch verification

Comment 22

[David :Bienvenu]

Ok, thx, Esther! I don't know much about Outlook Express, but I did set it up a
while back to see how it did some things.

Comment 23

[esther]

putting back to resolved fixed.  This is verified on trunk only.

Comment 24

[(not reading, please use [e-mail] instead)]

david and I discussed this, and we don't want this for 1.4 final.

the risk is higher than the reward, and 1.5 alpha is good enough for us.