138804 - importing messages from outlook express partially works, partially fails

Status: VERIFIED FIXED

(SeaMonkey :: MailNews: Message Display, defect)

Description

[miroac]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0rc1)
Gecko/20020417
BuildID:    2002041711

I tried importing all my email messages (see additional information) from OE 5.
Eventually, after 30 to 40 partial imports, I got all my email imported. It took
about 2-3 hours.

Mozilla would import 3 or 4 folders at a time, and that's it. So, I would do an
import, go over to OE and delete the folders it imported correctly, close OE,
and do another import, and repeat this process several times.

It seemed totally random as to which folders it found and imported. For example,
I had a folder named "a" with sub folders "b", "c", and "d". It would import "a"
and "b", and not even find "c" or "d". If I went into OE and deleted folder "b",
then did another import, it would import "c", and maybe "d", maybe not.
Sometimes, it would import all the messages from the parent "a", and sometimes
it wouldn't. At least half of the time, it wouldn't import messages from
subfolders like "b" and "c" if there were messages in the parent "a".

I had to move folders around in OE a lot of times before mozilla could find them
to import them. I think mozilla can't find folders that are more than 2 or 3
subfolders deep. For example, I have a folder "a", inside it is a folder "b",
inside "b" is a folder "c", inside "c" is a folder "d". No matter how many times
I tried, it couldn't find "c" or "d". If I moved "d" to the root level, it
seemed to find it.

Mozilla also appears not to like importing subfolders with '/' characters in
them. No matter how many times I tried, it would never import a folder with a
'/' character in it. I had to rename a few folders in OE before mozilla would
import them. It didn't seem to have a problem with spaces or commas or dashes or
anything other than '/' characters.

Reproducible: Always

Expected Results:  import all messages the first time

I've used Outlook Express a long time, and keep emails around for a long time. I
have several hundred or a few thousand emails I was importing.

I'm not sure if "Mail Window Front End" is the correct component for this, but
it seems to be the only one I found where other bugs mention importing messages.

Comment 1

[(not reading, please use [e-mail] instead)]

over to cavin, who has worked on OE import recently.

Comment 2

[Cavin Song]

Ok, I'll see if I can reproduce the problems here first. Seems like there are
two problems here: (1) nested folders up to 3 levels can't be imported, (2)
subfolders whose names contain '/' can't be imported.

Comment 3

[Cavin Song]

I was able to import folders up to 5 nesting levels without problems. My guess 
is that maybe there are other cases which did not get fixed by bug 123804. So 
I'm wondering if reporter or march can email me your zipped Outlook Express 
directory (where it contains a lot of .dbx files) so that I can debug it here?  
I understand that there may be personal info in the mail so it's only voluntary. 
The files will help me fix the problem much sooner if I can see the problem. 
Thanks.

I did see problem #2 though and I think there might be already a bug against it 
so let's address it separately.

Comment 4

[Michael March]

Attachment: Here is a screen shot of what gets imported into Mozilla.

Comment 5

[Michael March]

Attachment: Here is just a partial screen shot of what folders are in my OE setup.

I am still trying to see how I can send you my actual mail spools but until
then here is something that might help you out.

Comment 6

[miroac]

Unfortunately, I no longer have the original OE format files...
    "So, I would do an import, go over to OE and delete the
    folders it imported correctly, close OE, and do another
    import, and repeat this process several times."
In the process of importing, I was deleting what imported correctly... I can't
go back now. When I did the import, I wasn't thinking of how to make it easy to
reproduce or fix.
The main thing I remember was it seemed to be so random. I would do an import,
and it would find, maybe, one folder with only a few messages in it. Then I
would delete that folder from OE, and do another import, and this time it would
find and import 10 folders that it didn't import the previous time.
From looking at the 2 attachments by Michael March, I can say I didn't have
nearly that many messages, probably around 1000, but scattered across about as
many folders as Michael's attachments show.
I'll reiterate that eventually all my email did import, it just took so many
partial attempts to get it there.

Comment 7

[Michael March]

I am importing around 200,000 messages or about 1.6 gigs of email if that helps
at all.

Comment 8

[vladimir liba]

I have the same problem, same platform and OS. About 30 folders, about 300
messages . I was thinking problem is with levels, so i moved some folders to the
root (Local Folders), but it didn't help, some folders were not imported. I
tried to delete some folders, which were imported fine, and then some new
folders were imported, but again, not all. Now i don't know, if to try again and
again, or wait if the problem will be solved soon:)

Comment 9

[Cavin Song]

OK, I see what's going on here. The fix for bug 123804 was only checked into the
trunk build and is not in the current rv:1.0rc1 build. If you can get a trunk
build and try it the problem should be fixed. If not, please add comments to
this bug again. I tried the multi-level folders import on the trunk build when I
said I could not see the probmle. The version number for the trunk build should
read something like rv:0.9.9+. I just missed out the version number from the
beginning. Thank you all for the effort to help resolve the problem.

Comment 10

[Corrado Berti]

Tested with trunk build:
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.0.0+) Gecko/20020428

Much better than 1.0 Branch, but still not perfect;
missed folders: 2 of 77 
For the imported folders, all msgs are imported. 
if needed, i can send u the OE archive, just tell me where
(it's about 100 zipped mb)

i'll attach the result.

Comment 11

[Corrado Berti]

Attachment: OE and Moz folder pane after import

Comment 12

[Cavin Song]

Corrado, can you send your OE stuff to [e-mail] so that I can take a 
look at it. Thanks.

Comment 13

[Michael March]

Attachment: Reduced Outlook Express mail spool...

I d/l'd and installed the current nightly build and it did *much* better.
Instead of only importing 15  out approximately 166 folders it now imports a
little over 100. So it seems we are getting there.

Attached is a reduced version of my mail OE mail spool that is causing the
issues. I will email the passwd (and whoever else needs it) so you can
replicate the issue on your end.

Comment 14

[Cavin Song]

Thanks to Michael and Corrado for sending me their OE files to debug. I'm 
working on the problem right now and already have a good handle on it.

Comment 15

[Cavin Song]

OK. the problem is now fixed and it works for Michael's and Corrado's OE files.
 
One problem I have is that somehow folders like "Search Folder", "Search Folder 
(2)", "Search Folder (3), etc are created as well even if they don't exist in 
OE (ie, you can't see them from the UI). I tried to create folder "Search 
Folder" in OE and OE said that this folder already exists, so I guess this is a 
system folder that OE uses to store searched messages. I may have to filter out 
these folders. Not quite sure yet.

Comment 16

[Cavin Song]

I'm going to leave "Search Folder" issue alone since I'm not sure if I should 
filter it out.

Comment 17

[Cavin Song]

Nominating for nsbeta1 since folders are missing in some cases.

Comment 18

[Cavin Song]

The problem is that we did not handle the case where the root (ie, top level) 
folder shows up after all the child folders. A patch is coming.

Comment 19

[Cavin Song]

Attachment: Proposed patch, v1

Put child folders in the pending list when root folder is sitll missing and
process these child folders when root folder shows up.

Comment 20

[Jean-Francois Ducarroz]

Comment on attachment 81604 [details] [diff] [review]
Proposed patch, v1

only one comment:
+  for (int i = m_pendingChildArray.Count()-1; i >= 0; i--)

please don't declare i inside the for statment as the scope of if might differ
from one compiler to another one. It's much better to declare it outside the
loop do avoid any confusion. Also it should be a PRInt32. Appart that,
R=ducarroz

Comment 21

[(not reading, please use [e-mail] instead)]

Comment on attachment 81604 [details] [diff] [review]
Proposed patch, v1

sr=sspitzer, once you address ducarroz's 2 issues.

Comment 22

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 81604 [details] [diff] [review]
Proposed patch, v1

marking r= from comments
a=[e-mail] for checkin; please put into branch and trunk as needed.

Comment 23

[Cavin Song]

Fix checked into 1.0 trunk.

Comment 25

[scottputterman]

adding adt1.0.0-.  Let's get this in for rtm.

Comment 26

[Jaime Rodriguez, Jr.]

Uping impact to ADT2. This will be needed by a major customer to migrate users
from OE. 

adt1.0.0+ (on ADT's behalf) approval for checkin to the 1.0 branch. Pls check
this into the branch tonight, then add the fixed1.0.0 keyword.

Comment 27

[Ariana Ochoa]

excellent.

Comment 28

[Michael March]

Post-mortem:

I imported all my email it did not work at first.. The import result box was a
blank blue color with nothing in it.. The finish button was grayed out. It turns
out I was running out of disk space on the import. The issue is that Mozilla
does not gracefully handle running out of diskspace during an import.

Aside from that, all is well now.

Comment 29

[Cavin Song]

Michael, for the disk space issue can you file a separate bug to address it? 
Thanks. Glad that you can import all your folders now.

Comment 30

[Cavin Song]

Fix is in branch now.

Comment 31

[Ninoschka Baca]

Branch build 2002-05-02-06: WinMe
verified fixed on the branch since I was able to successfully import all Outlook
Express Mail with folders 5 levels deep and it now works for Michael March's
situation.

Leaving Resolved Fixed since this still needs to be verified on the trunk.

Comment 32

[Corrado Berti]

Trunk Build (Windows; U; Windows NT 5.0; en-US; rv:1.0.0+) Gecko/20020501

it works!

a big big tnx to you, Cavin;
it's time to migrate!

Comment 33

[Ninoschka Baca]

Trunk build 2002-07-09: WinMe
Verified Fixed.

Comment 34

[Ariana Ochoa]

nice work folks- our OE users out there trying to migrate will be happy to hear
this. Can you open a bug for dealing more gracefully when running out of disk space?