565010 - Importing a large Outlook 2003 folder (>4GB) causes always reindexing and data loss.

Status: NEW

(Thunderbird :: Migration, defect)

Description

[udx6bs]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; hu-HU; rv:1.9.1.9) Gecko/20100401 Ubuntu/9.10 (karmic) Firefox/3.5.9
Build Identifier: 3.0.4

Loss of data and slow annoying reindexing procedures after importing from outlook 2003. After importing one of the folder files became larger than 4gb (actually 6gb) and I suspect that this causes reindexing of folder anytime i select that folder. If i'm correct Thunderbird has a file size limit of 4gb.

I tried it on two different machines (see Additional Information) and the results were the same.

I solved it by splitting the Folder in Outlook 2003 in two. After reimporting everything went as expected, the file sizes where below 4gb and after one indexing procedure no reindexing occurred. Since this was on a clients machine, which i since delivered back, i cant reproduce it anymore or give any additional relevant information.

Hope i could be of any help. Sorry for my English.

Reproducible: Always

Steps to Reproduce:
1.) Took an Outlook 2003 Folder around 4 gb
2.) Imported it.
3.) After importing the folder file size became around 6 gb

Actual Results:  
Upon clicking on Folder, it always reindexes the complete Folder. Since this folder has around than 7000 messages. It takes quite a while (around 2-3 minutes on an i5 processor with ssd). Also the number of email messages is less then in Outlook 2003.

Expected Results:  
Suggestion: If folder file size reaches 4gb split it into 2 (sub)folders (or more) or at least give some warning/advices and the user should sort it out manually.

Computers:

1.) 
CPU: i5
RAM: 4gb
OS:  WinXP Pro Hungarian SP3
HDD: 128gb SSD + 500gb SATA
Antivirus: AVG 9.0

2.)
CPU: P4
RAM: 512mb
OS: WinXP Pro Hungarian SP3
HDD: 60gb
Antivirus: AVG 9.0

Comment 1

[Wayne Mery (:wsmwk)]

bienvenu, we don't have any means to check import size before the fact, do we?

Comment 2

[David :Bienvenu]

I don't know - I would think MAPI would expose the folder size somehow, but I don't know it for a fact.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to comment #2)
> I don't know - I would think MAPI would expose the folder size somehow, but
> I don't know it for a fact.

extended mapi supports this, based on skimming some web pages. 
but I didn't find anything for simple mapi

Comment 4

[David :Bienvenu]

(In reply to comment #3)

> 
> extended mapi supports this, based on skimming some web pages. 
> but I didn't find anything for simple mapi

Outlook import uses mapi 1, aka extended mapi.

Comment 5

[Wayne Mery (:wsmwk)]

xref Bug 537498 - In version 3.0, if IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac)

thanks for shedding that light. so, we should stop/warn if import might create folder >4gb/max thunderbird size. or something :)

Confirming, because I don't see a likely duplicate from skimming https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring&list_id=502382&field0-0-0=short_desc&bug_severity=critical&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&type0-0-0=anywordssubstr&component=Import&component=Migration&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc ... but it's hard to imagine we've never had import/migrate bugs reported involving 4gb folder size.

Comment 6

[Nikolay Shopik]

I can't be sure but AFAIK there limitation about 4Gb in outlook 2003. Maybe this can be related somewhat.

Comment 7

[David :Bienvenu]

Outlook used the limit the size of the .pst file to 4Gb, but this sounds like a problem with Thunderbird outlook import creating a local mail folder bigger than it can handle.

Comment 8

[Wayne Mery (:wsmwk)]

udx6bs, do you still have testcase data?

Comment 9

[Ilari Halminen]

I tried to import big folders and seems just to stuck. So no import almost nothing. Maybe I should try to slip folder and try again?

Comment 10

[Ilari Halminen]

(In reply to Ilari Halminen from comment #9)
> I tried to import big folders and seems just to stuck. So no import almost
> nothing. Maybe I should try to slip folder and try again?

Well, I tried again and it seems that transfer is just so slow that it seems to be jammed. I cannot give it enough time to see if it would really work.

Comment 11

[Wayne Mery (:wsmwk)]

I bet gloda indexing happens during the import. Might be good to defer until import is done

Comment 12

[David :Bienvenu]

Outlook import is really slow because we use the compose window backend to load the message. We've fixed our 4gb mailbox issue

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #12)
> Outlook import is really slow because we use the compose window backend to
> load the message. 

thanks for that info. So simulating someone typing in a message, like starting thunderbird with -compose ?  So is the easiest route to improvement to profile and improve the compose process?

Comment 14

[David :Bienvenu]

We use the compose window backend, so that we can use its code that creates an rfc822 message, and so we can handle mhtml (embedded images). I think caching the compose window backend would help, instead of recreating it for every message. And perhaps not using the compose window backend for simple messages.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #14)
> I think caching the compose window backend would help, instead of recreating it for
> every message. 

So this would not be impacted by jcranmer's de-cache front end work, correct?

Odd, but I don't see any open import/migrate performance bugs other than bug 77615, which is probably gone by now. So we could create a new bug for the cache idea.

Comment 16

[Joshua Cranmer [:jcranmer]]

(In reply to David :Bienvenu from comment #14)
> We use the compose window backend, so that we can use its code that creates
> an rfc822 message, and so we can handle mhtml (embedded images). I think
> caching the compose window backend would help, instead of recreating it for
> every message. And perhaps not using the compose window backend for simple
> messages.

Perhaps a better fix would be to expose APIs to make MIME messages that don't require the full compose window stack?

Comment 17

[Ilari Halminen]

(In reply to David :Bienvenu from comment #12)
> Outlook import is really slow because we use the compose window backend to
> load the message. We've fixed our 4gb mailbox issue

Well, good that reason is known, so there is no need to create another bug report. 

But if you plan to do some changes could you read other related bugs too? Like the that I cannot import everything at once, because Outlook is not shown. Note that most bugs are not marked new, they are just UNCONFIRMED.

Comment 18

[Wayne Mery (:wsmwk)]

(In reply to Ilari Halminen from comment #17)
> there is no need to create another bug report. 
I've filed bug 779840 because the outlook import speed issue is not part of the original bug report. plus, we only do one issue per bug report.

> But if you plan to do some changes could you read other related bugs too?
Sounds like you mean bug 432902. Problem is, at least prior to your comment, there is not enough information in the bug report for someone else to reproduce it.  If you have the skills, please provide more detail in that bug as to why the problem is happening to you.

Comment 19

[:aceman]

Starting with Thunderbird 52 folders do not have the 4GB limit of older version. Please retry with that version.

Are there any other issue remaining in this bug?