428316 - 'Rebuild Index' (Gmail/iMap) forces REdownload COMPLETE Folder (6.5GB!)

Status: RESOLVED INCOMPLETE

(Thunderbird :: Mail Window Front End, defect)

Description

[Peter Fischer]

User-Agent:       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.1)
Build Identifier: http://mozilla.isc.org/pub/mozilla.org/thunderbird/releases/2.0.0.12/win32/de/Thunderbird%20Setup%202.0.0.12.exe

'Rebuild Index' on an OFFline readable Folder (Gmail/iMap) bloats Size of MBOX ;-(
1st:  6.5 GB
2nd: 13 GB
3rd: 19.5 GB
4th: 26 GB, and so on...

and: if u don't COMPACT Folders, after a while 'New Messages' don't appear, BUT Message-Count seems to be correct...

Reproducible: Always

Comment 1

[Brian Kennelly]

I am not sure this is a bug, but a natural consequence of re-building the index on an offline IMAP folder.  

If the index is being rebuilt, then the old index cannot be trusted, including any information about the local copies.  With an IMAP folder, discarding the index almost necessarily invalidates the mailbox.

Comment 2

[Peter Fischer]

hmmm...
i don't think so :-)
i'm sure it's NOT a natural behaviour of Thunderbirds iMap-implementation !-)

THEN: 'Windoof Mail' does it PERFECT ,-)

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> 'Rebuild Index' on an OFFline readable Folder (Gmail/iMap) bloats Size of MBOX ;-(
> 1st:  6.5 GB
> 2nd: 13 GB
> 3rd: 19.5 GB
> 4th: 26 GB, and so on...
> 
> and: if u don't COMPACT Folders, after a while 'New Messages' don't appear, BUT
> Message-Count seems to be correct...

(Q1) "Rebuild Index" only? Work Offline(or Download Now) is used too, isn't it?
(Q2) Above is "[Gmail]/All Mail" folder case, isn't it?
(Q3) Above size of "6.5GB",...,"26GB" is number of what? (displayed at where)
     Real file size on HDD? (...\[Gmail].sbd\All Mail when [Gmail]/All Mail)

Same phenomenon was observed by next test.
(0) Set a folder (say XXX) as "Offline folder"(Properties/Offline), size=32KB
(1) "Rebuild Index", and Offline/"Download Now" => ...\XXX : Size= 32KB
(2) "Rebuild Index", and Offline/"Download Now" => ...\XXX : Size= 64KB
(3) "Rebuild Index", and Offline/"Download Now" => ...\XXX : Size= 96KB
(4) "Rebuild Index", and Offline/"Download Now" => ...\XXX : Size=128KB
(5) "Compact Folder" => ...\XXX : Size=32KB

"Download Now" downloads not-downloaded-yet mail only and appends newly downloaded data to ...\XXX. And you did clear status of "downloaded" by "Rebuild Index". So above behavior is very natural(not so happy for some users though).

Gmail provides 6GB space for free Gmail account. And "[Gmail]/All Mail" folder of Gmail IMAP(== "All Mail" folder/label of Gmail via Web) holds all mail data. Then  file size of downloaded data for offline-use can exceeds 4GB(MS Win. 2GB when other OS) limit of Tb's mail folder file size. This "4GB limit" is due to 32bit integer for offset. So mail data placed at 4GB or greater can not be accessed.

(Q4) What happens when next test?
     Copy file of "...\[Gmail].sbd\All Mail" to directory for "Local Folders",
     and restart TB. If .msf recreatio(rebuild index) works even for 6GB file,
     XXX can be seen as mail folder of XXX.
     Check number displayed in "Order Received" column.
     (0Gb-2GB:positive, 2Gb-4GB:negative. mail can be read)
     (4GB-6GB: mail can not be seen)

When IMAP, mail is accessed by UID(number in "Order Received" column). So above "4GB limit" has no relation when non offline-use folder. However, access to downloaded mail data for offline-use is affected by "4GB limit of local mail folder file".
Don't use offline-use option for "[Gmail]/All Mail" folder.

Comment 4

[Ludovic Hirlimann [:Usul]]

Peter could you answer wada's questions ?

Comment 5

[WADA:World Anti-bad-Duping Agency]

To Peter Fischer(bug opener):

(1) I've opened bug for "infinite increase of offline-store file size" based on your report and report in some other bugs.
> Bug 487992 File size of offline store of IMAP folder increases upon each "Rebuild Index"

(2) If your bug summary of "Rebuild Index forces REdownload COMPLETE Folder" correctly represents your "bug report at bugzilla.mozilla.org", it's apparently INVALID, because feature of "Rebuild Index" is designed so.

(3) Correction of my comment #3.
Offset-value for access of saved mail data in offline-store seems to be alredy changed to 64bits value, and access of mail data in more than 4GB offline-sore file seems to have no problem.

Comment 6

[[:Aureliano Buendía]]

Peter what you think about comment #5?

Comment 7

[Joshua Cranmer [:jcranmer]]

RESO INCO due to lack of response to last question. If you feel this change was made in error, please respond to this bug with your reasons why.

Comment 8

[Michał]

People,

imagine having 4GB of e-mail messages and a corrupted index file. One wants to rebuild the index and see ones precious messages fast. Unfortunatelly one has to wait until those 4GB are transfered back to ones computer regardless message store contents. Excuse my french but this "sucks".
Why don't you grab all the headers, compare with message store headers and retransmit only those missing or invalid?
Regards.