Closed Bug 537498 Opened 13 years ago Closed 11 years ago

In version 3.0, if IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac)

Categories

(MailNews Core :: Networking: IMAP, defect)

1.9.1 Branch
x86
Windows XP
defect
Not set
major

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: World, Unassigned)

References

Details

(Whiteboard: fixed in 3.1 by bug 532323 )

If IMAP offline-store file size excceds 4GB, mails written at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB.

Following is checked with Tb 3.0.

[Steps to reproduce]

(1) Create a 4GB file of text data (F0) in local IMAP mail directory
(2) Restart Tb, create an IMAP folder(F1), set offline use=on
(3) Emulation of bug 487992
(3-1) Rename F0 to F1
(3-2) Copy a mail(mail-1) to F1 folder
    => mail-1 is downloaded at offset=4GB of file of F1
    => No problem occurs.
       Mail data seems to be managed by two 32bits offset value;
       offset-1 : offset of fisrt mail in offline-store
       offset-2 : each mail's offset from offset-1
(4) Force overflow of offset-2 value.
(4-1) Terminate Tb
(4-2) Append 4GB text data to file of F1.
      i)   Get write-timestamp of F1
      ii)  Open F1 with "write append", append 4GB data to F1, close
      iii) Change write-timestamp of F1 back to value of i)
      Emulation of repeated mail add, mail delete, without folder compaction.
(4-3) Restart Tb
(4-4) Copy mail-2 to IMAP folder of F1
      => mail-2 is downloaded at 8GB+ of file of F1
         offset-2 of mail-2 overflows
(4-5) Click mail-2
      => fails to find in offline-store
      => fetched from IMAP server again, and saved in Disk Cache
      => mail-2 is displayed normally
(5) Emulation of removal of mail-2 from Disk Cache
(5-1) Terminate Tb
(5-2) Delete all files in Tb's Disk Cache
(5-3) Restart Tb, Work offline
(5-4) Clink mail-2
      => "Mail is not downloaded yet" like text is displayed
(6) Work online, clink mail-2
    => mail-2 is downloaded and appended to end of file of F1
       offset-2 of mail-2 still overflows
    => Sama as step (4-5)

Even if mail folder is smaller than 4GB, repeated download can happen.

If mail folder is smaller than 4GB, "Compact Folder" clears above situation.

If mail folder is greater than 4GB then offline-store size exeeds 4GB, fix of bug 462665 is required.
If mail folder is greater than 4GB, split of IMAP mail folder is required.
([Gmail]/All Mail case)
  1. Define Gmail Label of All-01, All-02, ... (IMAP folder of All-01, ...)
  2. Add one of All-01, All-02, ... to all conversations in "All" folder
  3. Disable offline use of [Gmail]/All
  Above can be done with user defined archive folder with archive_granularity.
Summary: If IMAP offline-store file size excceds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB → If IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB
As written in bug 532323 comment #3, problem can be said;
  If total synced data size exceeds 4GB, wrap-around of offset value occurs.
  Bug 532323 :
    IMAP mail folder size is larger than than 4GB.
    Synched data size always exeeds 4GB, if size of aut-sync data is not limited.
  This bug(Bug 537498) :
    IMAP mail folder size is smaller than 4GB.
    Total synced data size exceeds 4GB. 
    (no need of 4GB mail in IMAP folder to test in this case)
Depends on: 532323
we should try to get this for 3.0.2. 
you would think it's also affecting some trunk users
blocking-thunderbird3.0: --- → ?
Possible workaround for "larger than 4GB" case, if auto-sync of such huge folder is required.
 - Synchronization&Storage, Synchronize most recent NNN days.
   Periodical compaction may be needed to avoid this bug.
 - Options/Advanced/Network&Disk Space, up to 16000 MB of space for the cache.
   Unless cache is cleared, repeated download probably won't happen,
   except some special cases such as bug 246415, bug 516211. 
   (downloaded only twice. at over 4GB of offline-store, and in disk cache)
Can search be done on data in disk cache?
FYI.
Disk Cache may not be a effective workaround due to bug 175600.
blocking-thunderbird3.0: ? → needed
If I'm understanding correctly, I think this would make Thunderbird effectively unusable for anyone with a sufficiently large mailstore (which seems likely to be at least somewhat common given GMail and Yahoo data storage these days).  That said, there don't seem to currently be any markings on this bug that would indicate that folks are complaining about it regularly in support venues.  Roland, have you seen any complaints that might be this?  Marking as needed+ based on what I currently know and Wayne's comment.  Assigning to bienvenu for further triage, since he has more context in his head than I do.
Assignee: nobody → bienvenu
Flags: blocking-thunderbird3.1+
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 532323
(In reply to comment #6)
> *** This bug has been marked as a duplicate of bug 532323 ***

To David:Bienvenu: Why DUP of bug 532323?
This bug is mainly for problem of Tb 3.0.x(1.9.1 builds) even after fix of bug 532323 by Tb 3.N(where N>=1, 1.9.2 builds or later).
blocking-thunderbird3.0: needed → ---
Flags: blocking-thunderbird3.1+
Same root cause as bug 532323, and fixed by patch for bug 532323. But the patch for bug 532323(64bits offset use) is for Tb 3.1 only.
Re-opening in order not to loose "folder smaller than 4GB" case and problem on Tb 3.0.x even after fix of bug 532323.
Assignee: bienvenu → nobody
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Whiteboard: will_be_fixed_by_bug_532323
Summary: If IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB → If IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac)
Duplicate of this bug: 532415
No longer blocks: tb-enterprise
Severity: normal → major
Summary: If IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac) → In version 3.0, if IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac)
Whiteboard: will_be_fixed_by_bug_532323 → fixed in 3.1 by bug 532323
Tb 3.0.0 is no longer supported. No need to keep this bug for Tb 3.0.0.
=> Wontfix
Status: REOPENED → RESOLVED
Closed: 13 years ago11 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.