Closed Bug 532415 Opened 15 years ago Closed 15 years ago

280MB emails in Gmail produce 38GB All Mail file

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 537498

People

(Reporter: Happy.Cerberus, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Folder with 3 emails
5.65 KB, text/plain
Details
Same folder but after moving the emails back and forth to/from a local folder
5.66 KB, text/plain
Details 
User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; cs-CZ; rv:1.9.2b4) Gecko/20091126 SUSE/3.6b4-2.1 Firefox/3.6b4
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20091122 SUSE/3.0.0-4.1 Thunderbird/3.0

I have 280MB of emails in Gmail, but the All Mail file has 38GB (after compacting).

Reproducible: Always

Steps to Reproduce:
1. add Gmail account using IMAP
2. watch the file grow
Do you mean bug 517466 ?
No, this happens on the first fetch.
Bug 499630 and bug 487992 are already fixed. 
"Compact doesn't reduce file size" is possibly due to "deleted count=0".
Try next, please.
1. Show "Order Received" column(UID of mail) for [Gmal]/All Mail
2. Move mail of largest UID to [Gmail]/Trash
3. Move back the mail to [Gmail]/All Mail
4. "Compact" [Gmail]/All Mail via context menu
Will file size of "/.../[Gmail]/All Mail" be reduced?
No, it doesn't help, but the compacting ends very soon, so it could be that it just determines there's nothing to be done and continues.

I have a very stupid idea. Could it be that all emails are given the size of the largest one? In that case the 38GB would be relatively correct.
(In reply to comment #3)
> Bug 499630 and bug 487992 are already fixed. 
noting for the record that these fixes should be in Simon's build per comment 0

and getting this out of the horrible General component
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Version: unspecified → 1.9.0 Branch
Moving to IMAP component though as it's a specific offline-storage issue, which also may be Gmail-specific if only the All Mail folder is affected.

> (In reply to comment #4) Could it be that all emails are given the size of
> the largest one? In that case the 38GB would be relatively correct.

The size of each e-mail should be tracked individually in the offline storage, thus I don't think that's the issue.
Blocks: tb-gmailWIP
Component: Backend → Networking: IMAP
QA Contact: backend → networking.imap
Version: 1.9.0 Branch → Trunk
> Moving to IMAP component though as it's a specific offline-storage issue, which
also may be Gmail-specific if only the All Mail folder is affected.

I have the same issue with the Inbox and Sent folder, but yes this is a Gmail-specific issue. I have 4 IMAP accounts, and Gmail is the only one with this issue.
(In reply to comment #7)
> I have 4 IMAP accounts, and Gmail is the only one with this issue.

What size is the 38GB? File size of ".../[Gmail]/All Mail"? Total file size of a local directory for a Gmail IMAP accout?
If latter, it's normal if you requested auto-sync for all Gmail's IMAP folders, because Gmail Label is presented as IMAP folder by Gmail IMAP.
Err, for 280MB used on Gmails server to reach 38GB synchronized, you'd need
each message to have 138 labels assigned... don't think that's realistic. :-\
I'm talking about file size of single files.

Interesting thing just happened. I managed to compact Sent and Inbox, and they now have reasonable sizes. All Mail still refuses to compact.
I think this is the same issue that I'm experiencing with another IMAP server (where the bug has different manifestation).

See: https://bugzilla.mozilla.org/show_bug.cgi?id=532360

I can confirm that Gmail folders have no newlines between emails.
(In reply to comment #11)
> I think this is the same issue that I'm experiencing with another IMAP server
> (where the bug has different manifestation).
> See: https://bugzilla.mozilla.org/show_bug.cgi?id=532360
> I can confirm that Gmail folders have no newlines between emails.

According to comment by bug opener of bug 532360, null line(after last message header) is added in Tb's offline-store file if "offline use" is enabled, even though original mail downloaded from IMAP server doesn't have null line after last message header.
And, you apparantly enables "offline use" because you are talking about offline-store file for "[Gmail]/All Mail".

What do you mean by the "Gmail folders"?
How did you confirm "Gmail folders have no newlines between emails"?
Do you currently disable "offline use" of "[Gmail]/All Mail", after file size of offline-store file for "[Gmail]/All Mail" reached 38GB?
If I don't select offline use, the offline file is not created at all. But if I do, there are no newlines between emails.
(In reply to comment #13)
> If I don't select offline use, the offline file is not created at all.
> But if I do, there are no newlines between emails.

Can you attach offline-store file?
1. Create an IMAP folder(say xxx), delete file of xxx if exists,
   enable "offline use".
2. Copy some mails which produce "no newlines between emails" to the folder.
3. Open the folder, and wait for download by auto-sync.
4. Click other folder(close folder of xxx).
5. Keep back up of file named xxx.msf and xxx, and attach files to this bug.
Attached file Folder with 3 emails — 
3 emails sent by Thunderbird and manually moved to the folder (this is not Gmail account, I can't create a new folder with Gmail easily, but the problem is just the same).
Attachment #417994 - Attachment mime type: application/octet-stream → text/plain
(In reply to comment #15)
> Folder with 3 emails

Data in the file({LF}==0x0A).
>(snip)
> {last-maile-data-line-of-previouse-mail}{LF}
> From - Wed Dec 16 22:25:00 2009{LF}
> X-Mozilla-Status: 0001{LF}
> X-Mozilla-Status2: 00000000{LF}
> Return-Path: <SimonT@mail.muni.cz>{LF}
>(snip)

(i)   All data lines ends with {LF} (0x0A, instead of {CRLF}==0x0D0A).
(ii)  "From " line of Unix Mbox is same as one in local mal folder file.
(iii) X-Mozilla-Status:/X-Mozilla-Status2: is written.

How did you get the data? Copied mails to local mail folder?
"new-line of OS" is used for offline-store file of IMAP folder?
Copied to IMAP offline folder.
+ I always do a compact&reindex
Attachment #418010 - Attachment mime type: application/octet-stream → text/plain
Although "new line" caharacter is LF instead of CRLF, all mails you attached has null line after message headers(LF only line after last message header).
bug 532360 is irrelevant to mails you attached.
Are you sure? What I understand, the bug 532360 came to the conclusion that Thunderbird can't parse this format (missing newlines between emails).

This can be easily the cause of this bug. Thunderbird cannot parse the mailbox he stored and therefore he can't compact it.

Btw. I have an update, new profile now created a correctly sized file (I'm continuously deleting and receiving emails on the account, so I probably deleted an email that was causing this). Its with the same version of Thunderbird.
(In reply to comment #21)
> Are you sure? What I understand, the bug 532360 came to the conclusion that
> Thunderbird can't parse this format (missing newlines between emails).

Issue found in Bug 532360 : 
  If mail has no mail body and no null line after message headers(i.e. header
  only mail), Tb requires null line after message header to process mail
  properly. If mail in unix mbox file(local folder file, offline-store file),
  the null line(==null line just before "From " line) is inserted by Tb.
  So, problem of bug 532360 happens on IMAP folder of no "offline use".
The other bug manifests itself in both cases (offline/no-offline). Thunderbird only adds the null line ONLY if the mail is moved to a local folder.
wada, no fixes for this in 3.0.1?
(In reply to comment #24)
> wada, no fixes for this in 3.0.1?

I think I erased the mail that caused the problems, the issue disappeared before I upgraded thunderbird (simply by recreating the account, which didn't work before).
For original problem of this bug.

(In reply to comment #0)
> Gecko/20091122 SUSE/3.0.0-4.1 Thunderbird/3.0
> I have 280MB of emails in Gmail, but the All Mail file has 38GB (after compacting).
(In reply to comment #10)
> I'm talking about file size of single files.

Probably Bug 537498(fixed by bug 532323). Because offset value is corrupted, compact probably doesn't help. Even if compact is successful, data in offline-store is corrupted, because data is copied from wrong offset due to wrap around of offset value.
> Bug 537498 If IMAP offline-store file size exceeds 4GB, mails downloaded
> at over 4GB can not be read, and downloaded again & again, even if mail
> folder size is within 4GB (4GB limit is on Win, 2GB limit if Linux/Mac) 

As Bug 487992 is alreay fixed, rebuild-index is a recovery procedure.
> Bug 487992 File size of offline store of IMAP folder increases upon each "Rebuild Index"

Closing as DUP of bug 537498.
If wrong, re-open bug, please, with attaching required data for diagnosis.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: