Closed Bug 482486 Opened 15 years ago Closed 15 years ago

Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=2**32-1 bytes, while it is empty or small number of mails only) (~4GB)

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 494706

People

(Reporter: jvidal, Unassigned)

References

Details

Attachments

(1 file)

Mail folder file of mail data with 0x00
4.38 KB, text/plain
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2
Build Identifier: versión 2.0.0.19 (20081209)

This has happened to me a few times. Sometimes, for no reason I get a mesage saying that my trash folder has reached its limit, even though it's almost empty & I had compacted folders recently. looking into my user account, the file "trash" is about 4GB in size!, but it's totally bogus, it's not actually taking up 4GB of disk space (I know this because after i empty the trash, the file returns to its normal size, but the free space on the disk doesn't increase by 4 GB).
It's kindy complicated to explain, I hope I made myself clear.

Reproducible: Sometimes

Steps to Reproduce:
1.use TB normally
2.
3.
Actual Results:  
Trash reaches incredibly big size, totally bogus.

Expected Results:  
nothing weird should happen.
Jorge, have you asked about this in a support forum?
You know you need to compact the folder every now and again, right? (File | Compact folders, that deletes the messages "marked as deleted" and not show to you.)
did you read what i wrote?
I quote: "...even though it's almost
empty & I had compacted folders recently. looking into my user account, the
..."
Plus, By no means I had received 4GB of mail in such a short time, not even in my entire "internet-life"...
Ah, sorry, missed that. So how large is it really then? Check your profile folder, and look at the file size of the file named as your trash folder, and without extension. (http://kb.mozillazine.org/Profile_folder)
Like I said, it reaches 4GB, even if I had compacted recently. Plus, there's no way I received 4GB of e-mails in a short period of time. This has happened to me a few times in the last months, it's OBVIUOSLY a bug. And, like I said, the file doesn't even use 4GB of disk. After returning to its normal size (0 if it's empty), I DO NOT have 4GB of extra disk space.
Oh, and BTW, the trash folder doesn't need compacting, emptying it is enough, it returns to 0. And, moreover, I have "emty trash on exit" selected.
So there's no way in hell it can reach that size if this isn't a bug.
xref
Bug 61960 -  local folder empty trash doesn't compact all local folder
Bug 428947 -  Compact Gmail IMAP folder doesn't reduce folder size (except for in All Mail, Trash and Spam)
Bug 468722 -  Trash box will not accept an more mail - even after emptying and compacting folders
this is an imap account?
not gmail?

Bug 467305 -  Compact IMAP folder does not work
nope, pop account.
Finally, those bugs relate to my problem!
(In reply to comment #10)
> *** Bug 494706 has been marked as a duplicate of this bug. ***

Has few nice attachments that might help understand what is going on.
(In reply to comment #0)
> I get a message saying that my trash folder has reached its limit,
> even though it's almost empty

Please note next.
>  "mail folder file size" != total size of "active mail size"
>  "mail folder file size" == total of "active mail size" & "deleted mail size"  
See description about MSG_FLAG_EXPUNGED in X-Mozilla-Status: header.
> http://www.eyrich-net.org/mozilla/X-Mozilla-Status.html?en

> & I had compacted folders recently.

If "outdated msf" condition exists, bug 492344 can occur. Once bug 492344 occurs, compaction will not be invoked until next "deletion of active mails" will happen, or until normal(invoked by explicit folder open after restart) & successful rebuild-index will happen. (bug 492344 is phenomenon found during duplication test like this bug).

To Jorge Vidal Wulff(bug opener):

Do you still see problem frequently?  
Does your backup system changes timestamp of file? ("Trash" in your case).
Do you do "restore Trash file from backup"?
(Sorry i filed the duplicate 494706, but Bugzilla's search function apparently
needs to be improved or bug reports need to include keyword lists.)

This bug is probably related to the serious bugs causing data loss due to
defective compacting and causing Thunderbird to freeze due to long/interminable
compacting:  Bug 463359, Bug 468722, Bug 489959, Bug 493065

(In reply to comment #0)
> looking into my user account, the
> file "trash" is about 4GB in size!, but it's totally bogus, it's not actually
> taking up 4GB of disk space (I know this because after i empty the trash, the
> file returns to its normal size, but the free space on the disk doesn't
> increase by 4 GB).

Are you really sure about that claim of not "actually taking up 4 GB of disk
space"? In my case at least, the file definitely takes up that much room. It
takes 5 minutes to make a copy of it within the same folder using Windows
Explorer on a fairly new PC. I can send screenshots showing the size of the
free disk space decreasing by 4 GB, and you can analyse the second attachment
to bug 494706 to see what the beginning of the file contains.
(In reply to comment #0)
Could it be that the reason you thought the file doesn't take up 4 GB of space is because you deleted it using Windows Explorer? When something gets moved to the Windows recycle bin, the space it took up (and still takes up) isn't counted as free space by Windows Explorer (until you empty the recycle bin).
it seems you guys are not very good at reading.
I said I EMPTIED THE TRASH FOLDER, not deleted the file.
After emptying the trash, my free space didn't increase, even if now the trash was 0 bytes in size (vs 4 GB a few moments before).
BTW, since using TB 3 beta, I've never seen this problem again.
Still I think it's not a "folders-not-being-compacted-issue", because there's no way in hell I could have received 4 GBs of mail in only a few days.So the trash size is completely bogus.
(In reply to comment #15)
> it seems you guys are not very good at reading.
> I said I EMPTIED THE TRASH FOLDER, not deleted the file.
> After emptying the trash, my free space didn't increase, even if now the trash
> was 0 bytes in size (vs 4 GB a few moments before).

I'm very good at reading what other people say and have a lot of experience in dealing with other people's computer problems, and when someone claims something that doesn't make sense, it's usually a good idea to suggest a possible explanation instead of just claiming that what they say is probably incorrect. We are all human, and just because you say that you emptied the Trash it could very well have been that you were thinking one thing and saying another. 

In any case, what you say about a 4 GB file not taking up any disk space does not seem possible unless you looked wrong or Windows Explorer is not indicating the correct size of files or you forgot to press F5 to refresh the data or something similar. Unless you can attach a copy of a file whose size is much smaller than indicated in Windows Explorer, most people will have trouble believing you understood and described the situation correctly.

As explained, my 4 GB file definitely took and takes up that much disk space and the size of free space on my HD and of my backup folder on the external HD increased and decreased by exactly that amount when the file was created by TB and deleted by emptying the Trash folder of its 15 emails and manually compacting.

> Still I think it's not a "folders-not-being-compacted-issue", because there's
> no way in hell I could have received 4 GBs of mail in only a few days.So the
> trash size is completely bogus.

Did you read what i wrote? Did you take a look inside my 4 GB file (1/10 000 attached to bug 494706) or your 4 GB file? My file and no doubt yours consisted of only a few emails followed by almost 4 GB of junk data (rendered in Word as ÿÿÿÿÿ) created by Thunderbird. Since TB creates this kind of huge nonsense file, it is very possible that TB sometimes creates similar nonsense during the compacting process. In any case, such a huge file can make the compacting process take a long time or prevent it from ever completing and thereby cause problems either directly or due to users shutting down TB or using it during the compacting process, which can cause data loss.
it seemed fairly clear to me. but now that you restate it, it's slightly different from comment 0. If one puts exactly what actions are done in *numbered* steps as requested in the new bug form, it does make it much clearer for everyone.  It's extra work on for reporter, but it makes it easier for the 5-10 people that will read it later.

(In reply to comment #13)
> This bug is probably related to the serious bugs causing data loss due to
> defective compacting and causing Thunderbird to freeze due to long/interminable
> compacting:  Bug 463359, Bug 468722, Bug 489959, Bug 493065

good list. 489959 I'm not so sure is related

marking critical because the db is corrupted.
Severity: major → critical
Component: General → Database
Product: Thunderbird → MailNews Core
QA Contact: general → database
Summary: Trash folder reaches ridiculously big size → Trash folder reaches ridiculously big size populated with bogus data
(In reply to comment #17)
> (In reply to comment #13)
> > This bug is probably related to the serious bugs causing data loss due to
> > defective compacting and causing Thunderbird to freeze due to long/interminable
> > compacting:  Bug 463359, Bug 468722, Bug 489959, Bug 493065
> 
> good list. 489959 I'm not so sure is related

That bug has the same error message about Trash being full despite being empty or nearly empty (not yet compacted) described by reporter of this bug (482486) in comment #0 and as reported by me in bug 494706.
 
> marking critical because the db is corrupted.

bug 494706 could use same change
1) The 4 GB data garbage file is obviously produced by Thunderbird (i can't think of any other Windows or malware or antimalware or other program process that would or could do this). 

2) The 4 GB data garbage file is obviously produced during compacting (TB doesn't significantly alter the db at any other time).

1) and 2) are of course only parts of the answers to the questions of when and why TB does this nonsense, but we can understand the process better if someone can explain the following:

3) How is it physically possible for Thunderbird to produce a 4 GB file (out of one less than 200 kB) within 12 to 60 seconds?! (see bug 494706) It takes several minutes to copy a file that size (within the same Windows Explorer folder), so how can it be created in a fraction of that time?
Well, 3) explains why after emptying trash, the free space didn't increase by 4GB.
The file size is bogus.
(In reply to comment #20)
> Well, 3) explains why after emptying trash, the free space didn't increase by
> 4GB.
> The file size is bogus.

The file size is not bogus - it's really that big. It seems you didn't save your 4 GB file, still haven't bothered to look at the portion of mine attached to bug 494706, and misunderstood something when it *looked* to *you* (erroneously) like your file was simultaneously 4 GB in size but hardly took up any room. You haven't really explained this illogical observation/claim, but it seems you're saying that the same program (Windows Explorer) indicated the file's size to be both 4 GB and very small. That is simply not possible and a clear indication of user / observer error. (Different programs can possibly disagree about a file's size, but not the same program.) See above for some possible causes of your incorrect observation. Free space did definitely increase after emptying Trash in my case and most probably yours too - you just got confused by something or forgot to press F5 in Windows Explorer.

In any case, you're not being logical. If you accept that it takes 5 minutes to copy the file, that's already ample proof that its size is not bogus. Just because you and i don't know how TB can create a huge file much faster than Windows Explorer can copy it does not mean that it's not as big as shown by Windows Explorer (and Showman Usage Pie Chart).
I never said the file took 5 minutes to copy, that was someone else.
If a 4GB file was created in a lot less than 5 mins, then the only logical explanation is that the file was not actually 4GB in size.
I've seen stuff like this before, some p2p programs create the file on disk, they show the "final" size on windows explorer, but they're not actually taking any space. They start taking space as they are being downloaded, 'til they reach the final size.
I believe this is what's called "sparse file".
Please, I'm not a newbie, I have 15+ years of experience with computers.
I'm the one who said it took that long. 

You keep avoiding these crucial points: 

1) You apparently don't have a copy of your 4 GB file to back up your claim about "bogus size"

2) I presented you with a copy of my 4 GB file and you refuse to look at it and admit that it really was as big as Windows Explorer said it is
before you scare everyone off, how about we take a break until someone who knows the db or compact chimes in
Many file-system have a concept of "sparse" files.  You can seek to the 4 gigabyte marker, and the system just pretends you wrote zeroes in all the bytes you never actually wrote to.  Such file-systems only need to commit storage when bytes are actually written to the page.  There is still a book-keeping cost for the zero pages, of course.

I would presume that such an erroneous seek would be a good explanation for what is happening here, most likely as a result of unsigned integer underflow.  If someone has time to look into it, that is what to look for.
The compact code merely copies non-deleted messages to a temp file, and copies the temp file back over the original folder.  The message copy code does NOT preallocate space in the target file, afaik - it seeks to the end of the target file and reads data from the source message, and appends that date to the target folder. The code is in nsMsgLocalMailFolder::CopyData - I don't see how lineLength could become negative unless there's some reading off into memory past the end of a block. But even in that case, we would be writing an enormous amount of data, which should take time.
Of course I don't have a copy...why would I?
Like I said, the file "trash" returned to its normal size when I emptyied it, never had a chance to back it up (didn't want to, either).
I haven't got the time right now to look at your file (plus I don't know why I should).
I'm pretty sure my free space didn't increase after the file went from 4 GB to 0, I checked it, like I said, I'm not a newbie.
I think andrew is on to something here.
(In reply to comment #0)
> (I know this because after i empty the trash, the file returns to its normal size,
> but the free space on the disk doesn't increase by 4 GB).

If file is not shared, MS Win(XP SP3) freed used extents upon write open with replace(write at top of file instead of append).
  1. a file uses some extents.
  2. Stream(file_name,"C","OPEN WRITE REPLACE");  (Open Object REXX is used)
  3. file size becomes 0, and free disk space increases (checked by DIR command)
I guess file was opened by other one too (Tb's other process, other software such as anti-virus soft ware, ...).

Jorge Vidal Wulff(bug opener), free disk space did not increase even after shutdown of Thunderbird?
The missing 4GB disk space is still not returned to free extents?

For "4GB file size within short time".
As Andrew Sutherland, "SEEK" allocates extents at least on MS Win-XP SP3.
  1. file size=0 (opened with "OPEN WRITE REPLACE")
  2. Stream(file_name,"C","SEEK +65536");  (Open Object REXX is used)
  3. File size increases. Free disk space decreases.
To Ekhart(only one who can produce problem currently):

"Order Received" column is "offset of mail in mail folder file", which is kept in 32bits unsigned integer when MS Win(32bits signed integer if Linux/Mac OS X), when local mail folder. So "Order Received" column value can be value between 0 to 4294967295.(==2**32 -1 == 4294967295 -1).
(Q1) What value is displayed at "Order Received" column when problem occurs?
     (when 4Gb Trash file is observed shortly after Empty Trash)

Even if Trash folder file size became near 4GB, mail data can be added unless  Trash folder file size exceeds 4GB.
(Q2-A) File size of Trash exceeded 4GB(4294967296 bytes)?
(Q2-B) Copy of very small mail to Trash is possible when problem occurs?
       From - ...[CRLF]Subject: X[CRLF][CRLF][CRLF] is sufficent to test.
(Q2-C) If copy is sucsessful, what value is displayed in "Order Received" column
       for the copied mail?
Attached data to bug 494706 by Ekhart.
> N * (mail of X-Mozilla-Status: 0001)                     A  
> (last part of last mail)                                 |  Mail data before
> ---...---[CRLF]                                          |    corruption
> [CRLF]                                                   |
> [CRLF]                                                   V
> Garbage line, which is displayed in U+FFFD if interpreted as UTF-8.
>   First 80 bytes of the garbage : all 80 bytes is 0x00.

To Ekhart(bug opener of bug 494706):

Did you delete other mails(moved to Trash) and executed "Shift+Delete" for them, after you kept back up of normal Trash file, before you excountered the corruption and kept back up of corrupted Trash file?
If yes, "Compact Folder" can be invoked, but if no, "Compact Folder" won't be invoked because there is no mail of MSG_FLAG_EXPUNGED=On, unless internal deletion by "Retention Policy", "Junk Purge" etc. is invoked for Trash.
(In reply to comment #29)
> To Ekhart(only one who can produce problem currently):

It happens sometimes, but i haven't been able to reproduce it again since i opened bug 494706

> "Order Received" column is "offset of mail in mail folder file", which is kept
> in 32bits unsigned integer when MS Win(32bits signed integer if Linux/Mac OS
> X), when local mail folder. So "Order Received" column value can be value
> between 0 to 4294967295.(==2**32 -1 == 4294967295 -1).
> (Q1) What value is displayed at "Order Received" column when problem occurs?
>      (when 4Gb Trash file is observed shortly after Empty Trash)

As explained in bug 494706, i got and noticed the 4 gigabyte (GB) - not gigabit (Gb) - file after/when deleting emails in TB produced the error message "Trash full...empty Trash and compact (or something like that)" etc. 

As explained in that and this bug report, the 4 GB file is *not* observed shortly after Empty Trash - on the contrary, it's replaced by a small one (0 kB) when the trash is emptied.

As explained by others above, this is probably a sparse file that probably doesn't actually take up 4 GB of space until it's accessed by something like antivirus or backup software.

I just put the 4 GB file (renamed to TrashCopy) back into the TB profile, and TB displays an empty folder as it did when the bug occurred (and the error messages occurred). Making TB search in subject or sender produces nothing (at least so far, after several minutes).

> Even if Trash folder file size became near 4GB, mail data can be added unless 
> Trash folder file size exceeds 4GB.
> (Q2-A) File size of Trash exceeded 4GB(4294967296 bytes)?
> (Q2-B) Copy of very small mail to Trash is possible when problem occurs?
>        From - ...[CRLF]Subject: X[CRLF][CRLF][CRLF] is sufficent to test.
> (Q2-C) If copy is sucsessful, what value is displayed in "Order Received"
> column
>        for the copied mail?

Both times i noticed the bug, it produced a Trash file with the exact same number of bytes: 4,194,305 kB. (I can attach a copy of the beginning of the second one too if someone wants it.)
(In reply to comment #30)
> Attached data to bug 494706 by Ekhart.
> To Ekhart(bug opener of bug 494706):
> 
> Did you delete other mails(moved to Trash) and executed "Shift+Delete" for
> them, after you kept back up of normal Trash file, before you excountered the
> corruption and kept back up of corrupted Trash file?
> If yes, "Compact Folder" can be invoked, but if no, "Compact Folder" won't be
> invoked because there is no mail of MSG_FLAG_EXPUNGED=On, unless internal
> deletion by "Retention Policy", "Junk Purge" etc. is invoked for Trash.

Not sure what you're trying to say, but 
1) I did not use shift+delete
2) I did not make a backup before i encountered the corruption
(In reply to comment #31)
and 
(In reply to comment #29)

> Both times i noticed the bug, it produced a Trash file with the exact same
> number of bytes: 4,194,305 kB. (I can attach a copy of the beginning of the
> second one too if someone wants it.)

Oops, i meant 4,194,304 kB
(In reply to comment #32)
> 1) I did not use shift+delete

"Compact Folder" doesn't seem culprit, still I'm not sure though.

> Both times i noticed the bug, it produced a Trash file with the exact same
number of bytes: 4,194,304 kB.

If KB==1024, 4294967296 bytes. If KB==1000, 4194304000 bytes.
File size seems 4294967296(==2**32==4GB), because Tb says "Trash is full", and Tb doesn't append mail data if file_size+added_mail_size>4GB(issues error message), and probably if offset_of_added_mail>=4GB(file size somehow exceeded 4GB already).
SEEK to offset==0-1 using signed 32bits integer(2's complement if negative) which is interpreted as offset by unsigned 32bits integer(2**32-1) was executed? If so, by whom?
Remaining questions:
If SEEK related issue, why Trash only problem? Why no problem on Inbox or Junk?
"Delete(Move to Trash) by message filter" is relevant?
Mail Purge/Junk Purge is still "Move to Trash" and is relevant to problem?
To Jorge Vidal Wulff(bug opener) and Ekhart:

I don't know other people than you two people and opener of Bug 468722 who complaints about such phenomenon. All three peoples are MS Windows user.
AFAIR, I saw report of garbage in mail folder file produced by anti-virus software in other bug(s), although garbage was not 0x00 AFAIR.

What anti-virus software do you use? Which version? Newest?
(I use avast! 4, free Home Editon, and I have no experience of such problem.)
Note: If anti-virus software is cause, I can't explain why Trash only problem.
I use KAV 2009. But I don't think that's the problem. Since I moived to TB 3 beta, I've never seen this problem again, in several months.
(In reply to comment #36)
> I don't know other people than you two people and opener of Bug 468722 who
> complaints about such phenomenon. All three peoples are MS Windows user.
> AFAIR, I saw report of garbage in mail folder file produced by anti-virus
> software in other bug(s), although garbage was not 0x00 AFAIR.

In addition to another person responding to bug 468722, there are very many people experiencing the same problem of “full” Tb folders and similar problems as simple forum and Web searches show:
http://www.google.com/cse?cx=003258325049489668794%3Adrr0nlojlas&ie=UTF-8&q=trash+full&sa=Go
http://www.google.com/cse?cx=003258325049489668794%3Adrr0nlojlas&ie=UTF-8&q=inbox+full&sa=Go
http://www.google.com/cse?cx=003258325049489668794%3Adrr0nlojlas&ie=UTF-8&q=sent+full&sa=Go

Many perhaps have 4 GB files but are not aware that they do and only complain about Tb not working properly with symptoms ranging from sluggishness to data corruption to major data loss. Even the forum moderators don’t seem to be aware of this bug and its effect because they rarely ask users to search for monster files throughout their profile and instead concentrate their efforts on msf files. I didn’t either in the many cases of people asking me for help with never-ending compacting. I will update the knowledgebase article soon.

So the problem of folders full of bogus data is widespread and very serious due to the resulting data corruption and data loss and due to frustrated users dropping Tb. We don’t know whether the “full” folders are always 4 GB in size, but it seems probable based on your (WADA’s) comments.

> What anti-virus software do you use? Which version? Newest?
> (I use avast! 4, free Home Editon, and I have no experience of such problem.)
> Note: If anti-virus software is cause, I can't explain why Trash only problem.

The same problem occurs in other folders as shown above.

I’m very sure this is not really an antivirus problem, more a problem of Tb creating unnecessarily large sparse files and then antivirus and backup software accessing them and thereby filling them with bogus data. I use Avira AntiVir. 

David said that the compact code doesn’t pre-allocate space, but WADA mentioned that Tb files are opened by other Tb processes.

> "Delete(Move to Trash) by message filter" is relevant?
> Mail Purge/Junk Purge is still "Move to Trash" and is relevant to problem? 

I don’t have any filters moving messages to trash. 
Not sure what Mail Purge/Junk Purge is, but I never use “delete mail marked as junk in folder” or “run junk mail controls on folder”.
(In reply to comment #38)
> there are very many people experiencing the same problem of "full" Tb folders and similar problems as simple forum and Web searches show: (snip)
>(snip)
> So the problem of folders full of bogus data is widespread and very serious (snip)

Ekhart, are you sure that they are never "real 4GB folder size problem"?
(Simply no compaction, including "compact folder doesn't remove deleted mail data". If Trash, no delete+compaction, no empty trash.)
AFAIK, many of "Folder full" in forums are "no compaction of non-Trash folder" case.

FYI.
Problem like Bug 449741 Comment #8 or Bug 321371 Comment #69 exist.
Bug 449741 Comment #8 reported phenomenon of garbage in move target folder.
I don't know it's fixed by fix of Bug 450359(fixed1.8.1.18) or not.
Mail box has 4 mails. Mail data is; ([NULL]=0x00,[CRLF]=0x0D0A)
> Mail line number = 00000001[CRLF]
>(snip)
> Mail line number = 00000016[CRLF]
> ***...***[CRLF][CRLF]
> [NULL]***...***[CRLF][CRLF]
> [NULL]***...***[CRLF][CRLF]
> [NULL]***...***[CRLF][CRLF]

(1) Rebuild-Index & mail display is not affected by 0x00.
(2) Select all, Copy to a local folder
(3) At copy target folder.
    (3-A) mail-1 to mail-3 : top part of next mail is displayed.
    (3-B) mail-4           : data after 0x00 is lost.
Phenomenon of (3-B) is already found and reported to bug 477799.
We couldn't be aware of (3-A), because we tested with single mail only folder. 

Mail's offset value is correctly set. But mismatch between mail length data in MailDB and real mail data length in mail folder file is produced upon copy(upon move too). This may affect on calculation of next seek position.
Above test result is obtained by Tb trunk 2009/5/24 & 2009/5/27 build, and was different from Tb 2. Phenomenon by "copy of 0x00 to local folder" depended on Tb version. 
  Tb 2.0.0.19(Bug 477799 Comment #9): Mail data after 0x00 was replaced by 0x20. 
  Tb 2.0.0.21: Mail data including 0x00 was copied to target mail folder.
               No mail data loss when "copy to local folder". i.e. "copy to
               local folder" case of Bug 477799 is resolved by Tb 2.0.0.21.
               (I don't know about "copy to IMAP folder" case.)
This bug is for Tb 2.0.0.19, and Bug 494706(by Ekhart) is for Tb 2.0.0.21. So "0x00 in mail data" is not original cause of 4GB folder issue, although similar phenomenon to Bug 477799 or above can occur once garbages of 0x00 is generated, and if Tb 2 & Tb 3 is used alternatively when mail of 0x00 exists.
Sorry for my confusion.
(In reply to comment #39)
> Ekhart, are you sure that they are never "real 4GB folder size problem"?
> (Simply no compaction, including "compact folder doesn't remove deleted mail
> data". If Trash, no delete+compaction, no empty trash.)
> AFAIK, many of "Folder full" in forums are "no compaction of non-Trash folder"
> case.

I copied URLs to search results without "4 GB" - the following give more results *also* showing users with 4 GB files despite compacting and deleting msf files and despite having messages amounting to much less than 4 GB:
http://www.google.com/cse?cx=003258325049489668794%3Adrr0nlojlas&ie=UTF-8&q=trash+full+%224+GB%22&sa=Go

for example 
http://forums.mozillazine.org/viewtopic.php?f=39&t=1189715&p=6239975
http://forums.mozillazine.org/viewtopic.php?f=39&t=1189715&start=0
http://forums.mozillazine.org/viewtopic.php?f=39&t=1024815&start=15&st=0&sk=t&sd=a
http://forums.mozillazine.org/viewtopic.php?f=29&t=889395
http://forums.mozillazine.org/viewtopic.php?f=39&t=1024815&p=5410175
http://forums.mozillazine.org/viewtopic.php?f=39&t=1129915&p=5947205
Problem seems to have occurred(or be occurring) on much many users than I thought...

To Ekhart:

How long is the continous 0x00? Single continuous 0x00? Multiple continouis 0x00s?
Can you check last part of the corrupted/4GB Trash file? (several hundreds KB - 1MB)
All data is 0x00? Or something is written? If so, at where/what data? Continuous 0x00 is seen before the non 0x00 data?
(If it's hard for you, I can write Open Object REXX script or PHP script for it.)
I split the file into 100 parts using Split Files and split the last of those parts into 100 parts and added it to the attachments at bug 494706. Do you need more? Looks like all the same (ÿÿÿÿÿÿ in Word) as at the beginning of the bogus data.
(In reply to comment #45)

First part  : Non-Null=107844 bytes(mail data), and continuous Null=321652 bytes.
Last  part  : All data (429564 bytes) is Null(0x00)
Other parts : All data (429497 bytes) is Null(0x00)

If SEEK only(no data write before extent allocation), extent was not allocated upon CLOSE by test with Open Object Rexx on MS Win. At least one 0x00 was written at 4GB?  Or other data was written at over 4GB and truncated at 4GB? Special request/option is used by Tb or someone?
Update of Trash roughly consists of:
  - Append: Mail copy to Trash(Delete of mail at other folder)
  - Data update: Mail purge by retention policy
                 Delete(or Shift+Delete) of mail at Trash
                 Marking, Tagging etc.
  - Folder compaction: auto-compact or manual "Compact Folder(s)"
And, internal Rebuild-Index due to "corrupted or outdated msf condition" can interfere them.

To Jorge Vidal Wulff(bug opener) and Ekhart:  

What is set in "Retention Policy" of Trash?
Auto-compact is enabled?
> mail.prompt_purge_threshhold = true/false 
> mail.purge_threshhold = NNNN (threshold value in KB)
> mail.purge.ask = true/false
How big is your average Trash file size in daily use?
Do you use single-core(or CPU) PC? Or multi-core(or CPU) PC?
(In reply to comment #31)
> > To Ekhart(only one who can produce problem currently):
> It happens sometimes, but i haven't been able to reproduce it again since i
> opened bug 494706

Bug opener of Bug 468722 looks best tester of this 4GB problem. He could produce problem many times, and seems to be still able to produce problem frequently. :-)
(In reply to comment #15)
> I said I EMPTIED THE TRASH FOLDER, not deleted the file.
> After emptying the trash, my free space didn't increase, even if now the trash
> was 0 bytes in size (vs 4 GB a few moments before).

To Jorge Vidal Wulff(bug opener)

"0 bytes in size" is folder size at folder pane? Or file size displayed by "DIR Trash" or property of Trash file? Both?
I'm talking about the size of the file "Trash" in my TB profile, as shown in windows explorer.
I don't keep deleted messages, I have set it  to empty trash on exit and auto compact folders.
So as you see, the trash was either full of garbage or it wasn't actually taking up the 4GB, like a sparse file.
Adding "Trash", "full" "4GB" to bug summary for ease of search.
Summary: Trash folder reaches ridiculously big size populated with bogus data → Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=4GB or 4GB-1, while it is empty or small number of mails only)
I now experienced the 4GB problem with the junk file, but Tb was able to resolve the problem after i deleted a few emails in the junk folder and then compacted. Should i attach portions of the junk file? I have versions from before, during, and after the problem (created by automatic backup).

Tb's error message said to empty the junk folder - this is a very bad idea since people may not have time to first check the junk folder for important mail. Instead, Tb should first advise doing what i did (deleting a few messages and then compacting). Next, Tb should advise renaming the junk folder, not emptying it! I hope Tb at least doesn't ever advise the same nonsense for other folders!

If possible, someone please correct my comment #32 as follows:

> Not sure what you're trying to say, but 
> 1) I did not use shift+delete
> 2) I did not make a backup before i encountered the corruption

please delete "2) I did not make a backup before i encountered the corruption"

I must have been very tired when i wrote that nonsense - my automatic backup software did make a backup before the corruption, and that is the first attachment in bug 494706.


(In reply to comment #34)
> > 1) I did not use shift+delete
> 
> "Compact Folder" doesn't seem culprit, still I'm not sure though.

I can confirm that compact wasn't the problem at least in this new case. I had turned off automatic compact a little while ago because i got sick of having to wait for Tb all the time and because i was afraid of Tb again causing data corruption. The new 4 GB junk folder was created while i manually marked a few new emails as junk. However, i did notice that this happened after there had been more false negatives than usual, so perhaps Tb was already struggling with a growing or huge junk file. 


(In reply to comment #47)
> To Jorge Vidal Wulff(bug opener) and Ekhart:  
> 
> What is set in "Retention Policy" of Trash?
> Auto-compact is enabled?
> > mail.prompt_purge_threshhold = true/false 
> > mail.purge_threshhold = NNNN (threshold value in KB)
> > mail.purge.ask = true/false

mail.prompt_purge_threshhold = true (at the time of creation of 4 GB trash file) 
= false (at the time of creation of 4 GB junk file) 

mail.purge_threshhold = 100
mail.purge.ask = false

> How big is your average Trash file size in daily use?

usually less than about 300 kB, sometimes up to about 1000 kB

my junk file is huge - it has about 40 MB = about 8 000 emails

> Do you use single-core(or CPU) PC? Or multi-core(or CPU) PC?

Intel(R) Core(TM)2 Duo CPU     T5450  @ 1.66GHz, 1667 Mhz, 2 Core(s), 2 Logical Processor(s)
Now i got a 4 GB trash file after deleting a total of 180 junk messages = 1 100 kB. 

Interestingly, copying the messages by using Ctrl while dragging to new temp folder made the trash folder look empty (file remained same size). 

I will now turn off my antivirus temporarily to see if new 4 GB files are still created.
(In reply to comment #53)
> I now experienced the 4GB problem with the junk file,

As written in Bug 468722, problem looks to occur frequently on Trash & Junk.
Ekhart, if you encountered problem again, check items I wrote in Bug 468722 Comment #36, please.

> Should i attach portions of the junk file? I have versions from before, during, and after the problem (created by automatic backup).

No need to attach 4GB file. Check next instead please, to isolate problems after "4GB file creation" from problem of "4GB file creation" itself.
  1. Copy backup of 0x00's to Folder-X. Delete Folder-X.msf
  2. Display "Order Received" column and "Size" column.
  3. Rebuild-Index of Folder-X, and sort by "Order Received" or "Size".
  What is largest value of "Size"? What is largets value of "Order Received"?
  Mail of largest "Size" == Mail of largest "Order Received"?

> mail.purge.ask = false
Change it to true, and restart Tb. Even if auto-compact is enabled again, you can cancel compaction by "Cancel" reply.
> mail.purge_threshhold = 100
I recommend you to increase it, even if you usually use Tb with threshhold=false.

(Off-Topic, answers to your some questions)

If compaction is properly invoked, garbages of 0x00 is removed by compaction (not copied to new file), unless Rebuild-Index is invoked after creation of garbage of 0x00. See Bug 492344 for situation that compaction is not invoked when capmaction should be invoked.
I don't know about "after Rebuild-Index", because all 0x00 after last normal mail data is considered "mail data of the last mail" and it'll produce mail size which exceeds limit. I guess 32bits signed integer or smaller for mail size, instead of 32bits unsigned integer for mail size.

Ctrl+Drag/Shift+Drag etc. of mail/mail_folder has different meaning from Drag in some situations, and it depends on situation. (e.g. Shift+Drag of folder between accounts == Move folder)
(In addition to comment #55)

Ekhart, you don't need to do rebuild-index test with backup file. I checked it with next test.

(Phenomena once huge garbage of 0x00 is created by this bug)
  1. Put two 1KB-mails in a mail folder (say filename=FN)
  2. Open FN with "write / append" mode.
  3. Seek to offset=1GB-1, write 0x00 at 1GB-1, Close file
     => File size = 1GB, padded with 0x00.
        mail-1=1KB, mail-2=2GB-1KB(exceeds mail size limit of Tb)
  4. Delete FN.msf, restart TB => Rebuild Index of the 1GB file
  5. Only mail-1 is displayed at folder pane (huge mail-2 is lost)
  6. Shutdown & restart Tb, and click FN => Rebuild Index occurs
     i.e. MailDB data is corrupted due to too huge mail
  7. Click folder FN, Shift+Delete of remaining mail-1
     => Next dialog (Tb trunk 2009/6/01 build),
        i.e. something wrong in rebuild-index(or folder open)   
> Unable to delete messages in folder FN because it is in use by some other operation.
> Please wait for that operation to finish and then try again.
> [OK]
  8. Terminate Tb, rename FN/FN.msf to Trash/Trash.msf, restart Tb, click Trash
     => Rebuild-Index didn't occur.
        Shift+Delete of remaining mail-1 => Normally deleted
     Because folder cache of Trash folder is apparently invalid,
     MailDb data seems to be read from Trash.msf instead of folder cache.
     At step 6, folder cache of FN is probably used because no inconsistency.
     I guess it's the reason why rebuild-index occurs at every restart(step 6),
     and why subsequent Shift+Delete failure occurs.
  9. Empty Trash
     => File size of Trash becomes ZERO
        Free disk space increased by 1GB. i.e. 1GB is returned to free extent.

I guess issues around "rebuild-index of 4GB Trash file with gargage of 0x00" is reason why 4GB disk space was not returned to free disk extents in Comment #0.
To Ekhart:
Do you enable special feature such as "compression of disk drive"?
Do you use special software like "auto defrag", "on-the-fly-defrag"?
I now experienced the 4GB problem again with the trash file, and Tb was not able to resolve the problem after i deleted only a few emails in the trash folder and then tried to compact. Apparently Tb can only resolve the problem if the trash folder is emptied entirely, contrary to the same 4 GB problem with the junk folder.

I had automatic compact turned off (prompt turned on) and got an error message that IIRC spoke about not enough disk space, not about the trash folder being full. This seems to indicate the 4 GB trash file was created by simply deleting 7 emails. It's of course possible the 4 GB file was created during the last manual compacting earlier today, but i think this would have made Tb very sluggish, which i didn't notice.


(In reply to comment #57)
> To Ekhart:
> Do you enable special feature such as "compression of disk drive"?
> Do you use special software like "auto defrag", "on-the-fly-defrag"?

no, none of those
Noticed new very strange behavior:

Found file:///C:/Users/Ekhart/AppData/Local/Temp/Junk.htm open in Firefox although i hadn't touch that folder or that file. The only thing i had done that i can think of that triggered this was to manually mark as junk the email opened by Firefox. This is probably a different and equally serious bug, but it may be related.
Summary: Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=4GB or 4GB-1, while it is empty or small number of mails only) → Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=2**32-1 bytes, while it is empty or small number of mails only)
Accurate file size was: 4294967295 bytes (2**32-1). See Bug 494706 Comment #14.
doubt anyone's going to search on 2**32-1  :)
Summary: Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=2**32-1 bytes, while it is empty or small number of mails only) → Trash folder reaches ridiculously big size populated with bogus data (Trash file is full, Trash file_size=2**32-1 bytes, while it is empty or small number of mails only) (~4GB)
I can reproduce this every single time i delete between 60 and 100 emails when Memeo AutoBackup is running, and i haven't been able to reproduce it when it's not running. (Antivirus realtime protection has no effect.)

The interesting thing is that when i get the error message "The message could
not be moved or copied to folder 'Trash' because...", i can see in Windows
Explorer that the trash file has not been bloated yet (F5). It turns into 4 GB as soon as i click OK on the Tb error message. I attached such a non-bloated but "full" (corrupted?) trash file and its msf file to bug 494706 as attachment 382114 [details].
No longer blocks: 468722
No longer blocks: 489959
No longer blocks: 494706
Depends on: 494706
(In reply to comment #15)
> BTW, since using TB 3 beta, I've never seen this problem again.

Jorge Vidal Wulff(bug opener), if your problem is same as Bug 494706 Comment #35, I believe it can be explaned, because I couldn't reproduce phenomenon of Bug 494706 Comment #35 by test procedure of Bug 494706 Comment #35 with Tb trunk 2009/6/01 build.
I honestly don't know the exact steps needed to reproduce it.
What I DO know, however, is that this happened to me a few times and the only way to solve it was to "empty trash", not compacting. I emptied trash even though it was ALREADY EMPTY, but the file "trash" seemed to be about 4GB in size (which it wasn't actually, compared available disk space before and after, didn't change).
Anyway, after moving to TB 3 beta I haven't seen this problem anymore.
Jorge Vidal Wulff, do you use auto-backup software and run it for Tb's mail folder
files? Do you run "file scan by anti-virus software" like program periodically?
("Memeo AutoBackup" seems who interfered Tb's "file open with write" in Ekhart's case)
No, I don't use any backup software.
And no, I don't run my AV thru all the disk. AV Auto-protect (resident shield, resident protection,etc, call it what you want) is more than enough for me.
What anti-virus software do you use?
"resident shield" is transparent in theory. But if "resident shield" uses file handle which Tb uses and somehow sets offset pointer to ZERO, or if "resident shield" interferes Tb's "file open with write", problem may occur. (can be sais bug of "resident shield", if this is the case).
Do you use software which executes ad-hoc/on-the-fly defrag? It may interfere Tb's "file open with write" temporary.
If no external software who interferes Tb's copy is used in your case, "Compact folder(auto-compact)" becomes a suspect again...
As seen in Bug 497804(simultaneous compact of copy source folder case), "end of compact" can interfere copy operation, when end of compact(replaces folder file and .msf file). If similar situation occurs on "copy target folder", problem of Bug 494706 can be produced by Tb alone.
Answer from opener of Bug 468722 was same as yours (no specific external program).

Because "open with noshare" is a culprit as proven by bug 494706, "external software" is not limited to program executed under other than process of thunderbird.exe.
(a) Program runs under other than process of thunderbird.exe, and opens mail
    folder file with noshare: Example, auto-backup.
(b) Program other than Tb and runs under process of thunderbird.exe,
    and opens mail folder file with noshare: add-on, plugin, ...
    Example of this kind of add-on: SQL Manager for Firefox.
      Because Fx opens places.sqlite with noshare and never closes until
      shutdown of Fx, and because SQL Manager opens sql file with noshare,
      SQL Manager under Fx can't access active Fx's places.sqlite.
(c) Tb's module runs under process of thunderbird.exe, and opens mail folder
    file with noshare: I don't know this kind of modules.
And, because "compact folder" executes next at last step, "compact folder" and "file write(and read) by mail move/copy" can interfere each other, even though they never open mail folder file with noshare.
(d) delete XXX.msf & XXX, create XXX.msf, rename nstmp(-N) to XXX,
    and update XXX.msf(similar to rebuild-index) by "compact folder".

Jorge Vidal Wulff, do you use add-on/extension of Tb who tries to open/read/write Tb's mail folder file?
As written in bug 468722 comment #53, "who interfered" is irrelevant to problem.
So DUPing to bug 494706, although possibility of "your problem is not caused by bug 494706" is not ZERO(I can't say that your problem was absolutely same problem as bug 494706, based on data you provided.)

Jorge Vidal Wulff, re-open this bug if evidence of "different from bug 494706" will be found, please.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
Since I moved to TB 3 beta, this problem has gone away. I don't know what's happening in the 2.x branch right now...
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: