494706 - [1.8 branch only] Thunderbird creates 4 GB Trash file out of less than 200 kB of deleted mail (If data write to file for "target folder of mail move/copy" is temporary interfered by other software, Tb 2 generates file of file_size=4GB-1)

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Ekhart]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729)
Build Identifier: 2.0.0.21 (20090302) 

less than 200 kB Trash file turns into 4 GB file when user deletes a few kB more of normal mail (not spam, no attachments)

Reproducible: Sometimes

Steps to Reproduce:
1. delete email until you get the error message "Trash full...empty Trash and compact (or something like that)" etc., in my case this happened twice within a few days at little more than about 100/120 kB of email (15 to 16 emails)
2. 
3. 
Actual Results:  
less than 200 kB Trash file is turned into 4 GB file (see attachments showing same file before and after deleting a few more emails - restored using automatic backup program)

Expected Results:  
Trash size should be less than 200 kB 

This is probably related to the serious bugs causing data loss due to defective compacting and causing Thunderbird to freeze due to long/interminable compacting:  Bug 463359, Bug 468722, Bug 489959, Bug 493065

Comment 1

[Ekhart]

Attachment: Trash file before being bloated by TB

Comment 2

[Ekhart]

Attachment: first hundredth of first hundredth (made using Split Files) of almost same Trash file (with a few new messages) after being bloated by TB

most of the original 4GB Trash file is empty (at least in a text editor) so the first hundredth of the first hundredth of it (made using SplitFiles.exe) in the attachment is probably enough to see what Thunderbird produces in making the Trash file get 40 thousand times bigger within less than one minute (between two successive automatic backups)

Comment 3

[Ekhart]

In fact, Thunderbird did the same trick of making the
Trash file get 40 thousand times bigger within even less time (about 12 seconds between two successive automatic backups) a few days earlier. I have no idea how such a large file can even be produced so rapidly by Thunderbird since it takes several minutes to copy a file that size using Windows Explorer.

Comment 5

[Ekhart]

As you can check, the part of the 419 kB attachment that looks empty (899 - 41 = 858 pages with the same 15 emails that are in the 105 kB file) is completely filled with hidden code rendered as ÿÿÿÿÿÿÿ in Word. 

The same is probably true of the entire 4 GB file; at least it's true of those parts made using SplitFiles that i opened using Word. I was able to open the entire 4 GB file only using TheGun (Grown Up Notepad), but that only shows (about) 1000 x 10 000 pages of empty space after the 15 emails.

Comment 6

[Ekhart]

Attachment: last hundredth of last hundredth (made using Split Files) of almost same Trash file (with a few new messages) after being bloated by TB

Comment 7

[Ekhart]

Attachment: second to last hundredth of last hundredth (made using Split Files) of almost same Trash file (with a few new messages) after being bloated by TB

Comment 8

[Ekhart]

Attachment: third last hundredth of last hundredth (made using Split Files) of almost same Trash file (with a few new messages) after being bloated by TB

Comment 9

[WADA:World Anti-bad-Duping Agency]

Ekhart, there is no need to attach more 0x00 only files.  

> ------  ---------  -----------------------------------------------------
> ID      File size  Data(non-0x00 or 0x00)
> ------  ---------  -----------------------------------------------------
> 379477   107844    Non-Null=107844 bytes(mail data), No Null(0x00) data.
> 379479   429496    Non-Null=107844 bytes(mail data), 
>                    and consecutive Null(0x00,321652 bytes) follows.
> 380104   429564    All data(429564 bytes) is Null(0x00)
> 380107   429497    All data(429497 bytes) is Null(0x00)
> 380109   429497    All data(429497 bytes) is Null(0x00)
> ------  ---------  -----------------------------------------------------

Comment 10

[WADA:World Anti-bad-Duping Agency]

I think all data after normal mail data is 0x00.
Ekhart, can you check it by very simple script I made?
(1) Currently, I made script of Open Object REXX only.
    Can you install "Open Object REXX" to your PC?
(2) If no, I can convert REXX script to PHP script.
    PHP script(batch type) can be used by Download(zip file) and Unzip only.
    Can you add PHP modules to your PC?

Comment 11

[Ekhart]

I installed it.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Attachment: Rexx script to check null data length

1. Save attached script with extension of ".REX" (say C:\LIB\NULLCHK.REX)
2. Edit NULLCHK.REX, and put file name in variable fn.
3. At Command Prompt, C:\LIB\NULLCHK.REX.
When 4GB file, it may take 5 to 10 min.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Re-opening, because some works such as backup data analysis are required in this bug.

Comment 14

[Ekhart]

(In reply to comment #12)
> 2. Edit NULLCHK.REX, and put file name in variable fn.

Do i have to change anything except replace fn="C:\wada\TEST\Bug-org\bug-494706\trash-03.txt"; with fn="D:\test\trash"; underneath the text "Put full path of file here"?

Will the following lines create new folders?

fn_test="C:\wada\@@@\aaa.txt" ;
fn="C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\wxkq5msh.Gmail-IMAP\Mail\a.a.a\Big-Mbox.sbd\4MB-MAIL-T001024-M000001-0004MB-DT@1-N-RHM@0-Nolimit" ;
fn="C:\wada\TEST\Bug-org\bug-494706\trash-03.txt";

> 3. At Command Prompt, C:\LIB\NULLCHK.REX.
> When 4GB file, it may take 5 to 10 min.

Command prompt in cmd.exe or Try Rexx? And how do i get that in Try Rexx? All i have is a blank line.

When i enter D:\test\NULLCHK.REX in cmd.exe, i get:
C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trash
   6/07/09   8:41a  4294967295  A----
Number of Blocks=0
Logic Error-1 4294967295 0

When i enter D:\test\NULLCHK.REX in Try Rexx, i get:
D:\test\NULLCHK.REX
  Oooops ! ... try again.     Unexpected label
                              INTERPRET data must not contain labels; found "D"
  rc = 47 ................................... rexxtry.rex on WindowsNT

Comment 15

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #14)

In Rexx, "/*" is comment starter and "*/" is comment terminator.
So you need only to replace fn="C:\wada\TEST\Bug-org\bug-494706\trash-03.txt"; in non-comment part with fn="D:\test\trash";

> Command prompt in cmd.exe or Try Rexx?

Command prompt in cmd.exe.
If in Try Rexx, you need to type Rexx statement:
   Call "D:\test\NULLCHK.REX" ;  

> C:\Users\Ekhart>D:\test\NULLCHK.REX
>  D:\test\Trash
>    6/07/09   8:41a  4294967295  A----
> Number of Blocks=0
> Logic Error-1 4294967295 0

If someone opens the file(in write mode), "Read open" fails and total read data length becomes ZERO. My script doesn't do error checking, anf if total length=ZERO, "Logic Error-1" is displayed.
Some one opened the file you executed script?

>    6/07/09   8:41a  4294967295  A----

You said file size was 4294967296, didn't you?

Comment 16

[WADA:World Anti-bad-Duping Agency]

Oh, you said "4,194,304 kB". Both 4294967295 & 4294967296 is displayed as "4,194,304 kB" by MS Windows.

Comment 17

[Ekhart]

(In reply to comment #15)
> So you need only to replace fn="C:\wada\TEST\Bug-org\bug-494706\trash-03.txt";
> in non-comment part with fn="D:\test\trash";

Isn't that what i said i did?

> If someone opens the file(in write mode), "Read open" fails and total read data
> length becomes ZERO. My script doesn't do error checking, anf if total
> length=ZERO, "Logic Error-1" is displayed.
> Some one opened the file you executed script?

Not sure what you're saying. Is this what you mean?: "Did someone open the file before you executed the script?" Yes, i opened it with TheGUN, but i don't know what you mean with write mode. In any case, i also have a backup copy that i didn't touch and it produces the same result. I also have another 4 GB trash file and the 4GB junk file, both of which i didn't try to open. 

It seems that either Tb itself or my antivirus opened all these files and caused what you call "total read data length becomes ZERO" (which i don't understand). But why would an AV open a file in write mode?

Comment 18

[Ekhart]

what i meant to say was this:

(In reply to comment #15)
> So you need only to replace fn="C:\wada\TEST\Bug-org\bug-494706\trash-03.txt";
> in non-comment part with fn="D:\test\trash";

Isn't that what i said i did? Are you saying that what i did was right?

> If someone opens the file(in write mode), "Read open" fails and total read data
> length becomes ZERO. My script doesn't do error checking, anf if total
> length=ZERO, "Logic Error-1" is displayed.
> Some one opened the file you executed script?

Not sure what you're saying. Is this what you mean?: "Did someone open the file
before you executed the script?" Yes, i opened it with TheGUN, but i don't know
what you mean with write mode. 

In any case, i also have a backup copy that i didn't touch and it produces the same result. I also have another 4 GB trash file and the 4GB junk file, both of which i didn't try to open, and these also produce the same "Logic Error-1 4294967295 0" result.

It seems that either Tb itself or my antivirus opened all these files and
caused what you call "total read data length becomes ZERO" (which i don't
understand). But why would an AV open a file in write mode?

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #18)
> Yes, i opened it with TheGUN, (snip)
>(snip)
> I also have another 4 GB trash file and the 4GB junk file, both of which i didn't try to open, 
> and these also produce the same "Logic Error-1 4294967295 0" result.

Can you check next?
 (1) Run script with small text file (fn="D:\test\small.txt";)
 (2) Rename D:\test\trash to D:\test\trash.txt,
     and run script with fn="D:\test\trash.txt";

Comment 20

[WADA:World Anti-bad-Duping Agency]

> Are you saying that what i did was right?

Yes.
My script said next. It's similar to "DIR D:\test\Trash" result, so you were right.
>  D:\test\Trash
>    6/07/09   8:41a  4294967295  A----

Comment 21

[Ekhart]

(In reply to comment #19)
> Can you check next?
>  (1) Run script with small text file (fn="D:\test\small.txt";)

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\small.txt
   6/08/09   9:38a        1632  A----
Number of Blocks=1
Block=1 Non-Null=1632,Null=0

>  (2) Rename D:\test\trash to D:\test\trash.txt,
>      and run script with fn="D:\test\trash.txt";

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trash.txt
   5/23/09   1:27a  4294967295  A----
Number of Blocks=0
Logic Error-1 4294967295 0

(same result with automatically backed up and not manually opened version)

same result with older trash and new junk file:

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trashold.txt
   5/20/09  10:05a  4294967295  A----
Number of Blocks=0
Logic Error-1 4294967295 0

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Junk.txt
   6/05/09  10:12a  4294967295  A----
Number of Blocks=0
Logic Error-1 4294967295 0

Comment 22

[WADA:World Anti-bad-Duping Agency]

My script simply open file with "read mode", and reads each 1MB(rec=Charin(fn,,1MB); is read statement by binary mode).
4294967295 bytes is allocated, and DIR says file size=4294967295, but no record is returned to read? Or error occurs in open or read?
I'll create new version which rightly checks return code of open or Charin(read data statement).

Comment 23

[WADA:World Anti-bad-Duping Agency]

Can you see top part or last part of the 4GB-1 file by TheGUN?

Comment 24

[WADA:World Anti-bad-Duping Agency]

Attachment: Rexx script to check null data length (V 1.0.1)

I added return code check of Open/Close and added data display statements(Say ...;) for some important data.
Ekhart, check with new script, please.

Comment 25

[Ekhart]

(In reply to comment #24)
> Created an attachment (id=382101) [details]
> Rexx script to check null data length (V 1.0.1)
> 
> I added return code check of Open/Close and added data display statements(Say
> ...;) for some important data.
> Ekhart, check with new script, please.

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trash
   5/23/09   1:27a  4294967295  A----
File is opened(read mode).
  Data size to be read = 0
Total read bytes = 0 bytes
File is closed.
Number of Blocks=0
Logic Error-1? 4294967295 0

Comment 26

[Ekhart]

(In reply to comment #23)
> Can you see top part or last part of the 4GB-1 file by TheGUN?

Yes. The top part looks like a normal Tb mail folder, and the end is empty but "real", in other words you can use the scroll bar and the page down button to see many pages of empty text.

From - Wed May 20 09:56:04 2009
X-Account-Key: account2
X-UIDL: 45680396000067cf
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Received: via tmail-2006i.2007-sau for kw3661.200; Wed, 20 May 2009 06:57:02 +0300 (EEST)
(snip)

Comment 27

[Ekhart]

I've been able to reproduce the bug several times now, and it seems it only occurs when automatic backup and antivirus (only hard drive real-time protection, no email scanning) are running. This may be a coincidence though since the absence of the problem when they are not running is no proof that the problem doesn't sometimes occur when they are not running. I will continue testing with only automatic backup and only antivirus running.

Comment 28

[WADA:World Anti-bad-Duping Agency]

Attachment: Rexx script to check null data length (V 1.0.2, bypass bug of Chars() when file size>2GB)

It was bug of Chars(fn) of Open Object Rexx(Chars(fn), which returns not-read-yet bytes, returned 0, if file size>2GB). I tested with max 2GB file before I provide script to you. Sorry for my insufficient test.
I checked V 1.0.2 with file greater than 2GB, near 4GB, and it ran normally.

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #27)
> it seems it only occurs when automatic backup and antivirus
> (only hard drive real-time protection, no email scanning) are running.

What software/Which version is used for "automatic backup" and "hard drive real-time protection of antivirus"?

You can possibly hook "seek to 4294967295(==2**32-1==-1 of 32bits signed integer)" by Process Monitor for "path ends with Trash or Junk".
> http://technet.microsoft.com/en-us/sysinternals/bb545027.aspx
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

Comment 30

[Ekhart]

(In reply to comment #28)
> Created an attachment (id=382107) [details]

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\small.txt
   6/08/09  12:45p        1637  A----
File is opened(read mode).
  Returned value by Chars(fn) = 1637
Total read bytes = 1637 bytes
File is closed.
Number of Blocks=1
Block=1 Non-Null=1637,Null=0

C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trash.txt
   5/23/09   1:27a  4294967295  A----
File is opened(read mode).
  Returned value by Chars(fn) = 0
Processing 1 * 104857600 bytes
Processing 2 * 104857600 bytes
Processing 3 * 104857600 bytes
Processing 4 * 104857600 bytes
(snip: identical)
Processing 40 * 104857600 bytes
Total read bytes = 4294967295 bytes
File is closed.
Number of Blocks=1
Block=1 Non-Null=107844,Null=4294859451


C:\Users\Ekhart>D:\test\NULLCHK.REX
 D:\test\Trash
   5/23/09   1:27a  4294967295  A----
File is opened(read mode).
  Returned value by Chars(fn) = 0
Processing 1 * 104857600 bytes
Processing 2 * 104857600 bytes
Processing 3 * 104857600 bytes
(snip: identical)
Processing 38 * 104857600 bytes
Processing 39 * 104857600 bytes
Processing 40 * 104857600 bytes
Total read bytes = 4294967295 bytes
File is closed.
Number of Blocks=1
Block=1 Non-Null=107844,Null=4294859451

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #30)
> Number of Blocks=1
> Block=1 Non-Null=107844,Null=4294859451

107844 bytes of continuous non-0x00(mail data you attached to Comment #1), followed by continuous 4294859451 bytes of 0x00.

You can probably see phenomenon I wrote in Bug 482486 Comment #56, phenomenon on Tb once 4GB-1 file is generated by Tb himself or someone.
(1) Copy 107844 bytes file(mail data) to Folder-A, delete Folder-A.msf.
(2) Copy the 4GB-1 file with garbage of 0x00 to Folder-B, delete Folder-B.msf.
(3) Restart Tb, click Folder-A, Folder-B => Rebuild-Index is invoked.
(4) Display "Size" and "Order Received" columns, sort by "Order Received".
I think mail with largest "Order Received" column in Folder-A is lost in Folder-B.
(Size of last mail==mail data size + 4294859451 of 0x00 for Rebuild-Index.)
Is it right?

Comment 32

[Ekhart]

(In reply to comment #27)
> I've been able to reproduce the bug several times now, and it seems it only
> occurs when automatic backup and antivirus (only hard drive real-time
> protection, no email scanning) are running. This may be a coincidence though
> since the absence of the problem when they are not running is no proof that the
> problem doesn't sometimes occur when they are not running. I will continue
> testing with only automatic backup and only antivirus running.

OK, this is no coincidence. It happens every single time i delete between 60 and 100 emails when Memeo AutoBackup is running, and i haven't been able to reproduce it when it's not running. (Antivirus realtime protection has no effect.)

The interesting thing is that when i get the error message "The message could not be moved or copied to folder 'Trash' because..." the trash file in Windows Explorer has not been bloated yet. It turns into 4 GB as soon as i click OK on the Tb error message. I'll attach a non-bloated trash file and its msf file.

Comment 33

[Ekhart]

Attachment: trash file into which Tb cannot write (right before being bloated)

Comment 34

[Ekhart]

Attachment: and its msf file

In other words, Tb considers this trash file (and its msf file) corrupted because it cannot write into it. This seems to me to be the main or at least first problem. The bloating occurs afterwards. 

Next should probably be tested whether the first or second problem or both are caused by the backup software or compatibility problems with Tb. Perhaps the backup software locks the file and prevents Tb from accessing it.

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #32)
> The interesting thing is that when i get the error message "The message could
> not be moved or copied to folder 'Trash' because..." the trash file in Windows
> Explorer has not been bloated yet. It turns into 4 GB as soon as i click OK on
> the Tb error message.

Problem has been reproduced with Tb 2.0.0.21 on Win-XP, by following simple test.
(1) Trash has some mails. Restart Tb, open Inbox (don't touch Trash).
(2) Open file of Trash by text editor. (write open. I used Sakura Editor)
(3) Delete mails in Inbox
(4) Dialog of "The message could not be moved ..."
(5) Close text editor.
(6) Reply OK to the dialog => Trash file size of 4GB-1 
      File size    : 4,294,967,295 bytes
      Size on disk : 4,294,967,296 bytes
(I usually close text editor window after reply of OK in test like above, I couldn't be aware such phenomenon.)

Comment 36

[WADA:World Anti-bad-Duping Agency]

Confirming.

Comment 37

[WADA:World Anti-bad-Duping Agency]

Above phenomenon could be observed with any of Delete(move target=Trash), Move(move target=any folder), Copy(copy target=any folder).

Comment 38

[WADA:World Anti-bad-Duping Agency]

Tb trunk(2009/6/01 build) issued following dialog at step (4), and problem didn't occur.
> The operation failed because an other operation is using the folder.
> Please wait for that operation to finish and then try again.
>    [ OK ]
Error check of "file open in write mode" seems to be executed at earlier stage than Tb 2. (Tb trunk doesn't step into write operation if open fails at first stage)

Comment 39

[David :Bienvenu]

WADA, so TB 3.0 b3Pre builds don't have this problem at all?

Comment 40

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #39)
> WADA, so TB 3.0 b3Pre builds don't have this problem at all?

I'm not sure. I can say "I can't reproduce phenomenon of Comment #35 by test procedure of Comment #35 using Tb 3.0 b3Pre build" only.

Phenomenon of Comment #35 is reproduced with Seamonkey 1.1.15 on MS Win-XP SP3.
Changing to MailNews Core.

Comment 41

[WADA:World Anti-bad-Duping Agency]

Regression range:
> -------------------------------------------------------------------------------
> WORKSFORME : 2008-10-17-03-mozilla1.8 (thunderbird-2.0.0.18pre.en-US.win32.zip)
> -------------------------------------------------------------------------------
> CONFIRMED  : 2008-10-18-03-mozilla1.8 (thunderbird-2.0.0.18pre.en-US.win32.zip)
> -------------------------------------------------------------------------------

Comment 42

[WADA:World Anti-bad-Duping Agency]

Changes in Tb 2.0.0.18.
> http://www.rumblingedge.com/2008/11/22/thunderbird-20018-released/
Next two non-security fixes were shipped by Tb 2.0.0.18.
> Bug 450359 nsFileSpec::Truncate() expects PRInt32 as the first argument so mail may disappear.
> Bug 450991 Number in "Order Received" column becomes negative, when mail’s offset in local mail folder file exceeds 2GB
Mismatch between changes in nsFileSpec::Truncate() and some module of Tb?

Comment 43

[Magnus Melin [:mkmelin]]

Checkins in that range:
<http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=ThunderbirdTinderbox&branch=MOZILLA_1_8_BRANCH&branchtype=match&dir=&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2008-10-17+03%3A00%3A00&maxdate=2008-10-18+03%3A00%3A00&cvsroot=%2Fcvsroot>

Regression from bug 450359 then.

Comment 44

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log for ...\Trash & ...\Trash.msf of test procedure of Comment #35 (.CSV file)

Tested with Tb 2.0.0.21.
File name : C:\wada\@@@\Mail\Trash, C:\wada\@@@\Mail\Trash.msf

Comment 45

[WADA:World Anti-bad-Duping Agency]

Next log looks to correspond to "SEEK to 4GB-1" and "allocation of 4GB-1".
> "186664","14:32:48.7856310","thunderbird.exe","1452",
>          "IRP_MJ_SET_INFORMATION","C:\wada\@@@\Mail\Trash","SUCCESS",
>          "Type: SetEndOfFileInformationFile, EndOfFile: 4,294,967,295"
> "186667","14:32:48.8043817","thunderbird.exe","1452",
>          "IRP_MJ_SET_INFORMATION","C:\wada\@@@\Mail\Trash","SUCCESS",
>          "Type: SetAllInformationFile, AllocationSize: 4,294,967,295"

Comment 46

[Hiroyuki Ikezoe (:hiro)]

Thank you for the notice.

I suspect SetFilePointer does not return GetLastError() correctry at 
http://mxr.mozilla.org/mozilla1.8/source/xpcom/obsolete/nsFileSpecWin.cpp#547

But anyway, SetFilePointerEx might be better than SetFilePointer when handling 64bit value.

I will attach a patch soon.

Comment 47

[Hiroyuki Ikezoe (:hiro)]

Attachment: A patch

Use SetFilePointerEx instead of SetFilePointer.

WADA san, could you please test with this patch? I do not have the machine building Thunderbird 2.x now.

Comment 48

[WADA:World Anti-bad-Duping Agency]

> WADA san, could you please test with this patch?

I have no environment to build by myself. If build becomes available, I'll check it ASAP.

Comment 49

[Hiroyuki Ikezoe (:hiro)]

> 
> Problem has been reproduced with Tb 2.0.0.21 on Win-XP, by following simple
> test.
> (1) Trash has some mails. Restart Tb, open Inbox (don't touch Trash).
> (2) Open file of Trash by text editor. (write open. I used Sakura Editor)
> (3) Delete mails in Inbox
> (4) Dialog of "The message could not be moved ..."
> (5) Close text editor.
> (6) Reply OK to the dialog => Trash file size of 4GB-1 
>       File size    : 4,294,967,295 bytes
>       Size on disk : 4,294,967,296 bytes

I could not reproduce this issue with Thunderbird 2.0.0.21 on WinXP. I could see the error message but trash file size was not changed.

Comment 50

[Hiroyuki Ikezoe (:hiro)]

OK, I am getting closer to the reason of this issue.

PR_Seek64 returns -1 if the operation fails, but we do not handle this error code. 
http://mxr.mozilla.org/mozilla1.8/source/xpcom/obsolete/nsIFileStream.cpp#504

Comment 51

[Masatoshi Kimura [:emk]]

> +    if (SetFilePointerEx(hFile, li, &retLi, FILE_BEGIN) == 0 ||
> +        li.QuadPart != retLi.QuadPart)
> +    {
>          goto error;
SetFilePointerEx() will return nonzero if the function succeeds (NOT fails).

Comment 52

[Masatoshi Kimura [:emk]]

Sorry, I missed " == 0". I think "!SetFilePointerEx(...)" is better anyway.

Comment 53

[Hiroyuki Ikezoe (:hiro)]

Attachment: WIP v1

This patch is not completed yet.

TODO

* Return error code from nsMailDatabase::UpdateFolderFlag if tell() fails.
* Need post-cleanup if tell() fails in nsMsgLocalMailFolder::ChangeKey.

Comment 54

[Hiroyuki Ikezoe (:hiro)]

Attachment: WIP v2

> * Return error code from nsMailDatabase::UpdateFolderFlag if tell() fails.
> * Need post-cleanup if tell() fails in nsMsgLocalMailFolder::ChangeKey.

Fixed these and which I missed in the previous patch.
I hope this patch resolve this issue. 

Can anyone who is able to reproduce this issue test this patch?

Comment 55

[Hiroyuki Ikezoe (:hiro)]

Attachment: Revised

Return original error instead of NS_ERROR_FAILURE.

I hope this is the final one.

Comment 56

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #49)
> I could not reproduce this issue with Thunderbird 2.0.0.21 on WinXP.
> I could see the error message but trash file size was not changed.

Do you use NTFS? FAT32? "Sparse file" is possibly supported by NTFS only.

Comment 57

[Hiroyuki Ikezoe (:hiro)]

NTFS.

Comment 58

[WADA:World Anti-bad-Duping Agency]

File size of Trash before delete test was ZERO?

Comment 59

[Hiroyuki Ikezoe (:hiro)]

Attachment: Reveised patch

Oops! The previous patch did not return -1 in FileImpl::Tell if PR_Seek64 fails.

And handle some error cases in nsMsgDBFolder.cpp.

Comment 60

[Hiroyuki Ikezoe (:hiro)]

(In reply to comment #58)
> File size of Trash before delete test was ZERO?

No. There are couple of deleted mails.

Comment 61

[WADA:World Anti-bad-Duping Agency]

Why problem is not reproduced by Ikezoe-san? SP of MS Win is relevant?(I use XP SP3). Ekhart(bug opener), can you reproduce problem by procedure of Comment #35(which is emulation of your test with auto-backup software)? I think TheGUN can be used as who interferes Tb's file open of Trash.

Comment 62

[Hiroyuki Ikezoe (:hiro)]

Attachment: Revised patch

I found true reason.

http://mxr.mozilla.org/mozilla1.8/source/mailnews/local/src/nsLocalMailFolder.cpp#123

nsMsgKey should be int64 but it is very big deal. IMHO, it should be another bug.

Comment 63

[Hiroyuki Ikezoe (:hiro)]

(In reply to comment #61)
> Why problem is not reproduced by Ikezoe-san? SP of MS Win is relevant?(I use XP
> SP3). Ekhart(bug opener), can you reproduce problem by procedure of Comment
> #35(which is emulation of your test with auto-backup software)? I think TheGUN
> can be used as who interferes Tb's file open of Trash.

I can reproduce now. Free disk space was less than 4GB....

Comment 64

[WADA:World Anti-bad-Duping Agency]

> I can reproduce now. Free disk space was less than 4GB....
I thought "Sparse file" doesn't allocate extents, but it seems wrong when Tb's request on MS Win... Ekhart, you don't need to test Comment #35 any more.

Comment 65

[WADA:World Anti-bad-Duping Agency]

Next is a part of code in nsMsgLocalMailFolder::CopyFileMessage.

> (Tb 2.0.0.21)
> http://mxr.mozilla.org/mozilla1.8/source/mailnews/local/src/nsLocalMailFolder.cpp#2216
> 2216   rv = InitCopyState(fileSupport, messages, msgToReplace ? PR_TRUE:PR_FALSE,
> 2217                      listener, msgWindow, PR_FALSE, PR_FALSE);
> 2218   if (NS_FAILED(rv)) goto done;
> And, BeginCopy, CopyData, EndCopy are called unless NS_FAILED(rv)==true.

> (Tb Trunk)
> http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsLocalMailFolder.cpp#2153
2153   rv = InitCopyState(fileSupport, messages, msgToReplace ? PR_TRUE : PR_FALSE,
> 2154                      listener, msgWindow, PR_FALSE, PR_FALSE);
> 2155   if (NS_SUCCEEDED(rv))
> 2156   {
>           BeginCopy, CopyData, EndCopy is called at here.  
> 2190   }

I think problems are;
(1) Tb 2 executes copy operation even if write open failed, and tries to backout
    of written data(but not really written), and tries to trunk() at current-1
    even though current==ZERO due to file open failure.
    And, because external software closed file at this point, trunk() at
    current-1(==ZERO-1==2**32-1 if 32bits unsigned binary) is executed.  
(2-1) Till Tb 2.0.0.17 : Bucause 32bits signed binary was used by Trunk(),
      -1 produced error, and no problem occurred.
(2-2) Since Tb 2.0.0.18 : Bucause 32bits unsigned binary is used by Trunk(),
      current-1 generates 2**32-1, and it won't produce error.
(2-3) Current patch : Uses 64bits signed binary for offset,size etc.,
      then offset/size value of -1 due to (1) won't produce 4GB-1 sparse file.
Ikezoe-san, is it right?

Comment 66

[Hiroyuki Ikezoe (:hiro)]

(In reply to comment #65)

> I think problems are;
> (1) Tb 2 executes copy operation even if write open failed, and tries to
> backout
>     of written data(but not really written), and tries to trunk() at current-1
>     even though current==ZERO due to file open failure.
>     And, because external software closed file at this point, trunk() at
>     current-1(==ZERO-1==2**32-1 if 32bits unsigned binary) is executed.  
> (2-1) Till Tb 2.0.0.17 : Bucause 32bits signed binary was used by Trunk(),
>       -1 produced error, and no problem occurred.
> (2-2) Since Tb 2.0.0.18 : Bucause 32bits unsigned binary is used by Trunk(),
>       current-1 generates 2**32-1, and it won't produce error.
> (2-3) Current patch : Uses 64bits signed binary for offset,size etc.,
>       then offset/size value of -1 due to (1) won't produce 4GB-1 sparse file.
> Ikezoe-san, is it right?

Partially yes, partially no.

* TB remembers file size before writing data <http://mxr.mozilla.org/mozilla1.8/source/mailnews/local/src/nsLocalMailFolder.cpp#2321> and truncate the size if writing fails <http://mxr.mozilla.org/mozilla1.8/source/mailnews/local/src/nsLocalMailFolder.cpp#2571>. 
* the file size is 0xffffffff by default.
As you can see the code, TB does not check whether the file size is correct or not.  So If the file size is not correct (i.e. tell() returns -1), the file become 4GB.

Comment 69

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #66)
> * the file size is 0xffffffff by default.
> TB does not check whether the file size is correct or not.
> So If the file size is not correct (i.e. tell() returns -1), the file become 4GB.

Aha!
I think main culprit of murder is (1) in my Comment #65, and (2-2) + "no check of file size" is an accessory to the murder. What do you think?

Comment 70

[Hiroyuki Ikezoe (:hiro)]

(In reply to comment #69)
> (In reply to comment #66)
> > * the file size is 0xffffffff by default.
> > TB does not check whether the file size is correct or not.
> > So If the file size is not correct (i.e. tell() returns -1), the file become 4GB.
> 
> Aha!
> I think main culprit of murder is (1) in my Comment #65, and (2-2) + "no check
> of file size" is an accessory to the murder. What do you think?

I do not think (1) is true.

Comment 71

[Hiroyuki Ikezoe (:hiro)]

Comment on attachment 382697 [details] [diff] [review]
Revised patch

>Index: mailnews/local/src/nsLocalMailFolder.cpp
>===================================================================
>RCS file: /cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v
>retrieving revision 1.506.2.27
>diff -u -8 -p -r1.506.2.27 nsLocalMailFolder.cpp
>--- mailnews/local/src/nsLocalMailFolder.cpp	17 Oct 2008 19:16:36 -0000	1.506.2.27
>+++ mailnews/local/src/nsLocalMailFolder.cpp	11 Jun 2009 11:32:47 -0000
>@@ -2562,17 +2566,17 @@ NS_IMETHODIMP nsMsgLocalMailFolder::EndC
>   if (!copySucceeded || mCopyState->m_writeFailed)
>   {
>     if (mCopyState->m_fileStream)
>       mCopyState->m_fileStream->close();
>     
>     nsCOMPtr <nsIFileSpec> pathSpec;
>     rv = GetPath(getter_AddRefs(pathSpec));
>     
>-    if (NS_SUCCEEDED(rv) && pathSpec)
>+    if (NS_SUCCEEDED(rv) && pathSpec && mCopyState->m_curDstKey != 0xffffffff)
>       pathSpec->Truncate(mCopyState->m_curDstKey);

I confirmed that this part alone fixes the issue in comment#35. But for the safety, other part needs, I think

Comment 72

[David :Bienvenu]

in general, for a security fix release patch, the smaller the patch, the better, since we don't have a lot of resources for testing for regressions. Whatever parts are applicable for the trunk are more suitable candidates for tb 3.0...

Comment 73

[Hiroyuki Ikezoe (:hiro)]

Attachment: Fix

Comment 74

[David :Bienvenu]

Comment on attachment 383819 [details] [diff] [review]
Fix

a comment in the code would be nice, but this should help people w/ this problem; thx for working on it!

Comment 75

[Magnus Melin [:mkmelin]]

Comment on attachment 383819 [details] [diff] [review]
Fix

Was this change good for an sr too?

Would be nice to get into the next tb security release.

Comment 76

[David :Bienvenu]

Comment on attachment 383819 [details] [diff] [review]
Fix

yes, I agree.

Comment 77

[Magnus Melin [:mkmelin]]

Comment on attachment 383819 [details] [diff] [review]
Fix

Small well contained patch to fix an apparently fairly wide spread problem.

Comment 78

[Ekhart]

Seems to be fixed in beta 2 (3.0b2). I'm having trouble testing it because i can't find info on how to turn of the indexing function. This creates a new .wdseml file for every deleted email, so the backup software is busy most of the time dealing with those (and doesn't interfere with the deletion process often enough to enable testing well).

Once in a while i can create a conflict and then Tb deals with it elegantly by either 
1) pausing the deletion process (but apparently remembering how many times i pressed the delete button) or 
2) producing this error message with only minor English problems:

Alert
The operation failed because an other operation is using the folder. Please wait for that operation to finish and then try again. 

an other
=> 
another

finish and 
=>
finish, and 

Is it necessary to open a new bug, or is this bug or http://forums.mozillazine.org/viewforum.php?f=29 OK to report these English errors?

Comment 79

[Ekhart]

(In reply to comment #77)
> (From update of attachment 383819 [details] [diff] [review])
> Small well contained patch to fix an apparently fairly wide spread problem.

How does one apply this patch to the non-beta version?

Comment 80

[Magnus Melin [:mkmelin]]

You can check out a copy from cvs, and apply the patch the usual way. (See https://developer.mozilla.org/En/Creating_a_patch)
But if you've never done it before, wait for branch nightlies where it's applied.

Dan, are you handling approval1.8.1.next requests?

Comment 81

[Dan Mosedale (:dmosedale, :dmose)]

In general, ss and dveditz do most of that.  They typically do a sweep as the branch is getting close to shipping and ping bienvenu, Standard, or me for stuff they'd like assistance with.  In this case, I suspect bienvenu is the most suitable approver here, since he has his head around this best.

Comment 88

[Magnus Melin [:mkmelin]]

dveditz/bienvenu: ping on the approval

Comment 90

[Daniel Veditz [:dveditz]]

Comment on attachment 383819 [details] [diff] [review]
Fix

Approved for 1.8.1.24, a=dveditz for release-drivers

Comment 91

[Magnus Melin [:mkmelin]]

MOZILLA_1_8_BRANCH
Checking in mozilla/mailnews/local/src/nsLocalMailFolder.cpp;
/cvsroot/mozilla/mailnews/local/src/nsLocalMailFolder.cpp,v  <--  nsLocalMailFolder.cpp
new revision: 1.506.2.28; previous revision: 1.506.2.27
done

->FIXED