65303 - new folder created with same name as (and folder renamed to) previously renamed folder contains phantom mail - crashes when read

Status: VERIFIED FIXED

(MailNews Core :: Database, defect)

Description

[stevehodge99]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; 0.7) Gecko/20010109
BuildID:    2001010901

A folder created with the same name an exisitng folder had (before it was
renamed) contains the list of mail that is in the old folder. Trying to read any
of them causes a crash. Filters deliver to the old folder and not the new one.
After restart everything is as expected - mail is in the old folder, new one is
empty. Can work around by restarting Mozilla (maybe even only mail).

Reproducible: Always
Steps to Reproduce:
1. Create a folder in an account and move some mail there.
2. Rename the folder.
3. Create a new folder with the same name as the first folder had before the rename.
4. Try reading some of the mail that is in the new folder.

Actual Results:  Crashes when reading the phantom mail in the new folder

Expected Results:  New folder should be empty, any filter set to deliver to that
folder name should deliver to the new folder, not the renamed one.

I was using a POP account.

Comment 1

[Keyser Sose]

Confirmed
Platform: PC
OS: Windows 98
Mozilla Build: 2001011608

This also happens when you delete one account and then create a new account with
the same user/server you see the old messages etc. Marking NEW.

Comment 2

[(not reading, please use [e-mail] instead)]

re-assign to naving.

Comment 3

[Navin Gupta]

i am able to reproduce this one. adding nsbeta1

Comment 4

[Navin Gupta]

This one is again a leak of rdf resource but the good thing now is that 
it does not crash when you select the message.  

Comment 5

[James Paige]

This one still crashes for me in trunk build 2001101503 on Windows 95
talkback ID TB36785609K

Comment 6

[Yuri]

I run into what seems to be a related problem, which might be even more severe.

Create two folders: A and B. Put message X in A, and message Y in B. 
Rename A to C.  You now have folder C with message X and folder B with message Y.
Renamed B to A.  You now have too folders (A and C) with identical content X. 
Everything that was in folder B is now gone!

I am using build 2002052306 on WinNT.

Comment 7

[Henry Jia]

I can reproduce this bug on linux. No crash now, but with data lost. Just like
comment #6.

Comment 8

[Sheela Ravindran]

Reporter,
Do you still crash with the steps you have initially reported? Can you please 
try this in a new build and comment.

Yuri,
Can you please log a seperate bug? If not let me know I can log one and cc you 
on it.  Because this bug talks about the crash and filter not getting reset 
to the new folder. And your test case scenario results in a dataloss.  We don't 
want to combine two different scenarios into one bug.  

Comment 9

[stevehodge99]

It's still present in 1.1b, but no longer results in a crash. The new folder has
the same message list as the old folder but the contents of those message don't
display - the messages are all blank. I did not test the filtering.

I also had trouble deleting both the old and new folders afterwards. Neither
folder would go to the trash if I pressed the delete key or used the 'delete
folder' context menu item. I could drag one folder into the trash, but if I
tried to drag the second one in it said that 'a folder of that name already exists'.

Comment 10

[Arnaud Ladrière]

HEEELP

I encountered the behavior listed in comment 6, with 1.2b (build 2002101612) on
windows 2000, with a "real life" folder, and I lost several hundreds of
messages, as data loss is involved, I think that THIS BUG IS CRITICAL !!!!!!

Comment 11

[WADA:World Anti-bad-Duping Agency]

I think this bug is critical too.
Are there any way to clear folder cache manualy after rename?

If this bug will not be resolved until release of 1.2, this problem should be
described on release note.

Comment 12

[WADA:World Anti-bad-Duping Agency]

Deletion of panacea.dat in profile deirectory possibly reduces probability of
this folder rename related problem.
Deletion of panacea.dat clears garbage of old folder names.
However, it is not effective on reuse of old folder name without restarting Mozilla.

Comment 13

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 14

[Koike Kazuhiko]

Attachment: patch v1

Use nsFileSpec instead of nsIFileSpec.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Confirmed with Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7a)
Gecko/20040219

The attached patch is waiting for a review from Seth since 9 month. It seems
that's a very simple patch. Why it doesn't get a review?

Comment 16

[Tilman Schmidt]

Confirmed with Mozilla/5.0 (Windows; U; Windows NT 5.0; de-AT; rv:1.7)
Gecko/20040616. Deleting panacea.dat did *not* clear the problem.

Comment 17

[:aceman]

That may be because referrences to old folders are still kept in the file - bug
231941.

Comment 18

[:aceman]

Ah, you just said the opposite. Ignore my comment. But can you tell us at which
moment in the process did you delete the file? Was Mozilla running or not?

Comment 19

[Tilman Schmidt]

I did of course close all Mozilla windows *and* verify that the Mozilla process
was gone in Windows Task Manager (call me a burnt child) before I deleted
panacea.dat.

Comment 20

[:aceman]

You deleted the folder, closed Moz, deleted panacea.dat, started Moz, and when
you created a folder with the same name, the problem appeared - you see
messages, that shouldn't be there? Now, that would be weird :)

Comment 7, 10: are you sure the messages were lost? Didn't it only appear so?
Have you looked into the files containing the mail on disk? If files A, B anc C
are still there, deleting their .msf files could help.

The attached patch is really too simple. It changes some on-disk behaviour of
saving something. I think this problem is related with the memory representation
of the folders, that deleted/moved folders still influence their old location.
They are still there as phantoms and if a new folder is moved into the same
name, it gets the properties (and REFERENCES to messages, I really doubt that
their content too) of the phantom folder.

Comment 21

[Tilman Schmidt]

(In reply to comment #20)
> You deleted the folder, closed Moz, deleted panacea.dat, started Moz, and when
> you created a folder with the same name, the problem appeared - you see
> messages, that shouldn't be there? Now, that would be weird :)

Ah, sorry, I only answered the second part of your question.

No, I closed Moz, deleted panacea.dat, started Moz, then renamed an existing
folder with messages in it and created a new folder with the previous name of
the renamed folder.

Comment 22

[:aceman]

Ok, so try my scenario. What you did can't have any effect.

Comment 23

[Víctor Fernández]

This bug still happens on Mozilla 1.7.3. I can confirm it.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This bug isn't reproducable with a current build of Thunderbird (version 0.8+
(20041030)). Build from 20041024 still shows this behavior. 

The checkin for bug 264467 fixes this problem. Builds since 20041025 shouldn't
show this problem.

=> FIXED and removed blocking flag because we don't need it anymore.

Comment 25

[Jungshik Shin]

I don't see how the patch for bug 264467 could possibly fix this bug. Even if
it's fixed, we don't know how it's fixed so that this can't be resolved as 'fixed'.

Comment 26

[Jungshik Shin]

For now, resolve as WFM per comment #24, but should be reopened if necessary.
(I'm seeing this with my patch for bug 264071)

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sorry, i've linked against the wrong bug. Thank you for correction.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #26)
> For now, resolve as WFM per comment #24, but should be reopened if necessary.
> (I'm seeing this with my patch for bug 264071)

Really?
Phantom mail problem still exists on 2004102906-trunk/Win-2K,
although crash did not occur on first read.
(Your patch V1 on 10-27 for bug 264071 is applied but patch V2 on 10/29 is not
applied yet.)

Please try following very very simple test case.
 (1) Create a folder named "Test"
 (2) Create a subfolder of "A" under "Test"
 (3) Copy a mail to "A"
 (4) Rename "A" to "B"
 (5) Create a subfolder of "A" again under "Test"

 => Mail count of "A" becomes 1 instead of 0.
 => DIR says;
    2004/10/31  00:14                    0 A
    2004/10/31  00:14                  584 B
    2004/10/31  00:26                1,803 B.msf
 => "Copy Folder Location" says ;
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/B
I do not know what will happen when write/move mails to/from this currupted "A"
or "B".

Jshin, reopen this bug, please.
(I have no privilege to reopen)

Comment 29

[Jungshik Shin]

I did NOT mean that my patch for bug 264071 would fix this bug. (note that I
wrote I was seeing this bug even with my patch applied to my own build.) I just
resolved it 'WFM' **per** comment #24 because resolving it as 'FIXED' is wrong
when *nobody* knows how it's fixed if it's ever fixed. 

Reopening per comment #28

 

Comment 30

[WADA:World Anti-bad-Duping Agency]

Sorry for my misundestanding.
  - I misunderstood that "WFM" also means "Works for you, Jshin".
Jshin, thanks for re-opening.

I set blocking-aviary1.0? flag again because it was cleared.
I prefer "Thunderbird 1.0 with this bug fixed" to "Thunderbird 1.0 describing
this bug in Known Bugs section of Release Notes".  

Comment 31

[WADA:World Anti-bad-Duping Agency]

This is test result on 2004110406-trunk/Win-2K.
This is cyclicc renaming test - "a"->"b"->"c"->"a"->"b"...
In second rename cycle, strange behaviour was observed.
   - Different(previously used) file name was used from folder name
     when renamed to previously used folder name.
If this strange folder is created, mail will be lost.
Renaming to just previous name(b->a,c->b,a->c) failed in this test.
If this strange folder is moved, mail will be lost too.

(0) Preparation
   (0-1) Shutdown Mozilla(all of Browser and Mail&News
   (0-2) Delete panacea.dat from profile directry to avoid affect of this bug.
   (0-3) Start Mozilla Mail&News
   (0-4) Define folder of "Test"
   (0-5) Define subfolders of "X1", "X2", ... "Xn" under "Test" folder
   (0-6) Shutdown Mozilla(all of Browser and Mail&News)
   (0-7) Delete panacea.dat from profile directry to avoid affect of this bug.
   (0-8) Start Mozilla Mail&News

(1) Create folder "a" under "X1" under "Test"
    "Copy Folder Location" says; 
      mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/a
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:04                1,553 a.msf

    Subject Data in file of "a" : Subject: Test-1

(2) Rename folder "a" to "b"
    Mail count of folder "b" = 1
    Subject in thread pane   = Test-1

    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/b
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 b
      2004/11/05  04:04                1,651 b.msf

    Subject Data in file of "b" : Subject: Test-1

(3) Rename folder "b" to "c"
    Mail count of folder "c" = 1
    Subject in thread pane   = Test-1

    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/c
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 c
      2004/11/05  04:12                1,695 c.msf

    Subject Data in file of "c" : Subject: Test-1

(4) Rename folder "c" to "a"
    Mail count of folder "c" = blank(=0) => Mail lost!
    Subject in thread pane   = No mail is displayed
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/b
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:14                1,695 a.msf
      2004/11/05  04:14                    0 b
      2004/11/05  04:14                1,104 b.msf
    
(5) Copya a mail(Subject="Test-2") to folder "a" 
    Mail count of folder "a" = 1
    Subject in thread pane   = Test-2

    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/b
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:14                1,695 a.msf
      2004/11/05  04:18                  611 b
      2004/11/05  04:18                1,665 b.msf

    Subject Data in file of "a" : Subject: Test-1
    Subject Data in file of "b" : Subject: Test-2

    => Folder name is "a" but file of b/b.msf is used
    
(4) Rename folder "a" to "b"
    Mail count of folder "b" = blank(=0) => Mail lost!
    Subject in thread pane   = No mail is displayed

    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/c
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:14                1,695 a.msf
      2004/11/05  04:18                  611 b
      2004/11/05  04:18                1,665 b.msf
      2004/11/05  04:25                    0 c
      2004/11/05  04:25                1,104 c.msf        

(5) Copya a mail(Subject="Test-3") to folder "b" 
    Mail count of folder "b" = 1
    Subject in thread pane   = Test-3
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/c
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:14                1,695 a.msf
      2004/11/05  04:18                  611 b
      2004/11/05  04:18                1,665 b.msf
      2004/11/05  04:28                  611 c
      2004/11/05  04:28                1,665 c.msf

    Subject Data in file of "a" : Subject: Test-1
    Subject Data in file of "b" : Subject: Test-2
    Subject Data in file of "c" : Subject: Test-3

    => Folder name is "b" but file of c/c.msf is used

(6) Rename folder "b" to "c"
    Mail count of folder "c" = 1
    Subject in thread pane   = Test-1
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/a
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:30                1,715 a.msf
      2004/11/05  04:18                  611 b
      2004/11/05  04:18                1,665 b.msf
      2004/11/05  04:28                  611 c
      2004/11/05  04:28                1,665 c.msf

    Subject Data in file of "a" : Subject: Test-1
    Subject Data in file of "b" : Subject: Test-2
    Subject Data in file of "c" : Subject: Test-3

    => Folder name is "c" but file of a/a.msf is used
       (Timestamp of a.msf is updated)

(7) Try to rename folder "c" to "b"
    => Failed to rename with error message of ;
         The folder couldnot be renamed.
         Perhaps the folder is being reparsed,
         or the new name is not a valid folder name.

(8) Rename folder "c" to "a"
    Mail count of folder "a" = 1
    Subject in thread pane   = Test-2
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/b
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:30                1,715 a.msf
      2004/11/05  04:18                  611 b
      2004/11/05  04:18                1,665 b.msf
      2004/11/05  04:28                  611 c
      2004/11/05  04:28                1,665 c.msf
    
    Which msf is used?

(9) Copy a mail(Subject="Test-4) to folder "a"
    Mail count of folder "a" = 2
    Subject in thread pane   = Test-2 and Test-4
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X1.sbd/b
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:30                1,715 a.msf
      2004/11/05  04:41                1,222 b
      2004/11/05  04:41                2,221 b.msf
      2004/11/05  04:28                  611 c
      2004/11/05  04:28                1,665 c.msf

    Subject Data in file of "a" : Subject: Test-1
    Subject Data in file of "b" : Subject: Test-2 & Test-4
    Subject Data in file of "c" : Subject: Test-3

    => Folder name is "a" but file of b/b.msf is used
       (Timestamp of b/b.msf is updated)


(10) Move subfolder "a" from parent folder "X1" to parent folder "X2"
    Mail count of folder "a" = blank(=0) => Mail lost!
    Subject in thread pane   = No mail is displayed
    
    mailbox:/D|/MAIL-NEWS/Mail/Local Folders/Test.sbd/X2.sbd/a
    
    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X1.sbd
      2004/11/05  04:04                  609 a
      2004/11/05  04:30                1,715 a.msf
      2004/11/05  04:28                  611 c
      2004/11/05  04:28                1,665 c.msf

    D:\MAIL-NEWS\Mail\Local Folders\Test.sbd\X2.sbd
      2004/11/05  04:46                    0 a
      2004/11/05  04:46                1,330 a.msf
      2004/11/05  04:41                1,222 b
      2004/11/05  04:41                2,221 b.msf

    Subject Data in file of "a" : No data is included(Null file)
    Subject Data in file of "b" : Subject: Test-2 & Test-4

    => Upon move, files of b/b.msf were moved,
       because Mozilla thought b/b.msf is file for folder "a".
       But a/a.msf was used on folder open, I guess,
       because no garbage is kept for "X2" and whose subfolders.

(End of test)

Comment 32

[Owen]

I've read through the bug log for this bug, but I have little knowledge of the
underlying code.

Does this bug have something to do with the delayed compacting of folders?
It seems to me that when I delete a folder in mozilla it doesn't really get
deleted on the file system until I compact. This has cause bizarre behaviour 
for me from time to time, luckily no data loss yet...

Comment 33

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #32)
> when I delete a folder in mozilla it doesn't really get deleted on the file
> system until I compact.

What is youe exact action of your "delete a folder"?
 (A) Delete "mail folder" (POP3 or Local Folders)
     => Whole of the "mail folder" is moved to Trash folder.
        File of <folder_name> and <folder_name>. msf,
        and <folder name>.sbd directry if sub folder is defined,
        are moved to 'Trash.sbd' directry.
        These files/directries are physically removed from file system
        when "Empty Trash" is executed.
 (B) Delete mails in a "mail folder" (POP3 or Local Folders)
     => Mails are copied to "Trash" folder. ('Trash' & 'Trash.msf' file set)
        Mails in original folder is marked "deleted" in file of <folder_name>.
        This "marked with deleted flag" is physicaly removed from
        the <folder_name> file when "Compact folder" is executed.

Your complaint sounds (B) because you say 'compact', although you say 'delete
folder' and 'not delete on the file system'.
Owen, isn't it right? 
Please clarify terms of 'folder'/'mail', 'file/directry/folder', 'file/file
system', 'delete/compact', 'compact/empty' and so on used by Mozilla/Thunderbird
and Operating System.

Comment 34

[WADA:World Anti-bad-Duping Agency]

Change Product to Core.
Change "Assigned to:" to David, since original assignee is "not reading bugmail".

David, what's wrong?
Note:
I believe that this problem won't occur if user restarted before folder name
reuse, since your patch in other bug(Search result folder deletion case)
performs panacea.dat clean-up on shutdown. 
So possibility that the problem really occurs during user's daily use is
declined very much already.
But this is one of the worst flaw for mailer.

Comment 35

[WADA:World Anti-bad-Duping Agency]

*** Bug 275094 has been marked as a duplicate of this bug. ***

Comment 36

[:aceman]

May not be that easy, see Bug 231941. If the data isn't marked as deleted in
memory, no clean-up on shutdown will help you.

Comment 37

[Richard Trout]

Just noting that I reproduced this problem in Local Folders on Netscape 7.2, and
quiting, deleting panacea.dat (renaming it actually), and restarting cleared the
problem. 

Note: when I was attempting to re-create problem folders, it was creating a
FOLDER of the directory name instead of a file, and the New Folder dialog
remaining until closed - attempts to use folder obviously fail.

Comment 38

[David :Bienvenu]

Attachment: fix problem m-wada described in comment 31

Comment 39

[chris hofmann]

Comment on attachment 187415 [details] [diff] [review]
fix problem m-wada described in comment 31

a=chofmann

Comment 40

[David :Bienvenu]

it turns out this patch is neccesary but not sufficient. I developed a more
complicated fix on my laptop and at the end added that one change. I tried that
one change on my desktop and it appeared to work, but it's not quite sufficient.

Comment 41

[David :Bienvenu]

Attachment: rest of the fix

we need to clear out the path for the renamed folder, because calling rename on
the file spec actually changes the name in the path member variable for the
folder, and if that folder gets recreated, it ends up with the new name path
instead of the old path. Clearing the path will force us to regenerate it
correctly from the uri.

Comment 42

[David :Bienvenu]

Comment on attachment 187547 [details] [diff] [review]
rest of the fix

this just forgets the path of a folder object that's being renamed, and thus
removed.

Comment 43

[Benjamin Smedberg]

Comment on attachment 187547 [details] [diff] [review]
rest of the fix

a=bsmedberg for checkin on 6/30 only

Comment 44

[David :Bienvenu]

*** Bug 230513 has been marked as a duplicate of this bug. ***

Comment 45

[WADA:World Anti-bad-Duping Agency]

With Mozilla 2005070105-trunk(Win-2K), no problem is found in all cases of
comment #31, cyclic rename test.
No mail folder/mail data integrity probem is found in comment #28 too.
=> this bug is VERIFIED.

However, very very minor problem still remains in test case of comment #28,
folder creation & rename & folder name reuse.
 - When folder name is re-used without restart,
   mail count displayed in folder pane becomes mail count of old folder
   (or count of renamed folder). 
 - This garbage count is reset(set to blank) when the newly created folder  
   (folder of re-used name) is clicked.
I'll open new bug for this remaining very minor problem, if I feel this is
annoyance for me.

Anyway, a longlived problem, and one of the worst problems for mailer, has been
resolved at last!
This is another incident of my theory - critical mail&news bug for me is
resolved soon if David comes :-)
David, thanks for your great effort.
 

 

Comment 46

[WADA:World Anti-bad-Duping Agency]

Re-opening for remaining problem when special folder name case.
(Mozilla suite 2005070105-tunk, Win-2K) 

In cyclic rename test for special folder name, "/" -> "#" -> ":" -> "/" -> "#"
in my case, "<HEXA-STRING>.msf" was not deleted.
 (1) Create folder of "/", copy a mail to it, left click "/" folder => OK
 (2) Rename "/" to "#" thru context menu 
     => OK
 (3) Rename "#" to ":" thru context menu (No left click of folder is done)
     => OK
 (4) Rename ":" to "/" thru context menu (No left click of folder is done)
     =>  an hourglass icon is displayed and doesn't disappear.
 (5) Rename "/" to "#" thru context menu (No left click of folder is done)
     => <Hexa-string-old> is successufully deleted.
        <Hexa-string-old>.msf (for ":" or "/") is not deleted.
     => <Hexa-string-new>/<Hexa-string-new>.msf (for "#") are created.
        Folder of "#" itself has no problem.
".msf" refresh doesn't complete at step (4)?

Comment 47

[David :Bienvenu]

please file a new bug on that issue, thx! I don't believe it's a rename issue
anyway.

Comment 48

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #47)
> please file a new bug on that issue, thx! I don't believe it's a rename issue
> anyway.
Roger. I'll open new bug with low severity.
Changing this bug VERIFIED again.

Comment 49

[David :Bienvenu]

*** Bug 300118 has been marked as a duplicate of this bug. ***

Comment 50

[Richard Trout]

Does anyone know what mozilla suite release this appeared in? If so, can this be
added to the Target Milestone so it is easier to identify (instead of build
2005070105-trunk)

Comment 51

[David :Bienvenu]

I think this went into 1.8b3, and tbird 1.1a2, so will be in 1.5.

Comment 52

[Simon Paquet [:sipaq]]

Comment on attachment 124778 [details] [diff] [review]
patch v1

Since this bug has been resolved as FIXED, I'm removing the
review request.