749909 - Inbox destroyed by incomplete rename error handling. If localized folder name is used, original special folder file(e.g. Inbox) is deleted when rename of a folder to "English special folder name of different case"(e.g inbox) is requested.

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Tetsuo Handa]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120420145725

Steps to reproduce:

(step 1)
I created a mail account.

(step 2)
I received a lot of mails using POP3 into "受信トレイ" (the corresponding name
before translation would be "inbox" or "incoming" or "received" or something
like that. Excuse me, I'm using Thunderbird in Japanese language.).

(step 3)
I created a new folder named "test" into the mail account.

(step 4)
I tried to rename the folder from "test" to "inbox".
Then, a warning message window appeared:

  警告
  その名前のフォルダはすでに存在しています。他の名前を入力してください。

which the message in English would be something like

  Warning:
  A folder by that name already exists. Please enter a different name.

.

I assumed that while "受信トレイ" is displayed as a name of the folder that stores
incoming mails, "inbox" is used as an internal name for managing the folder
that stores incoming mails.

Therefore, I clicked the OK button to close the warning message window and
clicked cancel button to close the input dialog for rename.

(step 5)
I tried to drag&drop a mail from "受信トレイ" folder to the "test" folder.
Then, a warning message window appeared:

  警告
  test フォルダが満杯のため、これ以上メッセージを保存できません。メッセージを
  保存する領域を空けるには、古くて不要なメッセージを削除するか、フォルダを
  最適化してください。

which the message in English would be something like

  Warning:
  Unable to store more messages because the test folder is full. To make spaces
  for storing messages, either delete old unneeded messages or optimize the
  folder.

.

I clicked the OK button to close the warning message window.

Note that this warning message indicates that an inconsistency state problem
has occurred, for the "test" folder I've just created is empty and therefore
the "test" folder cannot be full.

(step 6)
I clicked a mail in the "受信トレイ" folder. However, while I can click the
messages in the title area, the contents of the chosen message no longer
printed in the message body area.

(step 7)
After I restarted the Thunderbird, the title area also became empty.



Actual results:

I lost all mails in the "受信トレイ" folder.

My guess is that while the warning message was printed that complains the
folder name duplication, internal state was not correctly rolled back.

This bug is 100% reproducible in Japanese environment. I don't know whether
this bug is reproducible in English environment, for this bug might
involve language translation functionality.

I encountered this bug with 11.0.1 release but this bug is not yet
fixed in 12.0 release.


Expected results:

The mails in the "受信トレイ" folder should have not been destroyed.

Comment 1

[Tetsuo Handa]

I tried to reproduce this bug on Thunderbird 11.0.1 for Linux using
ubuntu-12.04-desktop-i386.iso , but I couldn't reproduce this bug.

The folder name on Linux was "Inbox" rather than "受信トレイ", even when I
chose Japanese language. Maybe this bug is specific to Thunderbird for Windows.

Comment 2

[Tetsuo Handa]

I made a demo movie. http://www.youtube.com/watch?v=No3lyZNk9Hc

I succeeded to reproduce similar problem on Thunderbird 12.0.1 for Linux using
Japanese environment. Since I can't reproduce this bug in English environment
and I can reproduce this bug in non-English environment, I assume this bug
involves language translation functionality.

This is an important bug for non-English environments. Please fix.

Comment 3

[Hashem Masoud]

(In reply to Tetsuo Handa from comment #2)
Please read: http://kb.mozillazine.org/Disappearing_mail it might help you.

Comment 4

[Tetsuo Handa]

(In reply to Hashem Masoud from comment #3)
> Please read: http://kb.mozillazine.org/Disappearing_mail it might help you.

Thanks, but I don't think mine is one of the KB cases.

Before I try to rename test folder to inbox, the dir command shows:

2012/05/13  19:29    <DIR>          .
2012/05/13  19:29    <DIR>          ..
2012/05/13  19:28           872,362 Inbox
2012/05/13  19:29            88,297 Inbox.msf
2012/05/13  19:28                27 msgFilterRules.dat
2012/05/13  19:28             9,751 popstate.dat
2012/05/13  19:29                 0 test
2012/05/13  19:29             1,259 test.msf
2012/05/13  19:27                 0 Trash
2012/05/13  19:27             1,150 Trash.msf
               8 個のファイル             972,846 バイト
               2 個のディレクトリ   1,098,223,616 バイトの空き領域

After I tried to rename test folder to inbox, the dir command shows:

2012/05/13  19:30    <DIR>          .
2012/05/13  19:30    <DIR>          ..
2012/05/13  19:29                 0 inbox
2012/05/13  19:29            88,355 Inbox.msf
2012/05/13  19:28                27 msgFilterRules.dat
2012/05/13  19:28             9,751 popstate.dat
2012/05/13  19:30             1,666 test.msf
2012/05/13  19:27                 0 Trash
2012/05/13  19:27             1,150 Trash.msf
               7 個のファイル             100,949 バイト
               2 個のディレクトリ   1,099,096,064 バイトの空き領域

We can see that while the size of test.msf has increased a bit, the size of
Inbox became 0 and also the file named test was deleted. I strongly suspect
that Thunderbird by error truncated Inbox file. Unless Thunderbird kept backup
of Inbox file somewhere outside this directory, I can't recover mails in the
Inbox.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Problem is reproduced with Tb 12.0.1 on MS Win-XP, and it is not localized Inbox only problem and is not localized Tb only problem.
(Phenomenon-A) Rename to inbox, trash, archives, drafts, templates, junk.
  Once localized special folder name is used for special folder,
  problem occurs even in Tb of non-localized folder name(== en-US build),
  on 受信トレイ(Inbox of ja) / ごみ箱(Trash of ja),
  on Postausgang(Inbox of de) / Papierkorb(Trash of de), and so on.
  Noe: If English folder name is used in Tb 12 en-US, problem doesn't occur. 
(Phenomenon-B) Rename to other folder's hashed file name.
  Problem can be produced even on folder of ascii-only chars in Tb 12.01 en-US,
  if folder name has special character such as # and file name is hased.

(Phenomenon-B)
(1) Create folder named ABCDEFG# (file name = ABCDEFG1b889378.msf),
    put mail-1 in this folder.
    Folder name of ABCDEFG# is saved in .msf file.
(2) Create folder named XYZXYZ  (file name = XYZXYZ.msf),
    put mail-2 in this folder.
(3) Rename XYZXYZ to ABCDEFG#, abcdefg# is normally rejected.
    => "Comparison of Folder Name" is successful, and is case insensitive.
(4) Rename XYZXYZ to ABCDEFG1b889378.
    Rename looks rejected with "already exists" message, but, even after Cancel,
    original ABCDEFG1b889378/ABCDEFG1b889378.msf(contains mail-1) is deleted,
    and XYZXYZ/XYZXYZ.msf(contains mail-2) is renamed to
    ABCDEFG1b889378/ABCDEFG1b889378.msf.
    String of "XYZXYZ" is written in ABCDEFG1b889378.msf.
    ABCDEFG# and XYZXYZ are still shown at folder pane.
    "Rename to abcdefg1b889378" produces same problem.
(5) Terminate Tb.
    File of XYZXYZ.msf is created.
(6) Restart Tb.
    ABCDEFG# is not shown at folder pane.
    ABCDEFG1b889378 is not shown at folder pane.
    XYZXYZ is shown at folder pane. Folder content = mail-2.
    This is same even after delete of panacea.dat.
    i.e. Folder name(==XYZXYZ) is written in .msf and is shown as folder name.

Phenomenon-A on special folder of localized folder name is perhaps caused by following.
(i)  Localized special folder name is writen in .msf file.
     It seems change before Tb 12(seen in Tb 11. I don't know about older Tbs)
     This causes problem of bug 752423.
(ii) Case sensitive file name comparison from Tb 12, like Bug 752697.

Because Mail Folder Name(受信トレイ, Postausgang) != File Name(Inbox), Mail Folder Name(ごみ箱, Papierkorb) != File Name(Inbox), ...,
and because inbox != Inbox, trash != Trash, ...,
same problem as Phenomenon-B occurs on localized special mail folder.

Because file system of MS Win is;
  - case insensitive file name
  - case of file name is kept as-is
file named inbox == file named Inbox on MS Win.
However, file system of Linux is case sensitive, and file system of Mac OS X is similar to Linux's one and is case sensitive too, but Filder may be different.

In Linux, file named abc != Abc != ABC.
Is "mail folder name on Linux" case sensitive? insensitive?
IIRC, absolute "case insensitive mail folder name" is "IMAP INBOX" only.
Following is bugs which has "case sensitive" and updated within 3 months.
> https://bugzilla.mozilla.org/buglist.cgi?list_id=3078463;short_desc=case%20sensitiv;chfieldto=Now;query_format=advanced;chfieldfrom=3m;short_desc_type=allwordssubstr
Affected by fix of bug such as next?
> Bug 222223 - imap: Trash folder case sensitive?

Comment 6

[WADA:World Anti-bad-Duping Agency]

Checked with Tb 11.0.1 and Tb 9.0.1 on MS Win-XP.
(1) Tb 11.0.1 e-US, and Tb 9.0.1 en-US.
- English folder name is shown, even after localized name is saved in .msf
  by localized Tb 12,0.1 de or ja, or localized Tb 11.0.1 de or ja.
- "Rename to inbox" is normally rejected. This bug doesn't occur.
(2) Tb 11.0.1 ja, and Tb 9.0.1 ja.
- Japanese folder name is shown.
- "Rename FolderX to inbox" looks to do nothing. It silently fails, no error
  message is shown, and "rename box" is kept open.
  But file of Inbox is deleted and file of FolderX is renamed to inbox(size=0).
  Folder pane is not changed.
  If Inbox is already opened :
    Inbox.msf remains.
    Thread pane of Inbox is not changed(show as if mails exist).
    But when mail is clicked, nothing is shown.
  If Inbox is not opened yet :
    Inbox.msf is changed to inbox.msf.
    Thread pane becomes blank when Inbox is clicked.
  If FolderX is clicked, thread pane becomes blank, folder open doesn't finish.
  FolderX.msf remains. File of FolderX doesn't exist(renamed to inbox).

"Case sensitive compare" like issue was irrelevant to problem.
"Localized special folder name in .msf file" is perhaps culprit, and problem was generated by Tb 9 or before.
Sorry but I can't check reressed Tb release, because I can't find Tb 10 ja, Tb 8 ja or older. 
New in Tb 12 was;
  - Once localized name is used by localized Tb, localized folder name is used
    even by en-US build. Then, this bug occurs even in en-US build
    when localized(non-English) special folder name is used.
  - "Already exists" message is now shown kindly, but it's too late.
    Error meesage is displayed after Tb already replaced mail folder file.

Problem is perhaps;
  Due to "localized folder name in .msf"("file name != mail folder name"),
  Phenomenon-B occurs even on Tb's special mail folder.
  If en-US build, or English special folder name is written in .msf,
  or if localized special folder name is not written in .msf file,
  case sensitive check of "new name(inbox)" with "other's folder name(Inbox)"
  works well as expected, then problem doesn't occur.

Comment 7

[WADA:World Anti-bad-Duping Agency]

It was same in Tb 3.1.20 ja.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20
[Step to reproduce]
(0) Copy a local mail folder file(contains some mails) to Inbox,
    Delete file named Inbox.msf,
    Create empty/null FolderX file.
    Delete panacea.dat.
(1) Start Tb 3.1.20 ja
    Inbox.msf, FolderX.msf is created.
    "受信トレイ" is shown at folder pane. Confirm that mails are normally shown. 
    "FolderX"    is shown at folder pane. No mails in FolderX.
(2) Rename FolderX to inbox
    => Rename does nothing in UI. Rename box still kept.
       Rename silently fails.
       But Inbox is deleted and FolderX is renamed to inbox.
       Inbox.msf is uchanged, because Inbox folder was opened at step (1).
       FolderX.msf is unchanged.
(3) Cancel at rename box.
(4) Try to view mails in "受信トレイ"(Inbox of ja build)
    => fails because file of Inbox is already replaced by inbox of size=0.
(5) Try to view mails in "FolderX"
    => fails because file of FolderX is already renamed to inbox.

In localized Tb 3.1.20 ja, unless column selection change was executed at thread pane, localized special folder name was not seen in .msf file.
So, "localized folder name in .msf" is irrevant to above problem.

Comment 8

[Hiroyuki Ikezoe (:hiro)]

Attachment: Possible fix without test

nsIFile::MoveTo replaces the new path even if the path already exists, so we should care about such case in RenameFolder().

I just confirmed that Phenomenon-B described in comment #5 is fixed, and have not been confirmed the localized issue yet because I have no localized Thunderbird on my local machine now.

Comment 9

[Hiroyuki Ikezoe (:hiro)]

Comment on attachment 623963 [details] [diff] [review]
Possible fix without test

Review of attachment 623963 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/local/src/nsMsgLocalStoreUtils.cpp
@@ +316,5 @@
>    return rv;
>  }
> +
> +// Any errors which are not handled by us considers that the new folder
> +// already exists for the safety.

The comment here is something strange..  
This should be "consider as ..."?

Comment 10

[Hiroyuki Ikezoe (:hiro)]

Attachment: Test for Phenomenon-B

Comment 11

[David :Bienvenu]

Wada, thx very much for your detective work. And Hiro, thx very much for looking into this. I was going to ask you if you could, but you did it before I got around to it!

Comment 12

[Hiroyuki Ikezoe (:hiro)]

Attachment: Do not rename to special folders name

I comfirmed this patch fixes Phenomenon-A with mail/locales/en-US/chrome/messenger/messenger.properties modification.

I can not write unit test for this yet since there is no way to modify string bundles in test.

Comment 13

[David :Bienvenu]

Comment on attachment 624588 [details] [diff] [review]
Do not rename to special folders name

typo here:

+#define UNSENT_FOLDER_NAME "unsnet messages"

I'm a little worried we haven't quite figured out why the localized name is getting stored in the .msf file, and the various ramifications of that.

Comment 14

[Hiroyuki Ikezoe (:hiro)]

(In reply to David :Bienvenu from comment #13)
> Comment on attachment 624588 [details] [diff] [review]
> Do not rename to special folders name
> 
> typo here:
> 
> +#define UNSENT_FOLDER_NAME "unsnet messages"

Woohooooo, horrible typo.

> I'm a little worried we haven't quite figured out why the localized name is
> getting stored in the .msf file, and the various ramifications of that.

Though I do not the reason, I think it is reasonable that nsIMsgFolder has a property named "identicalName" to compare with (and stored in .msf?).

Comment 15

[WADA:World Anti-bad-Duping Agency]

"Unsent Messages" is special.
  English Folder Name = Outbox  (changed from Unsent Messages, around Tb 3?)
  Used File Name      = Unsent Messages.msf
I think File Name != Folder Name is reason why folder name is held in .msf. 

For localized folder name written in .msf.
One of them looks written by "saving thread column selection data in .msf", because string of localized name is seen as an element of JSON like format data for thread column definitions.
This looks different data from "actual folder name in .msf when file name is hashed".

Comment 16

[WADA:World Anti-bad-Duping Agency]

Localized folder name looks written in .msf even by en-US build if Tb 12.
Bug 752423 is for such issue.

Comment 17

[Hiroyuki Ikezoe (:hiro)]

Attachment: Tests covers both cases.

Yay! Now I've managed to override localized strings in xpcshell test.

Comment 18

[Hiroyuki Ikezoe (:hiro)]

Comment on attachment 624661 [details] [diff] [review]
Tests covers both cases.

clearing review request.

The tests does not work today..

Comment 19

[David :Bienvenu]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #18)
> Comment on attachment 624661 [details] [diff] [review]
> Tests covers both cases.
> 
> clearing review request.
> 
> The tests does not work today..

I just verified that they fail without the other patches, and was just about to check if they succeed with the patches. Guess I'll hold off...

Comment 20

[David :Bienvenu]

Comment on attachment 623963 [details] [diff] [review]
Possible fix without test

thx for the patch! I think this function: ExistsNewFolderNameOnDisk

reads better named FolderNameExistsOnDisk

otherwise, I think this looks good. But we should wait until you've got the test working before we start landing.

Comment 21

[Hiroyuki Ikezoe (:hiro)]

Attachment: Tests can work correctly

OK. The problem was 'do_throw' in StringBundleOverride object. Also removed thread.processNextEvent() that is unnecessary.

Comment 22

[Hiroyuki Ikezoe (:hiro)]

Attachment: Do not rename to special folders name

Fix a typo.

Comment 23

[David :Bienvenu]

Comment on attachment 624944 [details] [diff] [review]
Tests can work correctly

Since you're only dealing with the inbox name here, I think it would be clearer if getStringFromName only handled the inbox, instead of all the other possible folder names.

Or, if override_localized_strings() becomes a utility function shared between tests, then it would make sense to handle all the folder names.

Comment 24

[David :Bienvenu]

Comment on attachment 625553 [details] [diff] [review]
Do not rename to special folders name

The other patch in this bug effectively handles this issue at a lower level, doesn't it?

Also, what if the specialized folder doesn't exist, and the user wants to rename an existing folder to the specialized folder name?

Comment 25

[David :Bienvenu]

Comment on attachment 625553 [details] [diff] [review]
Do not rename to special folders name

I'm going to clear the review request based on the questions I raised earlier...

Comment 26

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #24)
> Comment on attachment 625553 [details] [diff] [review]
> Do not rename to special folders name
> 
> The other patch in this bug effectively handles this issue at a lower level,
> doesn't it?
> 
> Also, what if the specialized folder doesn't exist, and the user wants to
> rename an existing folder to the specialized folder name?

Comment 27

[Wayne Mery (:wsmwk)]

Unfortunately hiro is on longer able to assist

Comment 28

[Wayne Mery (:wsmwk)]

(unfortunately hiro is no longer with us)