491424 - Problems changing Trash -> Deleted Items (mismatch between namespace usage and trash_folder_name set by trash folder selection UI. namspace=INBOX&&trash_folder_name=INBOX/Trash => INBOX.INBOX.Trash is used)

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[MrC]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b5pre) Gecko/20090504 Shredder/3.0b3pre

I've been using "Deleted Items" as the name of my Trash folder on my IMAP server for ages (for easy compatibility with clients such as Outlook).

Very recently (not sure which build) the Mac version of Shredder started using Trash as the name of the trash folder.  It is treating my existing Deleted Items folder as an ordinary folder, and of course is not placing deleted items there (as per settings).

When I set the Account Setting->Server Settings->"When I delete a message"->"Move it to this folder" to "Inbox->Deleted Items", it see that TB is creating a new folder Inbox->INBOX->Deleted Items, where 'INBOX" is in italics. 

I have no idea why it is creating this extra INBOX folder level.  All my existing folders appear under Inbox, and show correctly.

I've deleted the profile and the same thing occurs.

Everything is fine on the same nightly version of Shredder on my PC.

Reproducible: Always

Steps to Reproduce:
1. Create a folder named Deleted Items
2. Follow instructions above to set the Deleted Items as the trash folder
3. Delete something, and notice that it does not go to Deleted Items
4. Also note that a new folder Inbox->INBOX->Deleted Items appears (provided the show even unsubscribed folders setting is configured in the account)
Actual Results:  
Items that are deleted disappear
Items that are deleted do not get placed into Deleted Items

Expected Results:  
Items should be moved (copied) to Deleted Items

Comment 1

[WADA:World Anti-bad-Duping Agency]

(Q1) What is set in next setting?
> mail.server.serverN.trash_folder_name = [Gmail]/Trash (Gmail IMAP case)
> Server Settings/Advanced
>   IMAP server directory: ???
>   Personal Name Space:   ???
>   [?] Allow Server to override these namespaces

(Q2) Are there multiple folders like next?
Trash, Inbox(. or /)Trash, INBOX(. or /)Trash
 
(Q3) What happens if next action is taken? (after change, restart of Tb is required) Which folder is uses as "trash folder for Tb"? (folder with trash can icon) 
> mail.server.serverN.trash_folder_name = Inbox(. or /)Deleted Items
                                          ==> Deleted Items

Comment 2

[MrC]

Q1: 
a) mail.server.server2.trash_folder_name: INBOX/Deleted Items
b) see attachment image

Q2: I no longer have the Trash folder - I tossed it.  I've been unsuccessful at getting a usable trash folder.  This is what I currently have under:  

ImapMail/myimap/INBOX.sbd
[code]
$ ls -ld Deleted\ Items* INBOX.* INBOX.sbd/Deleted\ Items.msf 
-rw-------@ 1 cappella  staff   3.3M May  4 19:22 Deleted Items
-rw-r--r--@ 1 cappella  staff    41K May  4 19:24 Deleted Items.msf
-rw-r--r--@ 1 cappella  staff   1.5K May  4 19:26 INBOX.msf
drwxr-xr-x@ 3 cappella  staff   102B May  4 19:22 INBOX.sbd/
-rw-r--r--@ 1 cappella  staff   1.7K May  4 19:27 INBOX.sbd/Deleted Items.msf
[/code]
The INBOX.* files/folders are created whenever I select the option to assign a folder as the trash folder (as mentioned in the first post).

Q3) The Deleted Items folder a) regains the special trash icon, and b) and deleted mail now goes to the Deleted Items correctly.

So, it appears the dialog to allow selecting the trash folder is not removing the IMAP namespace and creating an extra level in the tree.

Comment 3

[MrC]

Attachment: Server settings

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Shredder started using Trash as the name of the trash folder.

Probably following setting is isitially used. (You defined IMAP account newly on Shredder?)
> mail.server.server2.trash_folder_name: Trash

(In reply to comment #3)
> Server settings

Comibination of following is probably relevant.
> mail.server.server2.trash_folder_name: INBOX/Deleted Items
> Personal namespace: "INBOX."

(1) Because no "IMAP server directory:" and server pass name as "Inbox", 
    folder of "INBOX.Deleted Items" is displayed as INBOX->Deleted Items
    at folder pane and trash folder selection UI.
(2) When you select the folder at trash folder selection UI,
    Tb saves "INBOX/Deleted Items" in mail.server.server2.trash_folder_name.
(3) Because Personal namespace: "INBOX.", folder creation request becomes
    "INBOX.INBOX.Deleted Items".
(4) Top level "INBOX" is standard/common INBOX defined by IMAP(case insensitive).
    Second level INBOX is ordinal sub flder, and your server probably uses 
    case insensitive file system.
    => "Inbox.INBOX.Deleted Items" is created.
I think "issue due to mismatch between (2) and (3)" is same as Problem-1 in Bug 480393 Comment #11.

> I've been unsuccessful at getting a usable trash folder.

This phenomenon was not reported in Bug 480393.

According to your answer to Q3), following looks to be effective workaround in your case.
> mail.server.server2.trash_folder_name = Deleted Items
> Never change it via trash folder selection UI.  

Anyway, get IMAP protocol log for folder creation request by Tb based on "mail.server.server2.trash_folder_name = INBOX/Deleted Items" when Personal namespace: "INBOX.", and check IMAP level flow.

By the way, "Problem-1 in Bug 480393 Comment #11" is better to be manipulated by this bug, separately.

Comment 5

[MrC]

Thanks for your quick response, by the way.  Very nice.

> Probably following setting is isitially used. (You defined IMAP account newly
> on Shredder?)

Hmmm, I've been using the same account settings for some time now.  The sudden appearance of Trash, with it becoming the special folder instead of my long-used Deleted Items folder surprised me.  I've recreated the account settings several times, but this has never been a problem in the past.

>> I've been unsuccessful at getting a usable trash folder.
>
> This phenomenon was not reported in Bug 480393.

As an addendum, neither of the "Deleted Items" folder that existed (root level, and the deeper INBOX/Deleted Items) had the trash can icon.  When I deleted something, it just disappeared.  I could undo. 

The workaround is fine. 


This is probably the protocol info you wanted to see:


-1334579200[1a7db1f0]: 1d42c00:ImapSrv:NA:CreateNewLineFromSocket: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc.  See COPYING for distribution information.^M

...

-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:CreateNewLineFromSocket: 9 OK LIST completed^M
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:SendData: 10 create "INBOX.INBOX.Deleted Items"^M
-1336745984[1a4cdf20]: ReadNextLine [stream=1a4ce718 nb=44 needmore=0]
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:CreateNewLineFromSocket: 10 OK "INBOX.INBOX.Deleted Items" created.^M
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:SendData: 11 subscribe "INBOX.INBOX.Deleted Items"^M
-1336745984[1a4cdf20]: ReadNextLine [stream=1a4ce718 nb=26 needmore=0]
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:CreateNewLineFromSocket: 11 OK Folder subscribed.^M
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:SendData: 12 list "" "INBOX.INBOX.Deleted Items"^M
-1336745984[1a4cdf20]: ReadNextLine [stream=1a4ce718 nb=57 needmore=0]
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:CreateNewLineFromSocket: * LIST (\HasNoChildren) "." "INBOX.INBOX.Deleted Items"^M
-1336745984[1a4cdf20]: ReadNextLine [stream=1a4ce718 nb=22 needmore=0]
-1336745984[1a4cdf20]: 1c7aa00:ImapSrv:A:CreateNewLineFromSocket: 12 OK LIST completed^M
-1334579200[1a7db1f0]: ImapThreadMainLoop entering [this=1d42c00]
-1607350496[120ab30]: 1d42c00:ImapSrv:NA:SetupWithUrl: clearing IMAP_CONNECTION_IS_OPEN

Comment 6

[WADA:World Anti-bad-Duping Agency]

Confirming, adding "namespace" and "trash_folder_name" to summary for ease of search.

Comment 7

[Uwe Schröder]

I still see this in TB 3.0.2.

Situation: Web mail on server uses "trash" instead of "Trash" (case sensitive). Trying to make TB3 use the same folder. namespace.personal is "INBOX.".

1. Select "trash" folder in the UI using the folder picker.
2. trash_folder_name is set to "INBOX/trash" in prefs.js.
3. Next time when TB starts, it creates a folder "INBOX.INBOX.trash" on the server.
4. Editing prefs.js and deleting "INBOX/" from trash_folder_name resolves the problem.

Apparently the namespace prefix is applied twice. It should either be removed from the folder picker value before it is saved, or only be prepended if it is not already there when the folder is created.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> 2. trash_folder_name is set to "INBOX/trash" in prefs.js.
> 3. Next time when TB starts, it creates a folder "INBOX.INBOX.trash" on the
> server.

As setting is trash_folder_name, I guess trash_folder_name=INBOX.trash is another(and easy-to-understand) workaround. Is it right?
If right, problem can be said:
  Trash folder selection UI should put "IMAP folder name" in trash_folder_name,
  instead of "internal folder path representation".
  i.e. Trash folder selection UI should care for delimiter character.

Comment 9

[Uwe Schröder]

It was worth a try, but no, that doesn't work. "INBOX.trash" and "INBOX/trash" are totally synonymous. The only way is to omit the namespace prefix altogether.

trash_folder_name is not absolute, but relative to the namespace.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Additional mismatch between delimiter of "." in namespace and trash_folder_name.

Namespace.
> 5 namespace
> * NAMESPACE (("INBOX." ".")) NIL NIL
> 5 OK Namespace completed.
Trash folder set by trash selecttion UI.
> mail.server.serverN.trash_folder_name = INBOX/Trash

Tb uses "/" of trash_folder_name as-is in list command for existence check of trash even though delimiter=".", and it usually fails.
> 8 list "" "INBOX.INBOX/Trash"
> 8 OK List completed.
Then create is always requested, with correct delimiter of ".".
> 9 create "INBOX.INBOX.Trash"
> 9 NO Mailbox exists.

Comment 11

[Matt Hoskins]

Unsurprisingly, still in 3.1. I've hit this bug.

As others have said - selecting the imap folder to use for trash through the account settings UI puts the full path in trash_folder_name, always using "/" as the delimiter (even if it's "." on the server). Then when the value from trash_folder_name is used it's always prefixed with the personal namespace. So if I select "INBOX.Trash" I end up with "INBOX/Trash" stored, thunderbird doesn't recognise my Trash folder as being a trash folder (i.e. it doesn't show the icon) and it ends up creating INBOX.INBOX.Trash and shoves mails into there and everything gets confused. If I manually edit prefs.js and change "trash_folder_name" to just read "Trash" then I can get things working again.

The reason, perhaps, that more people haven't hit this is that the default trash folder name that the UI suggests doesn't cause a problem, so if the user just changes to use that and doesn't browse to pick an imap folder explicitly then perhaps things work for them.

Comment 13

[WADA:World Anti-bad-Duping Agency]

I attached test extension to bug 533140.
> https://bugzilla.mozilla.org/attachment.cgi?id=704299
This extension does following.
(A) change incomingServer.trashFolderName to currently selected folder.
    => mail.server.serverN.trash_folder_name is automatically changed by Tb.
(A-1) Use actual mbox name in msgFolder.URI when request change of incomingServer.trashFolderName, instead of msgFolder.prettiestName where localized special folder name is stored by localized Tb.
(A-2) Remove namespace from string set in mail.server.serverN.trash_folder_name (and incomingServer.trashFolderName), because logic which processs mail.server.serverN.trash_folder_name seems to request "Mbox name without namespace" in mail.server.serverN.trash_folder_name.
(perhaps uses getUriWithNamespacePrefixIfNecessary() on string set in mail.server.serverN.trash_folder_name)
(B) Request msgFolder.setflag(Trash) for currently selected folder.
    Request msgFolder.clearflag(Trash) for all other folder of the account.

Is the test extension effective in your environment?

Note:
Because of bug 831664, incomingServer.trahFolderName may be undefined, and if change of incomingServer.trahFolderName is requested when undefined, Exception occurs. To bypass bug 831664, do next, please.
(0) Install test extension(named WinBackMyTrash), Show Menu Bar, customize ToolBar, add two buttons to Menu Bar(right side of Help), open Error Console.
(1) Select Inbox at folder pane, click button-1, and check Error Console.
(2) If trahFolderName is undefined, click button-2 with Inbox selected.
    => long text is written to Error Console(known workaround of that bug).
(3) Click butto-1 again with Inbox selected.
(4) Go bottom of Error Console.
    If trahFolderName is normaly shown,
    select a folderyou want to use as trash(call ABC/DEF), and click button-1.
    If trahFolderName is still undefined, Goto step (2).

Comment 14

[WADA:World Anti-bad-Duping Agency]

Attachment: Test extension named WinBackMyTrash 0.0.2 : namespace is cared upon trashFolderName change

This is new version of test extension which was attachd to bug 533140.
Following is newly done:
- check namespace setting,
- check whether mbox is subfolder of namespace,
- check whether mbox is actual namespace folder or not,
- check whether parent of mbox is actual namespace folder or not,
- if selected folder is subfolder of second or later mboxr of
  personalNamespace or subfolder of public/otherUsers Namespace,
  reject it.
- if selected folder is subfolder of first mbox of personalNamespace,
  remove namespace part from string set in trash_folder_name of prefs.js.

Remaining problem:
If "IMAP server directory: XXX where XXX is not Inbox" is intensionally specified by user when XXX is mbox of namespace, test extension currently fails to reject server folder of /XXX/YYY, /XXX/YYY/ZZZ, because folder is presented as /YYY or /YYY/ZZZ to who accesses msgFolder object.
In this case, because Tb currently always adds <first_mbox_of_personalNamespace> to string in trash_folder_name of prefs.js, it's impossible to use selected folder under a namesspace as trash folder of Tb.
However, this is never big problem, because majority is "IMAP server directory=INBOX" when personalNamespace = "INBOX." only(or "INBOX/" only). I can imagine use case of "IMAP server directory: publicNamespace or otherUsersNamespace".

Test prcedure.
(1) Install test extension, add customized ToolBar button to Menu Bar(two buttons)
(2) Open Error Console
(3) Select a folder at folder pane
(4) Click button-2 several times in order to force long log in Error Console
    (bypass of Bug 8316649)
(5) Click button-1
    => if appropriate folder, icon is changed to trash-can-icon
    => if appropriate folder and Bug 8316649 is bypassed,
       trash_folder_name of prefs.js is updated as expected.
    Check trash_folder_name via Config Editor.
    Go to bottom of Error Console and check messages from test extension.

Note:
By using string in msgFolder.URI instead of strin in msgFolder.prettiestName, problem when "localized special folder name by localized Tb" is already worked around by test extension.

Comment 15

[WADA:World Anti-bad-Duping Agency]

type correction.
I can NOT imagine use case of "IMAP server directory: second or later personalNamespace or publicNamespace or otherUsersNamespace".

Comment 16

[WADA:World Anti-bad-Duping Agency]

Attachment: Test extension named WinBackMyTrash 0.0.3 : multiple namespaces, any namespace type, serverDirectory, are also cared upon trashFolderName change

Mutiple namespaces in any of personalNamespace/publicNnamespace/otherUsersNamespace, serverDirectory, are additionally cared.

Because Tb's spec on Trash folder is as follows;
  Trash folder used by Tb = Default_namespace + trash_foldr_name in prefs.js
                            where Default_namespace = First personalNamespace
following is done upon selected Mbox checking for Trash use:

(A) In any case, following folder is rejected.
    (this is not done by folder selection UI of Tb)
  Folder of Inbox/Drafts/SentMail/Templates/Junk/Archive/Queue flag.
  Trash is not rejected because it's set as Trash already.

(B) In any case, folder of \Noselect is rejected.
    (this is already done by folder selection UI of Tb)

(C) In any case, folder of canFileMessages=false or canCreateSubfolders=false is rejected. (I don't know this is done by folder selection UI of Tb or not)

(D) If namespace is not used, no limitation in trash folder selection except (A)/(B)/(C)/(D). No problem in setting Mbox name in msgFolder.URI to trash_folder_name of pres.js.

(E) In any case, namespace folder(folder of isNamespace=true) is rejected, because namespace folder is never appropriate for Trash.

(F) If personalNamespace is used, subfolder held under other than Default_namespace folder(==First personalNamespace folder) is rejected.

(G) Default_namespace part is removed from Mbox name in msgFolder.URI upon trash_folder_name setting if serverDirectory != Default_namespace.

(E)/(F)/(G) is not cared  by folder selection UI of Tb.
Due to lack of (F), following problem currently occurs.
>     <other_than_first_personalNamespace>/<subfolder_name_for_trash> is selected as trash at trash selection UI.
>  => trash_folder_name = <other_than_first_personalNamespace>/<subfolder_name_for_trash> is set by folder selection UI.
>  => "Default_namespace + trash_folder_name in prefs.js" is always Trash for Tb if personalNamespace is used.
>  => "imap://<serverURI>/Default_namespace/<other_than_first_personalNamespace>/<subfolder_name_for_trash>" is created and used by Tb.
If folder of /Defaul_namespace doesn't exist at server, by "create" command for above Mbox, Italic "Default_namespace" is generated(folder of \Noselect, can't hold messages in it, can hold subfolders only).
This problem(creation of unwanted/unexpected folder under Default_namespace) occurs too, even if <other_than_first_personalNamespace> part is removed from trash_folder_name string set in pefs.js.
This problem is reason why (F) is required.

In new version of my test extension, following case is cared.
>  namespace1=ABC and namespace2=ABC/DEF (subfolder of other namespace is specified as namespace) 
>    ABC is Inbox, or ABC s not Inbox
>    namespace1 is Default_namespace, or namespace2 is Default_namespace
>    serverDirectory=Default_namespace, serverDirectory=Non_Default namespace
>  Note: Bad useage of namespace, but Tb is pretty torelant above bad usage.
In any case, "subfolder held under other than Default_namespace folder(==First personalNamespace folder)" is normally rejected

Comment 17

[WADA:World Anti-bad-Duping Agency]

To resolve this kind of problem in trash folder selection, one of followings is needed.
(1) (1-a) Keep spec of trash_folder_name in prefs.js as-is,
    (1-b) refine "trash selection UI logic".
          (my test extension is to help this change of 1-b)
(2) (2-a) Chane spec of trash_folder_name to one like Copies&Folders/Junk Settings,
    (2-b) change folder selection UI and his logic according to design change,
    (2-c) change logic of code which uses trash_folder_name.
In other bug(s), (2) was considered in the past, but it's never decided yet, mainly due to compatibility issue and workload issue.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Following was partially inaccurate and slightly misleading.
>  "Default_namespace + trash_folder_name in prefs.js" is always Trash for Tb if personalNamespace is used.

(i) If personalNamespace is used, folder creation request without selecting specific folder at folder pane(i.e. account is selected, folder of msgFolder.isServer==true is selected), Tb always creates folder under Default_namespace folder(if multiple namespaces are used, RFC looks to request to give a chance user to choose namespace, but Tb looks to use Default_namespace folder always.)
Creation of folder in (i) is probably done via incomingServer.getUriWithNamespacePrefixIfNecessary().

(ii) When folder in trash_folder_name of prefs.js is used, request from Trash user is perhaps similar to one like next.
   With rootFolder is current folder(isServer==true folder==account),
     create folder named trash_folder_name,
     with option of "use existent one if already exists".
This is perhaps also done via incomingServer.getUriWithNamespacePrefixIfNecessary().
This is not intentional namespace use by Trash requester side. Trash requester side simply requests folder named trash_folder_name.

So, it's apparently fault of Trash folder selection UI. Trash folder selection UI should use getUriWithoutNamespacePrefixIfNamespaceIsUsed() like one upon setting trash name in trash_folder_name.

If Trash requester side logic is slightly changed to one like next,
  If trash_folder_name starts with protocol, use this as trash folder URL.
     trash_folder_name = imap://username@hostname/ABC/DEF/Trash
     Same as Copies&Folders/Junk Settings
  Else if trash_folder_name is valid string as folder name, do as currently done.
     trash_folder_name = Inbox/Trash
     Same as current
Tb can work well with both style of trash_folder_name.
If this change is first applied to recent Tb, and if Trash selection UI change & trash_folder_name spec change will be done in later release of Tb, compatibility problem with recent releases won't occur.

I don't think "Trash selection UI side logic change to remove Default_namespace part from string set in trash_folder_name" is pretty good solution of this bug.
I think following is better.
(1) Change trash_folder_name design/implementation by a release of Tb.
    Support both "imap:// ..." style and "Inbox/Trash" style.
(2) Change Trash folder selection UI to same one as Copies&Folders/Junk Settings by next or later release of Tb, in order to resolve this bug and some other bugs.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Needless to say, following is needed ASAP.
   Disable trash folder selection UI,
     if localized Tb whose trash!="Trash" : for bug 480393
     if personalNamespace is used         : for this bug
   By this change, both bugs automatically disappears,
   because responsibility of trash_folder_name setting is transferred to user.

Comment 20

[WADA:World Anti-bad-Duping Agency]

I've opened bug 836631 for a possible solution of bug 480393 and bug 491424.

Comment 22

[Jon Grossart]

How has this bug not been fixed in 4.5 years? I know Tb is lacking in dev resources, but in that same time it's had features like instant messaging added, but not a fix for basic IMAP functionality/broken UI.

Thanks WADA for doing all the debugging work.

Comment 24

[Jorg K (CEST = GMT+2)]

Should be fixed by bug 1335982, bug 1428666 and bug 1427507.