556113 - If "Sent" folder of account is set to an unsubscribed IMAP folder, TB silently uses arbitrary Sent folder instead

Status: UNCONFIRMED

(Thunderbird :: Untriaged, defect)

Description

[Philippp]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3

In a small workgroup, I try to put messages in folders common to all users so that anyone can see outgoing messages. I configured multiple identities; for the main identity (i.e. the account), I set the "save copy of sent messages" to a shared IMAP folder, which generally works. But, if the user accidentally unsubscribes from that folder, and then sends a message, that message is silently saved to the "Sent" folder of the first ID instead.

Reproducible: Always

Steps to Reproduce:
1. Set the "Sent" folder of an account to a shared IMAP folder, f.i. GlobalOut.
2. Create an additional identity
2. Set the "Sent" folder of that identity to folder XXX.
3. Unsubscribe from folder GlobalOut.
4. Send a message using main identity (account).
Actual Results:  
5. The message is silently saved to folder XXX.

Expected Results:  
5. The folder GlobalOut should be subscribed to automatically (as with ids, see below), and the message should be saved there.

If the same happens to an identity (i.e. the "Sent" folder of an identity is not subscribed), and you send a message using that id, then the "Sent" folder is automatically subscribed, and the message is filed there. Just perfect!
The same if you archive a message while the "Archives" folder is not subscribed. Perfect!
I would wish that TB does the same in the described case.

(A way around this could be to prohibit unsubscribing from any folder that is configured to be "Sent" anywhere, but please do NOT do that. My users have many identities configured in advance, and they only subscribe to the matching folders as they need.)

Comment 1

[WADA:World Anti-bad-Duping Agency]

Unsubscribing of an IMAP folder while Tb is running is same as "delete of the  folder" from point of IMAP folder accessing view. If "folder to save sent mail copy" is not found, Tb tries to create it. For example, if Archives doesn't exist for Tb, Tb requests "create Archives"(==ordinaly, Personal Namespace, already exists response), and requests "subscribe Archives"(successful). It's probably trick of other than non "shared IMAP folder" case.

What namespaces is used for "Public(Shared) Namespace" by your IMAP server? Which is used as hierarchy delimiter by your IMAP server? "/"? "."?
> http://tools.ietf.org/html/rfc3501
> http://tools.ietf.org/html/rfc2342
Can you get IMAP log? 
> https://wiki.mozilla.org/MailNews:Logging
> http://www.mozilla.org/projects/nspr/reference/html/prlog.html#25328
> SET NSPR_LOG_MODULES=timestamp,imap:5

Comment 2

[Philippp]

Ah, I see - yes, seems to be in the config - see below.
Delimiter is "/", pls. also see the log.

To reproduce:
1.) In TB, configure IMAP mailbox of bob:

1.1) Create a folder "Sent Items of bob2"
1.2) IMAP account (main identity) "bob":
For "Sent" select radio "(o) other folder", explicitly choose "Sent Items" in bobs mailbox
1.3) Create an additional identity "bob2":
For "Sent" select the folder "Sent Items of bob2" in bobs mailbox

2.) Now, unsubscribe from "Sent Items".

3.) Send a message from main identity ("bob")

4.) The sent message is filed to "Sent Items of bob2" instead of "Sent Items".
The folder "Sent Items" remains unsubscribed.

5.) Open account settings, look at the setting of "Sent" folder of main account:
It has changed to "Sent Items of bob2"!
Click Cancel.

6.) Close TB, Re-start TB, watch the tree:
Now, "Sent Items" suddenly is subscribed (with yellow icon, not the blue stamp)!
A few seconds later, the yellow icon turns into the blue stamp.

7.) Look at account settings: "Sent" folder of main account still is "Sent Items".


HA!!! The problem does not happen if in step (1.2) you select the radio button
"(o) Folder "sent" in [[e-mail]]" instead of specifying the folder explicitely, as described before.

So, I guess, the error might be related to the "folder picker mode" pref item!?
The picker mode refers to the radio buttons, is that right?
Probably the error only happens if mail.identity.id1.fcc_folder_picker_mode = 1?

(Obviously, the option to use the standard "Sent" folder is not a workaround if you want to use a different name.)

[Btw, I noticed that the folder icons (yellow folder/blue stamp) are not updated until you move the mouse over the tree row.
Perhaps relates to bug #497288? I left a comment there.]

Comment 3

[Philippp]

Attachment: IMAP log when problem occurs.

Attached IMAP log is from steps 1.) to 5.)

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> 1.1) Create a folder "Sent Items of bob2"
> 1.2) IMAP account (main identity) "bob":
> For "Sent" select radio "(o) other folder", explicitly choose "Sent Items" in bobs mailbox
> 1.3) Create an additional identity "bob2":
> For "Sent" select the folder "Sent Items of bob2" in bobs mailbox
> 2.) Now, unsubscribe from "Sent Items".

Question for ease of understanding of log, and to avoid my misunderstanding.

In your comment #0, main identity's sent folder which was unsubscribed was shared folder(public folder), and next is returned by your server.
> 5 namespace
> * NAMESPACE (("" "/")) NIL (("Public Folders/" "/"))
> 5 OK NAMESPACE command completed
However, your "steps to reproduce" in comment #2 looks irrelevant to Public(shared) Namenpace or not, because folder of "Sent Items" is seen instead of "Public Folders/Sent Items". Does it mean "Private folder or Public(Shared) folder" is irrelevant to your problem?

By the way, finding of "Sent Items" only in log(when both "Sent Items" and "Sent Items of bob2" exist in log too) is hard for me using text editor, because GREP is not supported by my text editor. And, space in name of something is good for readability and understanding of meaning of string, but space in search string is merely a blocker of easy search.
Could you please use "Sent-bob-1" / "Sent-bob-2" like string as folder name for ease of log search? (if both is reuired, search for Sent-bob can be used)

Comment 5

[Philippp]

Hi WADA,

to simplify things, I used all personal folders in the testcase in my comment #2.
It seems irrelevant where the folders are (personal or public). My impression is that the config items get mixed at some point.

Yeah, those spaces... infiltrated from redmond... ;-)
I will see if I can make a new log for you...

Comment 6

[Philippp]

Hi again,
new test case, less sleep, new findings - even more strange:

1.) Created Sent-bob-1, Sent-bob-2.

2.) In Account GUI, set main account "bob": Sent to "Sent-Bob-1",
identity "bob2": Sent to "Sent-bob-2".

3.) Deleted "Sent Items" folder and "Sent Items of bob2".

4.) Restarted TB, noticed that "Sent Items" is still there, having a blue "stamp" icon!
Checked account GUI: Id does correctly show settings according to step 2.)
Checked in pref-editor to be sure (search for "fcc"), it says:

mail.identity.id1.fcc_folder;imap://bob@localhost/Sent-bob-2
mail.identity.id1.fcc_folder_picker_mode;1
mail.identity.id2.fcc_folder;imap://bob@localhost/Sent-bob-1
mail.identity.id2.fcc_folder_picker_mode;1

(I wounder why the main account is id2, not 1? I had created the main account ("bob") first, and then added the identity "bob2", so I would have expected bob2 being id2.)

5.) "Sent Items" still has the blue stamp (See screenshot). Probably this is another bug, so lets go ahead. "Sent Items" cannot be deleted - annoying. Restarting TB does not help.

6.) Send messages from bob1 and bob2, messages are filed to the correct Sent folders (IMAP-bob-1.log).

7.) Unsubscribe "Sent-bob-1". Send a messages from bob, the message is filed in "Sent Items" instead of "Sent-bob-1" (IMAP-bob-2.log).
This is even more spooky than in my previouos test!
Where does TB take the "Sent Items" from? There was no identity that had this setting!
I look in the prefs editor: Indeed, it says
mail.identity.id2.fcc_folder;imap://bob@localhost/Sent Items

So when does this happen?
I might think I found it: In the moment when I unsubscribe "Sent-bob-1", the FCC pref changes to "Sent Items".
--> Question A: Why does it change the pref setting? It would be perfect to re-subscribe the folder on demand (as you said in comment #1).
--> Question B: Why does it choose exactly "Sent Items"? This name is nowhere in the prefs! (I searched prefs.js with text editor before closing TB.)


What happens if the folder was already unsubscribed when TB is started?
8.) I close TB, edit prefs.js to according to settings after step 4.
9.) Start TB: In the tree, "Sent Items" still has blue icon. Prefs editor shows settings like after step 4, so this would be ok.
10.) Send message from bob - now it works as expected - the message is filed under "Sent-bob-1", which was subscribed automatically (IMAP-bob-3.log).

--> Surprise - interesting that it works that way - I know that it had failed in that case, too - I will try again:

11.) I close TB, re-start TB. Look at prefs-editor: Now it says:
mail.identity.id2.fcc_folder;imap://bob@localhost/Sent Items
Voila, there we are!

--> Question C:  Why is the setting changed silently?

How many bugs does that make??

Comment 7

[Philippp]

Attachment: IMAP-bob-1.log

Comment 8

[Philippp]

Attachment: IMAP-bob-2.log

Comment 9

[Philippp]

Attachment: IMAP-bob-3.log

Comment 10

[Philippp]

Attachment: Screenshot: Wrong blue stamp icon on "Sent Items"

Comment 11

[WADA:World Anti-bad-Duping Agency]

Why "Sent Items" is always subscribed and has icon for "sent folder", and why "Sent Items" is used as fall back folder when "folder to save sent mail copy" is not availble, is probably next. (XLIST can be said "LIST + LSUB", from view point of meaning of response and response usage) 
> 6 xlist "" "%"
>(snip) 
> * XLIST (\Sent) "/" "Sent Items"
> * XLIST () "/" "Sent-bob-1"
> * XLIST () "/" "Sent-bob-2"
> 6 OK XLIST completed

Comment 12

[Philippp]

Thanks WADA,

not sure if I fully understand(*) (me no IMAP expert, just admin). On the production system, TB falls back to a totally different folder (the sent folder of the first additional identity - that folder is somewhere in the "Public Folders" area!).

[ (*) Does "XLIST (\Sent)" imply there is a way for IMAP client to query the name of the "Sent" folder of an IMAP account? If yes, then this does not work, because I always got the "Sent" vs. "Sent Items" confusion resulting in both being created, until I manually configured TB to use "Sent Items" (which btw. lacks localization), bug #543227. Sorry, perhaps off-topic question.]

I remember having similar icon issues with other "special" folders, f.i. Archives.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
> [ (*) Does "XLIST (\Sent)" imply there is a way for IMAP client to query the
> name of the "Sent" folder of an IMAP account?

Conceptually, yes, but slightly different for Tb + Gmail.
XLIST was implemented by Gmail, and XLIST is for "use folder at IMAP server listed in XLIST responce always for special folder".
  Gmail changes IMAP folder name according to Display Language setting.
    Trash : En(US)=[Gmail]/Trash, En(UK)=[Gmail]/Bin,
            Deutsch=[Gmail]/Papierkorb, French=[Gmail]/Corbeille
    (if non-ascii, modified utf-7 in XLIST response according to IMAP RFC)
  Folder name change at server due to Display Language change can occur
  while Tb is using with trash folder for other Display Language.
  So, Tb always tries to use above "localized IMAP folder at server" as "trash".
  See bug 476260 for XLIST support.
  It's also applied to other special folders of Tb.
It produced some mismatch'es with traditional concept of "any IMAP folder as special folder for Tb"(selectable via folder selection UI for trash, junk, and folders in Copies&Folders)
  For example, Gmail IMAP user can not use other folder than [Gmail]/Trash
  (Trash part is localized by Gmail), even if user selected other folder
  for "trash folder", even though specifiled folder has "icon for trash".
  "can not use" means "Empty Trash" is always executed on [Gmail]/Trash.
I think your case is one of some mismatch'es.
  If Sent, folder selected at folder selection UI is used as expected
  even when XLIST is used. But in special situation such as fall back,
  folder in XLIST response is used, because "\Sent" folder in XLIST response
  is also a valid folder to save sent mail copy.  

If you need to use "different folder from one listed in XLIST response" in many situations, disable XLIST at server side, please, because Tb desn't have capability to ignore XLIST yet.

By the way, if you need localized special folder name for any user, for any special folder, set the localized name at server in XLIST response("in modified utf-7" may be required.) The IMAP folder of localized name is always used by Tb.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> 5.) "Sent Items" still has the blue stamp (See screenshot). Probably this is
> another bug, so lets go ahead. "Sent Items" cannot be deleted - annoying.
> Restarting TB does not help.

It's probablly because your server returns * XLIST (\Sent) "/" "Sent Items".

> 7.) Unsubscribe "Sent-bob-1". Send a messages from bob, the message is filed in
> "Sent Items" instead of "Sent-bob-1" (IMAP-bob-2.log).
> This is even more spooky than in my previouos test!
> Where does TB take the "Sent Items" from? There was no identity that had this
> setting!
> I look in the prefs editor: Indeed, it says
> mail.identity.id2.fcc_folder;imap://bob@localhost/Sent Items

It looks two different phenomena.
  (A) Upon save error, fall back to \Sent folder in XLIST
      (saved in "Sent Items" in your previous test).
  (B) If folder specified in idX.fcc_folder doesn't exist,
      change setting to a folder which has attribute of "sent folder".

(1) IMAP account1(Gmail IMAP)
    * XLIST (\Sent) "/" "[Gmail]/Sent Mail"
(2) main identity
> mail.identity.id1.fcc_folder_picker_mode=1 (Others:)
> mail.identity.id1.fcc_folder=imap://[e-mail]/A
  Root-level folder of "A" has "icon for sent folder" by this change.
(3) Via config editor, change(corrupt) fcc_folder name
> mail.identity.id1.fcc_folder=imap://[e-mail]/ABCDEFG
  ABCDEFG doesn't exist.
(4) Restart Tb, send mail, message of copying to ABCDEFG.
    Sent mail is saved in [Gmail]/Sent Mail (by XLIST response)
(5) check prefs.js entry via Config Editor
> mail.identity.id1.fcc_folder=imap://[e-mail]/A

After restart, relation between id1 and folder "A" never exist in prefs.js. So, I guess the root-level folder "A" is selected from folders which has characteristics of "sent folder".
(in your previous test, "Sent Items". in comment #0, another identity's one.)
It may be auto-correction of corrupted Copies&Folders folder setting.
(fcc/draft/junk folder of deleted account, of hidden account, ...)

Comment 15

[Philippp]

(In reply to comment #13)
> (In reply to comment #12)
> > [ (*) Does "XLIST (\Sent)" imply there is a way for IMAP client to query the
> > name of the "Sent" folder of an IMAP account?
> 
> Conceptually, yes, but slightly different for Tb + Gmail.

[...]

Wow!! Looks like a terrible mess at first sight. But has good reason, if it does work, for standard users.

BIG thanks for your explanation!
I had spent hours trying to figure out when and why TB did not apply my "special folder" selections, and why I was getting duplicate/non-localized folders, wrong icons etc. I had finally given up the localization (and considered TB very buggy for that - I should change my bug #543227 then).

So is it the purpose of the radio buttons (and "folder_picker_mode" prefs) to decide whether to use the server-supplied (XLIST) names or not? The GUI should indicate more clearly that the upper ones are server-supplied names. I thought they were just a kind of static "quick-access" to the default english folder names!
Ok, so this explains one major part of the weirdness (folder names)!

I will have to check whether it is possible to disable XLIST on my kerio mail server, and/or use localized folder names on the server - at least, now I know where to search for that ghost!
(I fear that using the localized folder names on the server might conflict with kerio webmail and/or kerio outlook connector, but I will see.)

Again, thanks a lot!

Comment 16

[Philippp]

(In reply to comment #14)

Ok, now this makes some sence to me! :-)

I revise my complaint as follows:
It is generally a good idea to fall back based on reasonable assumptions, but there must be a notification to the user! (In my case, putting sent messages into the wrong folder is close to data-loss, because a user not finding their data has a similar experience as a user having lost their data [we cannot use gloda for storage/performance reasons].)

I think the only silent fallback in this case should be to try to subscribe or create the specified folder, as TB does it already in most cases. This is perfect, no matter whether it is an XLIST-based folder or a user-specified folder).
If there needs to be any additional fallback (for reason of gMail), TB *must* ask the user, or at least notify them, that something was probably done differently than specified (and whether this has caused permanent changes).

Comment 17

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> If there needs to be any additional fallback (for reason of gMail), TB *must*
> ask the user, or at least notify them, that something was probably done
> differently than specified (and whether this has caused permanent changes).

Needs of "asking user for fall back folder" is not for Gmail.
If IMAP server down, there is no way to save "sent mail copy" currently - if reply Cacel to dialog, sent mail copy is lost, and if reply Retry to dialog, user has to reply Retry until IMAP server will come back. Tb should ask for fall back location(local mail folder if IMAP server down) to protect user from loss of "sent mail copy" in the worst case.
It's already requested enhacement.
Because silent fall back can cause loss of mail for user(user frequently can't find mail even if predefined fall back location is defined by the user), I think "silent fall back to predefined fall back location" should be limited to "background mail sending".

Comment 18

[Philippp]

(In reply to comment #17)
> If IMAP server down, there is no way to save "sent mail copy" currently - if
> reply Cacel to dialog, sent mail copy is lost, and if reply Retry to dialog,
> user has to reply Retry until IMAP server will come back. Tb should ask for
> fall back location(local mail folder if IMAP server down) to protect user from
> loss of "sent mail copy" in the worst case.
> It's already requested enhacement.

Yes, it is certainly the best solution to ask the user for an alternative location, if specified location is unreachable (due to server down, or whatever).

But, the order should be consistent:
1.) Always try to auto-subscribe or auto-create the specified location;
 if that fails, then
2.) fallback / ask user to specify alternative location.

So, in my opinion, the current bug is that sometimes TB seems to do 2.) without having tried 1.) first.

> Because silent fall back can cause loss of mail for user(user frequently can't
> find mail even if predefined fall back location is defined by the user),

I fully agree!

> I think "silent fall back to predefined fall back location" should be limited
> to "background mail sending".

What do you mean by "background mail sending"?

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #18)
> > I think "silent fall back to predefined fall back location" should be limited
> > to "background mail sending".
> What do you mean by "background mail sending"?

I meant feature(s) around mailnews.sendInBackground=true/false.
AFAIK, it's implemented by bug 440794. But the new feature was deaulted to off because of problems around error handling and/or issues around error recovery.
> Bug 440794 Initial Support for "Leverage Offline capabilities
> to make sending email appear faster" (aka send in background)