Open Bug 697789 Opened 13 years ago Updated 2 years ago

Thunderbird keeps recreating #news^#news folder and Draft mailbox

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Platform:
x86_64
Windows 7
Type:
defect

Tracking

(Not tracked)

People

(Reporter: bigheadjer, Unassigned)

References

(Depends on 2 open bugs, Blocks 1 open bug)

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

This occurs randomly.


Actual results:

Thunderbird constantly recreates a folder called "#news^#news" on my IMAP server with a Drafts mailbox inside it.  Each time I right-click and unsubscribe from it, it reappears within 24 hours.


Expected results:

This folder should not appear at all.
Note appearance of this folder occurs with a popup error saying "Error saving to Drafts folder."
(In reply to Jeremy Tan from comment #1)
Check bug 517462.
(In reply to Jeremy Tan from comment #1)
> Note appearance of this folder occurs with a popup error saying "Error
> saving to Drafts folder."

Can you manually save mail as draft to folder which is set by you as "folder to save draft mail" at Copies&Folders setting of any identity(mail address which can be used as From: of composed mail)?
(In reply to Hashem Masoud from comment #2)
> (In reply to Jeremy Tan from comment #1)
> Check bug 517462.

Wow, according to that, which links to bug 503735, this bug has been around since 2009!
(In reply to WADA from comment #3)
> Can you manually save mail as draft to folder which is set by you as "folder
> to save draft mail" at Copies&Folders setting of any identity(mail address
> which can be used as From: of composed mail)?

When I manually save a draft email, it gets saved to the proper folder which I have set in Copies & Folders.  The popup error from comment 2 only occurs during autosaving it seems.
(In reply to Jeremy Tan from comment #0)
> Thunderbird constantly recreates a folder called "#news^#news"
> on my IMAP server with a Drafts mailbox inside it.

Where was the "#news" originally come from?
Did you create IMAP folder named "#news" once?

Setting dependency to bug 720911 for ease of anlysis and tracking.
Depends on: 720911
No I never created it myself.  The first time it was created was when I ran Thunderbird for the first time to my IMAP server.

I used to use two IMAP clients (one now, I had to stop using Thunderbird due to another bug), Apple Mail and Thunderbird.  Apple Mail never created any such folder, while each time I used Thundebird, the #news^news folder would get created on the IMAP server.  Now that I've stopped using Thunderbird, the folder no longer gets created randomly, so it's definitely Thunderbird that's causing this.
(In reply to Jeremy Tan from comment #7)
> No I never created it myself.  The first time it was created was when I ran
> Thunderbird for the first time to my IMAP server.
> I used to use two IMAP clients (one now, I had to stop using Thunderbird due
> to another bug), Apple Mail and Thunderbird.  Apple Mail never created any
> such folder, while each time I used Thundebird, the #news^news folder would
> get created on the IMAP server.  Now that I've stopped using Thunderbird,
> the folder no longer gets created randomly, so it's definitely Thunderbird
> that's causing this.

Did #news or folder whose name contains # already exist at IMAP server before start of Tb use?  
If actually created by Tb, which is actually created folder name at IMAP server by Tb wrongly?
  #news^#news  (in comment #0)
  #news^news   (in comment #7)
Or #news, #news under #news, ^news under #news like one? (bug 720911)

Where the string of "#news" or "#" come from in your case?
Is there "#" in profile directory path? (bug 725555)
Is there "#" in folder/subfolder/newsgroup/feed name? (bug 720911)
Is there "#" or "#news" in your prefs.js? (for example, #news in account's name)

"With a Drafts mailbox inside it" indicates that garbled folder is perhaps created upon Drafts folder access. Did Drafts folder already exist at IMAP server before you started to use Tb?

Tb has issue of "tries to access IMAP folder of account's label and creates it". If account's label is #news and such problem happens, #news may be created. And once #news is created, bug 720911 surely occurs and problem of bug 720911 continues forever until garbled data is manually cleaned up.
If IMAP folder of account's label was created by Tb upon Drafts folder access, it's only when Drafts folder which is requested by Copies&Folders doesn't exist at IMAP server.
And "^" in garbled folder name was seen only when namespace is used and delimiter is dot instead of slash.
"With a Drafts mailbox inside it" is simply a phenomenon after bug 720911.
(i) Because "#news^#news" was created, bug 720911 occurs. So root == #news^#news.
(ii) mail.identity.idX.draft_folder = imap://userName@hostname/Drafts
Because Root is changed to #news^#news, actual Drafts at IMAP server is /#news^#news/Drafts for Tb. So Tb can't find folder set in draft_folder.
(iii) Because draft_folder is not found, Tb tries to create "Drafts" which is obtained from draft_folder setting upon draft save.
(iv) Due to bug 720911, create is requested for /#news^#news/Drafts.
(v) Because created folder is not "Drafts" which is specified in draft_folder, draft save fails.
(vi) Even if #news^#news is manually unsubscribed/deleted, (iii)/(iv) occurs again sooner or later. 

Any problems after creation of "#news^#news" is dup of bug 720911.
Problem in your case is "Why/how the garbled IMAP folder was generated by Tb".
I guess problematic string is "#news" in your environment. And I suspect "^" in "#news^#news" is a result by "delimiter=dot" or "namespace=Inbox. with delimiter=dot".

Do you still need to stop using Tb due to other problems?
If no, can you test with Tb?
After "manual cleanup of garbage by bug 720911", Tb works as expected unless you intentionally create IMAP mail folder of "# in folder name".
(In reply to Jeremy Tan from comment #7)
> the #news^news(or #news^#news) folder would get created on the IMAP server.

I could observe following IMAP log for "# and ^ in folder name" with Tb 9.0.1, with Gmail IMAP(# is allowed but ^ is not allowed in Gmail Label), by forcing namespace="#XYZ#/".  
> :imap.gmail.com:S-INBOX:ProcessCurrentURL:imap://yatter%2Eone%40gmail%2Ecom@imap.gmail.com:993/create%3E/%23XYZ%23%5ESent/AAA:  = currentUrl
> :imap.gmail.com:S-INBOX:SendData: 9 create "#XYZ#/#XYZ#^Sent/AAA" 
> :imap.gmail.com:S-INBOX:CreateNewLineFromSocket: 9 NO [CANNOT] Folder name is not allowed. (Failure)

Following data was generated in ...\ImapMail\imap.gmail.com.msf by Tb, as done in bug 720911.
> <(80=1)(82=#XYZ#^Sent)(87=40404400)>

Once #XYZ#^Sent is generated in .msf file by bug 720911, creation request of root level AAA becomes creation request of #XYZ#^Sent/AAA, and if namespace="#XYZ#/" is used, it is always changed to creation request of #XYZ#/#XYZ#^Sent/AAA.

If namespace of "#news/" or "#news." is used by server, it should be shown as root level #news in Tb's folder pane.
Is namespace used by your server? Do you set "IMAP Directory: #news" at Server Settings/Advanced?
(In reply to WADA from comment #8)
> (In reply to Jeremy Tan from comment #7)
> > No I never created it myself.  The first time it was created was when I ran
> > Thunderbird for the first time to my IMAP server.
> > I used to use two IMAP clients (one now, I had to stop using Thunderbird due
> > to another bug), Apple Mail and Thunderbird.  Apple Mail never created any
> > such folder, while each time I used Thundebird, the #news^news folder would
> > get created on the IMAP server.  Now that I've stopped using Thunderbird,
> > the folder no longer gets created randomly, so it's definitely Thunderbird
> > that's causing this.
> 
> Did #news or folder whose name contains # already exist at IMAP server
> before start of Tb use?  

No it did not exist until after I started using Tb.

> If actually created by Tb, which is actually created folder name at IMAP
> server by Tb wrongly?
>   #news^#news  (in comment #0)
>   #news^news   (in comment #7)
> Or #news, #news under #news, ^news under #news like one? (bug 720911)

Sorry for the confusion, it was #news^#news.

> Where the string of "#news" or "#" come from in your case?
> Is there "#" in profile directory path? (bug 725555)

Yes.

> Is there "#" in folder/subfolder/newsgroup/feed name? (bug 720911)
> Is there "#" or "#news" in your prefs.js? (for example, #news in account's
> name)

Yes they appear here:

user_pref("mail.server.server1.namespace.personal", "\"#mh/\",\"#mhinbox\",\"\"");
user_pref("mail.server.server1.namespace.public", "\"#public/\",\"#news.\",\"#ftp/\",\"#shared/\"");

However, I did not add them myself.

> "With a Drafts mailbox inside it" indicates that garbled folder is perhaps
> created upon Drafts folder access. Did Drafts folder already exist at IMAP
> server before you started to use Tb?

No, the Drafts folder did not exist within #news^#news, it existed in my mail subdirectory on the server.
(In reply to WADA from comment #9)
> Do you still need to stop using Tb due to other problems?

Yes I can't use Tb right now because of https://bugzilla.mozilla.org/show_bug.cgi?id=413240.

> If no, can you test with Tb?
> After "manual cleanup of garbage by bug 720911", Tb works as expected unless
> you intentionally create IMAP mail folder of "# in folder name".

Once that bug gets resolved I will test this as well.
(In reply to Jeremy Tan from comment #12)
> Yes I can't use Tb right now because of bug 413240

Does bug 413240 occur on every mail send in your environment?
Is sent mail actually saved in correct Sent folder upon every mail send but "Copying to sent folder" never disapppears upon every mail send?  
(I can't imagine environment where bug 413240 occurs upon every mail send, because that bug can't be reproduced intentionally and occurs accidentaly.)
> Bug 413240  : after sending message,
> "Copying to sent folder" doesn't finish even after Save to Sent is successfull,
> or zombie "Copy complete" is generated after successfull Save to Sent
Or do you mean "save mail to Sent after send always fails or is not executed always"?

In both cases, it may be relevant to #XYZ#^Sent like one which is seen in my IMAP log.
> create%3E/%23XYZ%23%5ESent/AAA:  = currentUrl
> 9 create "#XYZ#/#XYZ#^Sent/AAA"
I never did send operation nor "copy to Sent folder" operation. As for Sent folder, I merely opened Copies&Folders setting panel. This log indicates that Tb wrongly generates internal path for Sent folder requested at Copies&Folders setting.

Main reason why you can't use Tb is bug 720911. Because "#news^#news" was already created once by Tb and its name is held in <servername>.msf, bug 720911 continues to occur forever until garbage by bug 720911 is manually cleaned up.

This bug's particularity: 
Because "#" is used in Mbox name for namespace, folder of "# in folder name" is created sooner or later even if 'folder of "# in folder name" used as namespace' doesn't exist at server. So bug 720911 surely occurs always.
(Note: "folder used as namespace doesn't exist at server" is usually called "server configuration error" in our country.)

So, we can say "you can't use Tb due to this bug".

"^" in "#news^#news" and "#XYZ#^Sent" part is apparent Tb's actual bug found in this bug. This is different issue from bug 720911 although relevant.
Can you get IMAP log of TB in your environment with new profile and one IMAP account only?

By the way, simplest workaround is "no # in folder name". Can you change server configuration?
(In reply to Jeremy Tan from comment #11)
> Yes they appear here:
> user_pref("mail.server.server1.namespace.personal",
> "\"#mh/\",\"#mhinbox\",\"\"");
> user_pref("mail.server.server1.namespace.public",
> "\"#public/\",\"#news.\",\"#ftp/\",\"#shared/\"");
> However, I did not add them myself.

(In reply to Jeremy Tan from comment #11)
> Yes they appear here:
> user_pref("mail.server.server1.namespace.personal",
> "\"#mh/\",\"#mhinbox\",\"\"");
> user_pref("mail.server.server1.namespace.public",
> "\"#public/\",\"#news.\",\"#ftp/\",\"#shared/\"");
> However, I did not add them myself.

These are writtten by Tb when NAMESPACE response is returned from IMAP server, when Server Settings/Advanced, "Allow server to override these namespaces" is Ckecked.
("Checked" is default, in order to use namespace requested by server)
("Unchecked" is for server of "server doesn't support namespace command well but server requires namespace=Inbox. or Inbox/)

> \"#news.\"

Is delimier of "/" and "." intentionally mixed at server?
Or it's Mbox named "#news." which can hold mails but can't hold subfolder?
(In reply to WADA from comment #13)
> (In reply to Jeremy Tan from comment #12)
> > Yes I can't use Tb right now because of bug 413240
> 
> Does bug 413240 occur on every mail send in your environment?

When I stopped using it, yes it was occurring on every mail send on that particular IMAP server.

> Is sent mail actually saved in correct Sent folder upon every mail send but
> "Copying to sent folder" never disapppears upon every mail send?  
> (I can't imagine environment where bug 413240 occurs upon every mail send,
> because that bug can't be reproduced intentionally and occurs accidentaly.)

Even if it occurs randomly, I can't use it as it is for my work email and I can't have my outgoing emails not being archived in the IMAP server's outbox correctly because I need to access mail from multiple locations.

> So, we can say "you can't use Tb due to this bug".
> 
> "^" in "#news^#news" and "#XYZ#^Sent" part is apparent Tb's actual bug found
> in this bug. This is different issue from bug 720911 although relevant.
> Can you get IMAP log of TB in your environment with new profile and one IMAP
> account only?
> 
> By the way, simplest workaround is "no # in folder name". Can you change
> server configuration?

No I do not have the authority to change the server configuration.  AFAIK it is a very vanilla UW-IMAP installation.
Is personal namespace of "#mh/", "#mhinbox" actually used and required?
Does Mbox named #mh, #mhinbox actually exist at server?
Does Mbox named "#public", "#news.", "#ftp", "#shared" actually exist on server? Actually used?
If no, following setting(Server Settings/Advanced) can be a workaround in your case.
  mail.server.server1.override_namespaces = false
  mail.server.server1.namespace.personal    = "" (or "INBOX/" or "INBOX.")
  mail.server.server1.namespace.public      = ""
  mail.server.server1.namespace.other_users = ""
Phenomenon was following.
1. root level #XYZ# exists. namespace="#XYZ#/"
2. Junk folder setting : "Junk" folder of
     mail.server.serverN.spamActionTargetAccount = imap://userName@hostname
3. Because namespace="#XYZ#/", ensureExists of Junk is executed on "#XYZ#/Junk",
     ensureExists%3E/%23XYZ%23/Junk:  = currentUrl
   and create "#XYZ#/Junk" is executed and is subscribed normally.
4. Because "#" is delimiter of hash in URL, its's "/" is escaped as "#XYZ#^Junk".
   This garbled "#XYZ#^Junk" is written in ...\ImapMail\<servername>.msf
   due to bug 720911.
5. After that, create request of ABC is changed to #XYZ#^Junk/ABC by bug 720911,
   and is changed to #XYZ#/#XYZ#^Junk/ABC by namespace="#XYZ#/".
Status: UNCONFIRMED → NEW
Ever confirmed: true
FYI.

(In reply to Jeremy Tan from comment #11)
> user_pref("mail.server.server1.namespace.public",
> "\"#public/\",\"#news.\",\"#ftp/\",\"#shared/\"");
> However, I did not add them myself.

If non-personal namespace(shared/public namespace, other users namespace) is used, and if folder(or parent folder) for non-personal namespace is accessed and is shown at folder pane of Tb(returned as LSUB command and/or LIST command), Bug 540378 currently occurs, and that bug may be relivant to this bug.
Blocks: 160644
Component: General → Networking: IMAP
Product: Thunderbird → MailNews Core
QA Contact: general → networking.imap
No longer blocks: 517054
Depends on: 517054
Developers of IMAP RFC looks to have had assigned special meaning to # ... 
> http://tools.ietf.org/html/rfc3501#section-5.1.2
> 5.1.2. Mailbox Namespace Naming Convention
>    By convention, the first hierarchical element of any mailbox name
>    which begins with "#" identifies the "namespace" of the remainder of
>    the name.  This makes it possible to disambiguate between different
>    types of mailbox stores, each of which have their own namespaces.
> 
>         For example, implementations which offer access to USENET
>         newsgroups MAY use the "#news" namespace to partition the
>         USENET newsgroup namespace from that of other mailboxes.
>         Thus, the comp.mail.misc newsgroup would have a mailbox
>         name of "#news.comp.mail.misc", and the name
>         "comp.mail.misc" can refer to a different object (e.g., a
>         user's private mailbox).

However, such special meaning is never defined by RFC for Namespace. Rather, developer of namespace RFC recommends to avoid "# in namespace".
It's merely that # was a convension used by UW-IAP developer, isn't it?
> http://tools.ietf.org/html/rfc2342
>   Example 5.9:
>   Historical convention has been to start all namespaces with the "#"
>   character.  Namespaces that include the "#" character are not IMAP
>   URL [IMAP-URL] friendly requiring the "#" character to be represented
>   as %23 when within URLs.  As such, server implementers MAY instead
>   consider using namespace prefixes that do not contain the "#"
>   character.
> Gahrns & Newman      Standards Track           [Page 7]
As for "#" in namespace part, it looks to have resolved by "correct escaping of #" in internal URL used by Tb at many places, because problem due to "#something/" of any namespace is not observed.
Main culprit is perhaps mixture of "namespace of something + . with delimiter of ." in sample setting of shipped UW-IMAP even though "delimiter is /" is used, which is merely to show that "delimiter of /" and "delimiter of ." can be mixed in IMAP by sample namespace setting, instead of one shipped as "recommended or practically useful default setting".
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.