720911 - create new folder dialog, parent folder picker is ignored for imap account (when folder name and/or namespace has #, e.g. #news, "#news" is written in <servername>.msf as folder name of <servername> directory, then root of the account becomes "/#news")

Status: NEW

(Thunderbird :: Mail Window Front End, defect)

Description

[Wayne Mery (:wsmwk)]

+++ This bug was initially created as a clone of Bug #467824 +++

I have a similar situation. Perhaps it's a dupe. 
creating new folders insists on putting new folders in
 #news

if I delete #news folder from thunderbird and server it keeps returning. 

File | new  sometimes defaults to #news when I do click the account and do File | New | Folder

Whether or not #news is shown as the default selection, the folder selection ignores my selection of the account and puts new folder in #news.  All other eligible folders are listed, but it won't let me pick them.

-Imap account- Advanced settings
IMAP server directory: /home/wsm0/mail/
Personal namespace: ""
Public: ""
Other users: ""
Allow server to override: unchecked

the other settings don't seem to affect behavior.

Comment 1

[Wayne Mery (:wsmwk)]

Attachment: imap log

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #1)
> imap log

(a) No bad is seen in log for create, except "\NoInferiors bug" of your server.
> 3 create "/home/wsm0/mail/#news/print"
> 3 OK CREATE completed
> 4 subscribe "/home/wsm0/mail/#news/print"
> 4 OK SUBSCRIBE completed
> 5 list "" "/home/wsm0/mail/#news/print"
> * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/#news/print
> 5 OK LIST completed
(b) \NoSelect is also seen. IIRC, this is also "\NoSelect bug" of your server.
> * LIST (\NoSelect \HasChildren) "/" /home/wsm0/mail/
> * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/Junk
(c) No LSUB is seen in log.
    You probably unckeck "Show subscribed folders only".

I could see "#news" only in above log lines. i.e. You created #news/print in one step(correct operation), instead of "create #news, open #news, create print under #news".
(if you did it, create print perhaps failed. "create #news/, create print" was needed, and if it was done, subscription error occurred due to "subscribe #news/" by Tb :-) )

> if I delete #news folder from thunderbird and server it keeps returning.

Where is IMAP log which corresponds to this operation?
Is there any error in Activity Manager for "delete #news folder"?
Is there IMAP log for the "server it keeps returning"?
What is shown in subscription list(account's context menu, subscribe) 

\NoSelect(server wants to say \Noselect/\Noinferiors, but...) is seen for parent folder of mail of Junk folder. Your server perhaps doesn't support "folder contains both subfolders and mails".
> * LIST (\NoSelect \HasChildren) "/" /home/wsm0/mail/
> * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/Junk
Do you uncheck Tb's option for it at Server Settings/Advanced?

> Whether or not #news is shown as the default selection, the folder selection
> ignores my selection of the account and puts new folder in #news.

sp < ! < " < # < $ < % < & < ... < 0 < ... < 9 < A < ... < Z ... 
It looks "folder of lowest alphebetic order name" == #news.
What happens if "!news/print" is created?

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #2)
> (In reply to Wayne Mery (:wsmwk) from comment #1)
> > imap log
> 
> (a) No bad is seen in log for create, except "\NoInferiors bug" of your
> server.
> > 3 create "/home/wsm0/mail/#news/print"
> > 3 OK CREATE completed
> > 4 subscribe "/home/wsm0/mail/#news/print"
> > 4 OK SUBSCRIBE completed
> > 5 list "" "/home/wsm0/mail/#news/print"
> > * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/#news/print
> > 5 OK LIST completed
> (b) \NoSelect is also seen. IIRC, this is also "\NoSelect bug" of your
> server.
> > * LIST (\NoSelect \HasChildren) "/" /home/wsm0/mail/
> > * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/Junk
> (c) No LSUB is seen in log.
>     You probably unckeck "Show subscribed folders only".
correct


> I could see "#news" only in above log lines. i.e. You created #news/print in
> one step(correct operation), instead of "create #news, open #news, create
> print under #news".
> (if you did it, create print perhaps failed. "create #news/, create print"
> was needed, and if it was done, subscription error occurred due to
> "subscribe #news/" by Tb :-) )
after I deleted all instances of #news on the server and PC, #news appeared nowhere in the thunderbird UI at the time I specified create "print" folder 

> > if I delete #news folder from thunderbird and server it keeps returning.
> 
> Where is IMAP log which corresponds to this operation?
if you mean the delete, I didn't capture that - but I can try if you want.
for the create, it appears only where you see it in the log above :)

> Is there any error in Activity Manager for "delete #news folder"?
i think no. but I'll recheck

> Is there IMAP log for the "server it keeps returning"?
> What is shown in subscription list(account's context menu, subscribe) 
show subscribed only is normally unchecked
should it matter for this problem?

> \NoSelect(server wants to say \Noselect/\Noinferiors, but...) is seen for
> parent folder of mail of Junk folder. Your server perhaps doesn't support
> "folder contains both subfolders and mails".
correct

> > * LIST (\NoSelect \HasChildren) "/" /home/wsm0/mail/
> > * LIST (\NoInferiors \UnMarked) "/" /home/wsm0/mail/Junk
> Do you uncheck Tb's option for it at Server Settings/Advanced?
correct

> > Whether or not #news is shown as the default selection, the folder selection
> > ignores my selection of the account and puts new folder in #news.
> 
> sp < ! < " < # < $ < % < & < ... < 0 < ... < 9 < A < ... < Z ... 
> It looks "folder of lowest alphebetic order name" == #news.
> What happens if "!news/print" is created?
Do  you mean, what happens if I manually create "!news" and then try to create "print"?

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #3)
> > > Whether or not #news is shown as the default selection, the folder selection
> > > ignores my selection of the account and puts new folder in #news.
> > sp < ! < " < # < $ < % < & < ... < 0 < ... < 9 < A < ... < Z ... 
> > It looks "folder of lowest alphebetic order name" == #news.
> > What happens if "!news/print" is created?
> Do you mean, what happens if I manually create "!news" and then try to create "print"?

Partially YES, but partially NO. 
I asked you to check Tb's behavior when "!news/print" is created by single request when "#news/print" already exists by your creation of "#news/print".

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #4)
> (In reply to Wayne Mery (:wsmwk) from comment #3)
> > > > Whether or not #news is shown as the default selection, the folder selection
> > > > ignores my selection of the account and puts new folder in #news.
> > > sp < ! < " < # < $ < % < & < ... < 0 < ... < 9 < A < ... < Z ... 
> > > It looks "folder of lowest alphebetic order name" == #news.
> > > What happens if "!news/print" is created?
> > Do you mean, what happens if I manually create "!news" and then try to create "print"?
> 
> Partially YES, but partially NO. 
> I asked you to check Tb's behavior when "!news/print" is created by single
> request when "#news/print" already exists by your creation of "#news/print".

create new folder !news/print as subfolder of <account> results in new folder 
  #news/!news/print
:(
nothing in activity manager 
error console has

folderDisplayListener: onMakeActive
aWasInactive: undefined
 and
Timestamp: 1/26/2012 8:42:41 AM
Error: aFolderDisplayWidget.view.dbView is null
Source File: chrome://lastmessage/content/overlay.js
Line: 219

but i don't know if these are related to the folder create - i don't remember seeing them before


on a different PC, same account, I tried to create new folder in same account, and it totally, silently fails. nothing in activity manager. error console has the following but again i don't if it is related.

Timestamp: 1/26/2012 8:40:16 AM
Error: [Exception... "Component returned failure code: 0x80550007 [nsIMsgFolder.getStringProperty]"  nsresult: "0x80550007 (<unknown>)"  location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2464"  data: no]
Source File: chrome://messenger/content/folderPane.js
Line: 2466

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #5)
> create new folder !news/print as subfolder of <account> results in new folder 
>   #news/!news/print
> :(

How about rename by Tb? (I guess fails...)

It may be "#" problem, because "#" is delimitor of hash in URL. It may be "first letter # of root level folder" only problem. (As IMAP server directory is used it's shown as root level folder at folder pane.
See dependency tree for meta bug 124287 with "Show Resolved" and Find #.

Can you rename "#news" to news#news, !news or something at IMAP server, Web mail, other IMAP client?

Comment 7

[Wayne Mery (:wsmwk)]

all types of attempts with thunderbird fail. I tried other changes in advance settings. I removed all remnants of #news from the server and the PC.  No joy. I logs from the simplest change if you like. Seems to be not caused by any feedback from the server, i.e. thunderbird is doing something wrong.  I will try one more time from yet a different PC.  And I may try version 2.

Using TheBat! I was able to create a directory under the account name. No problem. 

I found some old emails - I had a short exchange in July 2010 with bienvenu. And those also implied we had talked about it in a prior year.

bug 124287 is quite a long list of issues. :(

Comment 8

[WADA:World Anti-bad-Duping Agency]

I couldn't see any problem with # in folder name, with Gmail IMAP, Tb 9.0.1, Tb 12.0a1(2012/01/19 build). I can't check with Yahoo! IMAP, because Yahoo! IMAP rejects folder creation if # is contained in folder name.

It may be "IMAP server directory:" related problem.
See bug 511331 for an issue when "IMAP server directory: [Gmail]" was intentionally set for Gmail IMAP even though namespace="". In that bug, mismatch between implementation of "IMAP server directory:" and special folder path selection UI was observed(Junk in that bug).
- IMAP server directory : 
  Specified path is hidden at folder pane display, at folder selection list.  
- Junk folder selection UI :
  Path shown by folder selection list is written to spamActionTargetFolder,
  even though spamActionTargetFolder requests absolute path.

> IMAP server directory: /home/wsm0/mail/
> Personal namespace: ""
> Allow server to override: unchecked

What happens when setting is changed?
  IMAP server directory: ""
  Personal namespace: "/home/wsm0/mail/"
  Allow server to override: unchecked
This is similar to setting for ordinal namespace="INBOX/" or "INBOX.". Although home -> wsm0 -> mail is always shown at folder pane and child folders are shown as sub folder of home -> wsm0 -> mail, full path is always shown at folder selection list. So, mismatch happens only in Trash folder selection UI.

How about with next?
(similar to "hide Inbox. part at folder pane when namespace="Inbox".) 
  IMAP server directory: "/home/wsm0/mail/"
  Personal namespace: "/home/wsm0/mail/"
  Allow server to override: unchecked

> location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: ...

Smart Folder is not created automatically(or initial creation is unsuccessful) until enter Unified view once. Enter "Unified Folder" view, go back "All" view. Do you still see exception?

Comment 9

[WADA:World Anti-bad-Duping Agency]

I checked "# in folder name" with Gmail IMAP of ordinal setting(no IMAP server directory, no namespace).
  Gmail Label        Gmail IMAP folder       Folder at Tb
                      N/A                     #ABC#       (italic. grayed out)
                      N/A                     #ABC#/#DEF# (italic. grayed out)
   #ABC#/#DEF#/XYZ    #ABC#/#DEF#/XYZ         #ABC#/#DEF#/XYZ
This is created by typing "#ABC#/#DEF#/XYZ" as folder name.
"Folder of (italic. grayed out)" is similar to "\Noselect, No \Noinferiors" which is for folder of "can contain subfolder only, can't cantain mail".

After enjoying funny phenomenon by "rename #DEF# #DEF!", "rename #ABC# #ABC!", "rename #DEF! !DEF#", "rename #ABC! !ABC#", "rename !DEF# #DEF#", "rename !ABC# "ABC#", I could experience this bug's phenomenon. New folder was always created under "/#ABC#/#DEF#/".
And, even after clean up of such garbled folders(delete Gmail Label and restart, delete folder files/sbd direcories, and so on), This bug revived and new folder was created under "/#ABC#/#DEF#/".

I could see following data in ...\ImapMail\imap.gmail.com.msf (mail directory for this Gmail IMAP account=...\ImapMail\imap.gmail.com).
> <(80=1)(87=#ABC#/#DEF#)(85=1082004)>
When I changed it to !ABC#/!DEF# and restrted Tb a few times, folder was created under !ABC#/!DEF#.
> <(80=1)(87=!ABC#/!DEF#)(85=1082004)>
When I changed it to #ABC!/#DEF! and restrted Tb a few times, folder was created under #ABC!/#DEF!".
> <(80=1)(87=!ABC#/!DEF#)(85=1082004)>
When I deleted ...\ImapMail\imap.gmail.com.msf and restarted Tb, Tb returned to normal behaviour.

Because folder name contains "#", file name is hashed. 
  #ABC#       => ...\e7a0b3f7.msf
  #ABC#/#DEF# => ...\e7a0b3f7.sbd\12febf78.msf 
I guess Tb searches parent folder upon rename and fails to find it because of hashed name or "#" in internal imap:// URL, then goes to higher parent==account, and writes current folder name in .msf file of the topmost parent=account, and once it's written in .msf, root becomes /#ABC#/#DEF#.

Comment 10

[Wayne Mery (:wsmwk)]

FWIW, I added 
 "#public/","#news.","#ftp/","#shared/"
to personal namespace in advanced settings and I am able to create new folders under the account.

however, that still leaves a #news folder in my folder pane :(

Comment 11

[Wayne Mery (:wsmwk)]

thunderbird 2 lets me create folder in account without namespace, even if #news exists in the folder pane

Comment 12

[Wayne Mery (:wsmwk)]

premature accusation about regression.
v5, v3.1, v3.0 and trunk all work if run from new profile

Comment 13

[Wayne Mery (:wsmwk)]

ah, there it is mail.lehigh-1.edu.msf

<(80=1)(84=#news)(86=Mon Jun 01 13:33:12 2009)>
{1:^9F {(k^A0:c)(s=9u)} 
  [1(^AC=1)(^AD=1)(^AE^84)(^A7^84)(^BF^86)]}

Comment 14

[WADA:World Anti-bad-Duping Agency]

Following is log for rename "XYZ under /#ABC#/#DEF#/" to "#XYZ#". 
> S-#ABC#/#DEF#/XYZ:ProcessCurrentURL:imap://[e-mail]:993/rename%3E/%23ABC%23/%23DEF%23/XYZ%3E/#ABC#/#DEF#/%23XYZ%23:  = currentUrl
Following is log for rename "#DEF# under /#ABC#/" to "#DEF@". 
> A:ProcessCurrentURL:imap://[e-mail]:993/rename%3E/%23ABC%23/%23DEF%23%3E/#ABC#/%23DEF%40:  = currentUrl

Escaping of "#" is not executed on "parent folder part of new foler name" in currentUrl. This may be relevant to problem of this bug, although I think main cause is "Escaped %23ABC%23 == Unescaped #ABC# != Filename of e7a0b3f7".

Comment 15

[Wayne Mery (:wsmwk)]

something related+new...

I successfully created new folders off the account root, named 1, 2, 3, ... 9. 
And I can put messages in those folders. 

But today I tried renaming 7 to a7, for the first time...
which succeeds according to UI and activity manager.

then I click on a7 and get 
 the current operation on a7 did not succeed. the mail server for account ... responded: Can't open mailbox .../mail/#news/a7: no such mailbox

And a7 folder is not displayed under #news folder.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #15)
> But today I tried renaming 7 to a7, for the first time...
> which succeeds according to UI and activity manager.
> then I click on a7 and get 
>  the current operation on a7 did not succeed. the mail server for account
> ... responded: Can't open mailbox .../mail/#news/a7: no such mailbox
> And a7 folder is not displayed under #news folder.

Garbage of root="/#news" is saved in at least two places.
  (a) <servername>.msf
  (b) cached data (panacea.dat etc.)
"a few times" in next comment means that it took at least two restarts of Tb to fully reflect .msf content change on "create folder".
> When I changed it to !ABC#/!DEF# and restrted Tb a few times, folder was created under !ABC#/!DEF#.
Cached data is not reset by "delete <servername>.msf and restart Tb" only. Additional "termination of Tb/restart of Tb" is needed to replace cached data and to use clean cached data.

Did you terminate Tb after "delete <servername>.msf and restart" before you execute "create folder" and "rename folder"?

Comment 17

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #16)
> (In reply to Wayne Mery (:wsmwk) from comment #15)
> > But today I tried renaming 7 to a7, for the first time...
> > which succeeds according to UI and activity manager.
> > then I click on a7 and get 
> >  the current operation on a7 did not succeed. the mail server for account
> > ... responded: Can't open mailbox .../mail/#news/a7: no such mailbox
> > And a7 folder is not displayed under #news folder.
> 
> Garbage of root="/#news" is saved in at least two places.
>   (a) <servername>.msf
>   (b) cached data (panacea.dat etc.)

unfortunately, blowing away panacea in the past has caused all messages to be redownloaded. How do I avoid that?

> "a few times" in next comment means that it took at least two restarts of Tb
> to fully reflect .msf content change on "create folder".
> > When I changed it to !ABC#/!DEF# and restrted Tb a few times, folder was created under !ABC#/!DEF#.
> Cached data is not reset by "delete <servername>.msf and restart Tb" only.
good point

> Additional "termination of Tb/restart of Tb" is needed to replace cached
> data and to use clean cached data.
> 
> Did you terminate Tb after "delete <servername>.msf and restart" before you
> execute "create folder" and "rename folder"?

yes

Comment 18

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #17)
> (In reply to WADA from comment #16)
> > (In reply to Wayne Mery (:wsmwk) from comment #15)
> > > But today I tried renaming 7 to a7, for the first time...
> > > which succeeds according to UI and activity manager.
> > > then I click on a7 and get 
> > >  the current operation on a7 did not succeed. the mail server for account
> > > ... responded: Can't open mailbox .../mail/#news/a7: no such mailbox
> > > And a7 folder is not displayed under #news folder.
> > 
> > Garbage of root="/#news" is saved in at least two places.
> >   (a) <servername>.msf
> >   (b) cached data (panacea.dat etc.)
> 
> unfortunately, blowing away panacea in the past has caused all messages to
> be redownloaded. How do I avoid that?

I had once deleted <servername>.msf and iirc it doesn't cause redownload of messages. Does that sound correct?

Also, are you suggesting if I delete panacea and <servername>.msf, and delete the #news related folders and files in the profile, that I can work around this problem?

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #10)
> FWIW, I added 
>  "#public/","#news.","#ftp/","#shared/"
> to personal namespace in advanced settings (snip)

Same IMAP server as bug 697789?

> however, that still leaves a #news folder in my folder pane :(

"#news." is mbox named "#news."?
or Mbox named "#news" and delimiter="."?
Is such Mbox existent Mbox and such namespaces actually used?

Comment 20

[WADA:World Anti-bad-Duping Agency]

To delete garbage, following seems sufficient.
(1) Delete <servername>.msf, delete folders of "# in folder name" at server,
    restart Tb.
(2) Subscribe at Tb to force open/read of <servername>.msf,
    and terminate Tb to force close of<servername>.msf.

Comment 21

[WADA:World Anti-bad-Duping Agency]

When garbage is created was observed in bug 697789.
Phenomenon of bug 697789 was following.
Created garbage string is different from folder name with "#" if namespace is used, so phenomenon is clearer then this bug's case.
1. root level #XYZ# exists. namespace="#XYZ#/"
2. Junk folder setting : "Junk" folder of
     mail.server.serverN.spamActionTargetAccount = imap://userName@hostname
3. Because namespace="#XYZ#/", ensureExists of Junk is executed on "#XYZ#/Junk",
     ensureExists%3E/%23XYZ%23/Junk:  = currentUrl
   and create "#XYZ#/Junk" is executed and is subscribed normally.
4. Because "#" is delimiter of hash in URL, its's "/" is escaped as "#XYZ#^Junk".
   This garbled "#XYZ#^Junk" is written in ...\ImapMail\<servername>.msf
   due to bug 720911.
5. After that, create request of ABC is changed to #XYZ#^Junk/ABC by bug 720911,
   and is changed to #XYZ#/#XYZ#^Junk/ABC by namespace="#XYZ#/".

file for "#XYZ#" is 8971b6d4.msf. ensureExists%3E/%23XYZ%23 may search file named "#XYZ#.msf".

Comment 22

[WADA:World Anti-bad-Duping Agency]

Quick check result.
(1) #XYZ# : set of 8971b6d4.msf, 8971b6d4.sbd, 8971b6d4.sbd\Junk.msf exists.
(2) copy above files/directory to file/directory named #XYZ#.
                   #XYZ#.msf,    #XYZ#.sbd,    #XYZ#.sbd\Junk.msf
(3) Restart Tb, rename Junk under #XYZ# to JunkX
    => #XYZ#.sbd\Junk.msf is renamed to #XYZ#.sbd\JunkX.msf
       8971b6d4.sbd\Junk.msf still remains.
(4) After that, #XYZ#.msf and #XYZ#.sbd was used for IMAP folder named #XYZ#.

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to Wayne Mery (:wsmwk) from comment #18)
> are you suggesting if I delete panacea and <servername>.msf, and
> delete the #news related folders and files in the profile,
> that I can work around this problem?

No, I merely asked about phenomenon when panacea.dat is deleted.
Tb apparently knows about previous garbage in <hostname>.msf file even after delete of <hostname>.msf or modification of <hostname>.msf contet. Garbled data is not cleared nor new garbled data is not reflected until Tb accesses <hostname>.msf file.
Subscribe looks a way to force read of <hostname>.msf. Because Subscribe is account level operation, Tb looks to use <hostname>(directory) as if "root folder" and <hostname>.msf seems created as a result of "access as a mail folder".

Following is a recovery operation.   
(1) Delete folders of "# in name" at server.
(2) Delete <hostname>.msf, restart Tb, access Inbox.
(3) Subscribe, Refresh, to force access to <hostname>.msf, terminate Tb. 
Possible temporary workaround if folders of "# in name" are required.
(1) copy hashed-name.msf to "...#...msf"
    copy hashed-name.sbd to "...#...sbd"
(2) Delete <hostname>.msf, restart Tb, access Inbox.
(3) Subscribe, Refresh, to force access to <hostname>.msf, terminate Tb.