503735 - IMAP folder of account's label name(string in mail.server.serverN.name) is created, if specified junk folder doesn't exist

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[Hendrik Duerkop]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090711 Minefield/3.6a1pre
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1pre) Gecko/20090711 Shredder/3.0b3pre

http://nw.io/z/subs2.png
folders i didn't create and can't unsubscribe off (see bug 498292)

http://nw.io/z/subs3.png
a minute later doing nothing there's more folders and i can unsubscribe them
again

Reproducible: Sometimes

Steps to Reproduce:
1. delete the folders
2. empty trash
3. restart shredder
4. get mails via imap a few times
Actual Results:  
sometimes you will get such a folder

Expected Results:  
i just want to see my normal mail tree

not folders i can do nothing with

Comment 1

[Magnus Melin [:mkmelin]]

So how is this different from bug 498292? I assume one gives the other...

Comment 2

[Hendrik Duerkop]

i think so too lol
but i was asked to open this

Comment 3

[WADA:World Anti-bad-Duping Agency]

Folder structure/Subscription list like yours can easily be observed using Gmail IMAP by one of next.
 (1) At Gmail Web Interface, create Gmail Label of X/Y without creating X.
 (2) By Tb, create folder named X/Y when X doesn't exist.
     (Not "create X then create Y under X". Put "X/Y" as new folder name.)
Above is also seen on [Gmail] and subfolders of [Gmail]. 
If Tb requests "delete of X/Y" (rename to Trash/X/Y and unsubscribe X/Y) to Gmail IMAP, Gmail IMAP automatically removes "X", because Gmail Label of "X" doesn't exist.

(A) What is you server's spec for request of "create a.b.c / subscribe a.b.c" while "a" and "a.b" doesn't exist?
Note:
RFC 3501 requests creation of "a", "a.b", and "a.b.c".
> http://tools.ietf.org/html/rfc3501#section-6.3.3
I don't know about attributes of folder "a" and "a.b" in this case.

(B) What is you server's spec for "a" and "a.b" when "a.b.c" is deleted after (A)?

Does you IMAP server keep folder of "a", "a.b" after "delete of a.b.c"(rename to Trash.a.b.c)?
Does your IMAP server keep subscription of "a", "a.b", "a.b.c" after "delete of a.b.c"(rename to Trash.a.b.c and unsubscribe a.b.c)?
If yes, how about next case?
   "delete of a.b.c"(rename to Trash.a.b.c and unsubscribe a.b.c)
   followed by "Empty Trash"

Do you enable "Show only subscribed folders" option? (Server Settings/Advanced)
If yes, does phenomenon change by disabling "Show only subscribed folders"?
Note:
"Show only subscribed folders" is not recommended for Gmail IMAP, because Gmail IMAP always returns all folders to LSUB, and because it can produce error like deletion of non-existent folder due to mismatch between spec of IMAP and implementation of Gmail IMAP.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #1)
> So how is this different from bug 498292? I assume one gives the other...

Bug 498292 is for creation of folder like h[at]nw.io which is string specified in mail.server.severN.name(label name of accounts!). "create h[at]nw.io" was requested by Tb even when such folder is not returned to LSUB, and the command was requested even to imap.gmail.com. 

This bug is for phenomenon after "create h[at]nw.io" was issued.
Even if garbages folder like h[at]nw.io was created, it should be cleanly purged by "delete of folder" operation. But it looks to fail because IMAP server seems to return the folder to LSUB even after "delete of folder" operation.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> http://nw.io/z/subs3.png

Account named [e-mail] (another account of hostname="nw.io") has Inbox and Trash only. In attached data to Bug 498292(screen shot, log), garbages were seen for this account too. Clean up of [e-mail] was successful? No garbages were created for [e-mail]?

Comment 6

[Hendrik Duerkop]

for some reason the inbox of [e-mail] is not subscribed at all on this machine.
http://nw.io/z/fold9.png


http://nw.io/z/maildir1.png
http://nw.io/z/maildir2.png
i hope these help.
i will clean up the mess now and we will see if they return

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> http://nw.io/z/maildir1.png

Aha.
Your server creates directory named "a.b.c" and doesn't create "a" and "a.b" when "create a.b.c" is requested. It's similar to "Gmail label of A/B/C and no A, A/B".
It's the reason why no check box in subscription list for "a" and "a.b".
In this case, deletion of "a.b.c" automatically removes "a" and "a.b", because IMAP server never says "a or a.b exist" nor "a or a.b is subscribed" after delete of "a.b.c".

I checked log attached to Bug 498292 Comment #22 & #23 again.
After deletion of "a.b"(rename a.b to Trash.a.b, unsbscribe a.b) and Empty Trash (delete Trash.a.b, unsubscribe Trash.a.b), your IMAP server didn't return data for "a" nor "a.b" to LSUB, LIST etc.
I misunderstood that 4-th log attached to Bug 498292 Comment #34 was obtained after delete of garbage. Sorry for my confusion.

Problem looks for me:
(1) Log attached to Bug 498292 Comment #22 & #23 and Bug 498292 Comment #34.
    Data for mail.server.serverN.name seems to be considered as an entry of
    subscription list at least for [e-mail](one of two hostname=nw.io".
    And "list [e-mail]" is issued, then "create [e-mail]" is issued
    because response to "list [e-mail]" says "[e-mail] doesn't exist".
(2) Log attached to Bug 498292 Comment #3.
    If (1) occurs during communication with imap.gmail.com,
    "list [e-mail]" and following "create is [e-mail]" is somehow executed
    under task for imap.gmail.com.
I want to keep Bug 498292 for analysis of (2) and to change this bug for analysis of (1). Do you agree?

Comment 8

[Hendrik Duerkop]

sure, do whatever you think. i will help you as best as possible

Comment 9

[WADA:World Anti-bad-Duping Agency]

Changing summary for analysis of (1) in this bug.
Confirming based on IMAP logs attached to Bug 498292.

Comment 10

[Hendrik Duerkop]

something new today:
http://modelle.nw.io/z/imap4.png

the usual harvest (done on my laptop, latest shredder, no plugins, xp sp3,
solid state disc - which is badly shreddered during updates... ):
http://modelle.nw.io/z/fold10.png
http://modelle.nw.io/z/imap.log
http://modelle.nw.io/z/prefs.js

Comment 11

[WADA:World Anti-bad-Duping Agency]

Hendrik, can you get log with unique accounts's label name? (h[at]nw.io appears at many places, so it's not obvious that folder name is obtained from serverN.name)
 (1) Change all accounts name to string like "XXX-1 ...".
     "-1" part is version number of account's name change.
 (2) Delete all previous garbages and Empty Trash => garbage of "XXX-1 ..." only.
 (3) If test with new version of account's name, change to "XXX-n ..."
     where n=version_number of account's label name.
 (4) If test with single account's name change, change to name of
     version_number+suffix ("XXX-n-A", "XXX-n-B", "XXX-n-C", ...)

Can you check with hostname of imap-X.nw.io and realhostname of "nw.io"?
(3rd level qualifier exists. similar to request of Bug 498292 Comment #35) 

> user_pref("mail.server.serverX.hostname", "imap1.nw.io");
> user_pref("mail.server.serverX.realhostname", "nw.io");
> user_pref("mail.server.serverX.spamActionTargetAccount", "imap://[e-mail]");
> user_pref("mail.server.serverY.hostname", "imap2.nw.io");
> user_pref("mail.server.serverY.realhostname", "nw.io");
> user_pref("mail.server.serverY.spamActionTargetAccount", "imap://[e-mail]");
 
Note:
Filter won't work, because internal folder path is changed by manual hostname modification. If "folder doesn't exists" occurs, change filer definition(copy/move target folder) to appropriate one via Message Filter UI.

Comment 12

[WADA:World Anti-bad-Duping Agency]

I could see problem of this bug when I enabled "Advanced IMAP Controls" of Gmail Lab, during test for other bug. Upon every change of account name of Gmail IMAP account, Tb trunk(2009/8/13 build) kindly created IMAP folder of account name :-) 
After disable of "Advanced IMAP Controls" of Gmail Lab again, I couldn't see problem of this bug again.
I guess Gmail IMAP's behaviour becomes near to ordinal IMAP server's one if "Advanced IMAP Controls" of Gmail Lab is enabled, or Gmail IMAP's bug is exposed by "Advanced IMAP Controls" of Gmail Lab.
I'll try to check difference between IMAP log of "Advanced IMAP Controls"=Enabled and IMAP log of "Advanced IMAP Controls"=disabled, using Gmail/Gmail IMAP.

Comment 13

[WADA:World Anti-bad-Duping Agency]

Correction of Comment #12.
"Advanced IMAP Controls" of Gmail Lab was irrelevant.

If folder specified in spamActionTargetFolder doesn't exist and moveOnSpam=true, IMAP folder of account label is created upon restart and/or upon each rename of account. 
> mail.server.server2.moveOnSpam = true
> mail.server.server2.spamActionTargetFolder = imap://<userName>@<hostname>/Spam
> But root-level Spam folder doesn't exist.
>(Note: Gmail IMAP, so next is set) 
> mail.server.server2.is_gmail = true
> mail.server.server2.namespace.personal = ""

If "IMAP server directory:" is specified, root-level Junk was created instead of folder of account label. (phenomenon of Bug 511331 comment #0) 
> 
mail.server.server2.server_sub_directory = [Gmail]

To Hendrik Duerkop(bug opener):

Correct path is set in mail.server.serverN.spamActionTargetFolder?

Comment 14

[WADA:World Anti-bad-Duping Agency]

Oh, your case was "Junk of Local folder" case.
> user_pref("mail.server.server4.moveOnSpam", true);
> user_pref("mail.server.server4.spamActionTargetAccount", "imap://[e-mail]");
> user_pref("mail.server.server4.spamActionTargetFolder", "mailbox://nobody@Local%20Folders/Junk");

How did you set spamActionTargetFolder=mailbox://nobody@Local%20Folders/Junk?
(a) Via folder selection list at Others: of Junk setting? (b) Or manually? (c) Or imported or copied from other profile?
Does Junk folder of "Local Folders" exist? If exists, "wastepaper ison" for junk folder is set?

I guess (c) and Junk of "Local Folders" doesn't exist, because I could observe this bug's problem with next condition.
  - spamActionTargetFolder=mailbox://nobody@Local%20Folders/Junk
  - Junk folder of "Local Folders" doesn't exist (never created),
    or Junk is manually created but before restart of Tb (attribute of special
    folder is not set, icon for special folder is not set, until restart. This
    is known bug which is fixed recently.)

Comment 15

[Hendrik Duerkop]

sorry for the long time away and will be away for some more days from today on. after that i try better...

i never touched the prefs.js.
my profile i usually copied from machine to next machine.
the junk settings i always set in tools -> account settings -> [account] -> junk settings -> (checked) move new junk messages to: -> (selected) "junk" folder on: [account].
why isn't this active by default anyway?

this applies to hendrik[at]vonduerkop.eu and h[at]nw.io. the other accounts i use don't filter anything (there's no need to).
so i wonder why moveonspam is true on my gmail account...

Local junk folders won't help me because i need all folders online - just in case i put something in spam or a client calls me to ask if i didn't get his mail and i have to have a look.
also i can't remember to have ever seen a local junk folder (on none of my 3 machines is one).

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Local junk folders won't help me because i need all folders online (snip)

You like or dislike local Junk folder is irrelevant to you problem :-)
Key point is: folder pointed by spamActionTargetFolder exists or not. 

> also i can't remember to have ever seen a local junk folder (on none of my 3
> machines is one).

Folder pointed by spamActionTargetFolder doesn't exists in your case.

I could see "IMAP Junk folder creation", "local Junk creation", and "IMAP folder of account label creation" by next setting of an Gmail IMAP account.
Note:
moveOnSpam is always true, folder specified in spamActionTargetFolder doesn't exist. spamActionTargetFolder is set via Others: of Junk Setting(exsitent folder), then altered via Config Editor.

(1) spamActionTargetFolder=mailbox://nobody@Local%20Folders/ABC
    => Junk of "Local Folders" is created.
       File of ABC.msf, Junk.msf, Junk was creatd. 
(2) spamActionTargetFolder=imap://username@hostname/Spam
    => root-level Junk of the IMAP account is created.
       Sometimes icon is changed to "paperwaste icon" but sometimes not.
       The Junk folder can be renamed, deleted. (not real junk folder)
(3) spamActionTargetFolder=imap://username@hostname/Spam
    IMAP server directory: NS (NS exists, but NS/Spam doesn't exist)    
    => NS/Junk is created (NS/Junk is displayed as if root-level Junk while IMAP
       server directory:NS is set.)
 
(4) spamActionTargetFolder=imap://username@hostname/
    => IMAP folder of account label is created upon restart,
       or upon each account label change. Same phenomenon as this bug.
(5) Probably same case as yours.
(5-0) Root-level Junk of the IMAP account doesn't exist.
(5-1) Reset moveOnSpam, spamActionTargetAccount, spamActionTargetFolder
      via Config.editor, and restart Tb.
(5-2) Enable moveOnSpam, select this account's Junk folder at Junk setting.
      => spamActionTargetFolder=mailbox://nobody@Local%20Folders/Junk is set.
         But this "account's Junk" is displayed at Junk setting panel.
(5-3) Change account label => IMAP folder of IMAP account label is created
      upon each account label rename.
(5-4) Delete IMAP folders of account label and restart Tb
      => IMAP folder of IMAP account label is created.
 
See Bug 511331 Comment #0 or phenomenon of (1) to (3).

It looks next:
  If spamActionTargetFolder doesn't exists, Tb tries to fall back to "Junk",
  and phenomenon like (1) to (3) occurrs.
  But in special situation like (5), similar condition to (4) happens,
  then IMAP folder of account label is created.
  (probably folder name="" like condition)

Comment 17

[WADA:World Anti-bad-Duping Agency]

Phenomenon was simple.

If folder specified in mail.server.serverN.moveTargetFolder doesn't exist,
(i)  When mail.server.serverN.moveTargetMode is not set.
     (mail.server.serverN.moveTargetMode is not defined. "Junk folder on:")
     IMAP folder of account label name is created upon restart, and upon each
     rename of account label name(string in mail.server.serverN.name), and so on.     
(ii) When mail.server.serverN.moveTargetMode=1 ("Others:").
     root-level Junk is created at location specified in serverN.moveTargetFolder.
        imap://...     : Junk is created at specified IMAP server.
        mailbox://...  : Junk is created at specified local account.
(iii) If "IMAP server directory:" is secified, created folder becomes;
      - when (i)  : (IMAP server directory:) + / + (account label name)
      - when (ii) : (IMAP server directory:) + / + Junk

Comment 18

[WADA:World Anti-bad-Duping Agency]

Phenomenon of this bug is IMAP account version of Bug 505949. Bug 505949 is for problem on local account(POP3, Local Folders) when junk folder doesn't exist.

Comment 23

[Mark Banner (:standard8)]

This bug and some of it's duplicates feel like blockers. Therefore marking and passing to bienvenu initially.

Comment 24

[David :Bienvenu]

WADA, is this specific to the Junk folder (e.g., does it occur for any other type of special folder, like Sent)? I have a fix for the Junk folder case, but it's specific to code that only runs for the Junk folder.

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)
> is this specific to the Junk folder?

Yes, I could see problem for Junk only.
As written in Bug 505949, "fall back to Trash" looks to occur on Local Folders(may be on POP3 too. Bug 516183). Code around it seems to have some problems on both IMAP account and POP3/Local Folders account.

Comment 26

[David :Bienvenu]

Attachment: this fixes the server case

This handles the server case, which I think should be sufficient for rc1.

Comment 27

[WADA:World Anti-bad-Duping Agency]

Checked with next build(patch for Bug 505949 is landed).
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.4pre) Gecko/20090925 Shredder/3.0pre

I couldn't see "IMAP folder of account name" by ordinal problem recreation procedure I wrote in this bug. I could see "IMAP folder of account name" by next crafted procedure only.
  (1) Junk Settings : Others: IMAP account's existent folder
  (2) Config editor : mail.server.serverN.spamActionTargetFolder
                      = imap://<userName>@<servername>/
                   or = imap://<userName>@<servername>
  (3) Change account label of this IMAP account.
  (4) IMAP folder of account name for <userName>@<servername> is created
      at IMAP account of <userName>@<servername>.
Your patch looks to be able to resolve this crafted case also.

If spamActionTargetFolder is set to non-existent folder,
  e.g. imap://<userName>@<servername>/Non-Existent-Junk,
  emulation of delete of the IMAP folder by other client
or if Junk Settings is "Junk folder on: an IMAP account" and Junk folder doesn't exist(original case of this bug), problem of "IMAP folder of account name" didn't occur. In these cases, root-level Junk looks to be created.
It looks resolved by patch of Bug 505949 for root cause.

New phenomenon was observed.
If dummy IMAP account is selected at "Junk folder on:", nothing happens upon setting change. And manually marked Junk mail is lost. Mail is simply deleted from mail folder with storing flag for Junk(no connection error message etc. was observed).
  At Inbox, mark as Junk -> mail disappears from Inbox.
  The mail is not moved to Junk because selected Junk doesn't exist.
  If Gmail IMAP, Junk mark can be seen on mail copy in [Gmail]/All Mail.
I think this is problem of Junk feature instead of networking IMAP. However, I think error(IMAP folder of non-accessible account) is beter to be detected by networking:IMAP while processing Junk Settings, if possible.

Comment 28

[WADA:World Anti-bad-Duping Agency]

To Hendrik Duerkop(bug opener) and reporters of DUPed bug:

Patch for Bug 505949 is landed. Can you reproduce problem with newest nghtly build?

Comment 29

[WADA:World Anti-bad-Duping Agency]

Creation of root-level Junk seems current design/implementation. Removing it from bug summary to avoid misleading.

Comment 30

[Hendrik Duerkop]

works pretty well one two of three machines. the third machine i will test on monday.

Comment 31

[David :Bienvenu]

I think the fix for bug 505949 makes this non-blocking, though I'd certainly land this fix once it is reviewed.

Comment 32

[WADA:World Anti-bad-Duping Agency]

I could see file of <server_name>.msf (imap.googlemail.com.msf for an Gmail IMAP account created by autoconfig) sometimes, even after I deleted the garbage .msf file.
It indicates that directory of "imap.googlemail.com" was opened as "mail folder file". (phenomenon of "directory without .sbd is treated as "mail folder file" is already know phenomenon.)
In the used profile, I unsubscibe root-level Junk folder for ease of testing, so root-level Junk is hidden. The garbage .msf may be produced by this bug.

Comment 33

[WADA:World Anti-bad-Duping Agency]

Culprit is perhaps garbage of "account-label-name" in <hostname>.msf. Because mbox name of <hostname> (directory!) is changed from "" to the garbled name, root of create is changed to the garbled mbox name.
See bug 720911 for this phenomenon with folder of "# in folder name".

Cause of garbage may be different, but mechanism of garbage production is similar and phenomenon after garbage is same.
Setting dependency to bug 720911, for ease of tracking.