303114 - Thunderbird mangles IMAP server directory path string

Status: RESOLVED INVALID

(Firefox :: General, defect)

Description

[Richard Ketcham]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.7) Gecko/20050416 Fedora/1.0.3-1.3.1 Firefox/1.0.3
Build Identifier: Thunderbird 1.0.2 (20050317)

We are running uw-imap server.  By convention, users' mail folders are stored on
the server in "~/Mail/".  This is set in the Thunderbird mail account property
named "Imap server directory" (via the dialog box at Account
Settings->Server->Settings->Advanced Account Settings->IMAP(tab)), either using
the absolute path "~/Mail/" or the relative path "Mail" (Either way should
work). Thunderbird sometimes mangles the path string, converting slash
characters (/) into pipe characters (|).  For example, ~/Mail/Sent becomes
~|Mail|Sent, or Mail/Trash becomes Mail|Trash.

I have closely examined the problem using Thuderbird 1.0.2 on Windows XP in a
simple and reproducible test scenario.  However, we have encountered many
variations of this problem in a variety of real-life scenarios, involving
various versions of Thuderbird clients and Mozilla mail clients, in combination
with various versions of uw-imapd, in various configurations.

Reproducible: Always

Steps to Reproduce:
1. Create a new user acount on a mail server running uw-imapd.  The new account
skeleton has an empty "~/Mail/" directory.  Initially, .mailboxlist does not exist.

2. Create a new imap user account in Thuderbird, with the following properties:  

  Imap server directory: ~/Mail/ (or, alternatively, just "Mail")
  Show only subscribed folder: CHECKED
  Server supports folders that contain subfolders and messages: UNCHECKED
  Use idle commnad if server supports it: CHECKED
  Allow server to override the name spaces: UNCHECKED

3. Connect to server and attempt to create or access folders on the server.

Actual Results:  
Upon initial connection, TB attempts to create a Trash folder on server.  Upon
sending a message, TB attempts to create a Sent folder on server.  But the path
sent in the IMAP CREATE command is mangled.  Therefore, the server either fails
to create folder or else creates folder with incorrect path/name.  

More specifically, TB translates slashes (/) into pipes (|), and so attempts to
create folder names like "~|Mail|Sent" (if using tilde path) or "Mail|Sent" (if
using relative path).


Expected Results:  
TB should have created the directories in their proper paths, ie. ~/Mail/Trash, etc.

Here is the exchange which follows the initial authentication (from tcpdump):

  UW-IMAPD -> Thunderbird IMAP Response: 2 OK [CAPABILITY IMAP4REV1 IDLE
NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES
THREAD=ORDEREDSUBJECT MULTIAPPEND] User testuser authenticated
  Thunderbird -> UW-IMAPD IMAP Request:  3 lsub "" "~/Mail/*"
  UW-IMAPD -> Thunderbird IMAP Response: 3 OK LSUB completed
  Thunderbird -> UW-IMAPD IMAP Request:  4 list "" "INBOX"
  UW-IMAPD -> Thunderbird IMAP Response: * LIST (\NoInferiors) NIL INBOX
  Thunderbird -> UW-IMAPD IMAP Request:  5 list "" "~|Mail|Trash"
  UW-IMAPD -> Thunderbird IMAP Response: 5 OK LIST completed
  Thunderbird -> UW-IMAPD IMAP Request:  6 create "~|Mail|Trash"
  UW-IMAPD -> Thunderbird IMAP Response: 6 NO CREATE failed: Can't create
~|Mail|Trash: invalid name

My explanation of the above: Thunderbird initially queries subscription list
(lsub) using the correct server path "~/Mail/*".  This being a newly created
account, the server correctly returns nothing (i.e., ~/Mail/* is empty). 
Thunderbird apparently doesn't like this response, and so Thunderbird mangles
the the server path string to ~|Mail|, and continues to use this mangled path
string for all subsequent communications with server, causing subsequent
commands to fail.

For comparison, I created another test case, the only differnce being that I
manually edited the uw-imap subscription list file, .mailboxlist, adding the
fake entry "~/Mail/nonexistentfolder".  The ~/Mail/ directory still empty, as
before.  Compare:

  UW-IMAPD -> Thunderbird IMAP Response: 1 OK [CAPABILITY IMAP4REV1 IDLE
NAMESPACE MAILBOX-REFERRALS BINARY UNSELECT SCAN SORT THREAD=REFERENCES
THREAD=ORDEREDSUBJECT MULTIAPPEND] User testuser authenticated
  Thunderbird -> UW-IMAPD IMAP Request: 2 lsub "" "~/Mail/*"
  UW-IMAPD -> Thunderbird IMAP Response: * LSUB () "/" ~/Mail/nonexistentfolder
  Thunderbird -> UW-IMAPD IMAP Request: 3 list "" "INBOX"
  UW-IMAPD -> Thunderbird IMAP Response: * LIST (\NoInferiors) NIL INBOX
  Thunderbird -> UW-IMAPD IMAP Request: 4 list "" "~/Mail/Trash"
  UW-IMAPD -> Thunderbird IMAP Response: 4 OK LIST completed
  Thunderbird -> UW-IMAPD IMAP Request: 5 create "~/Mail/Trash"
  UW-IMAPD -> Thunderbird IMAP Response: 5 OK CREATE completed
  Thunderbird -> UW-IMAPD IMAP Request: 6 subscribe "~/Mail/Trash"
  UW-IMAPD -> Thunderbird IMAP Response: 6 OK SUBSCRIBE completed
  Thunderbird -> UW-IMAPD IMAP Request: 7 list "" "~/Mail/Trash"
  UW-IMAPD -> Thunderbird IMAP Response: * LIST (\NoInferiors \UnMarked) "/"
~/Mail/Trash

This time the server responds to the initial lsub query, returning the fake
folder name.  Thunderbird is happy and goes on to create the Trash folder in the
proper (~/Mail/) directory.

So these test cases show clearly the nature of the problem in a simple,
reproducible scenario involving a newly created user account, but the problem,
unfortunately, is more pervasive.  (If the problem were only in creating new
user accounts, the above findings suggest a simple work-around.)  We also
encounter the same path-mangling in other common situations: (1) When migrating
a user to a new desktop PC and reconnecting to his existing imap server account
(2) in deployment testing of a new, upgraded mail server, when we migrate users
mail accounts to the new mail server and connect from existing clients.  In
these cases, when the user attempts to access an existing mailbox, he gets a
message like:

"The current command did not succeed.  The mail server responded: SELECT failed:
Can't open mailbox ~|Mail|Sent: no such mailbox." 

So the common thread here is that the server path has been mangled, and so the
folder (in this case a real, existing folder) cannot be accessed.  I have not
yet had the opportunity to analyze these cases in depth, but from what I know
now, I am guessing it might also have something to do with the contents server
subscription file, or more precisely, Thunderbird's pathological response to
what is being returned by the lsub command.  

It would seem that Thunderbird's refresh function (i.e., the "refresh" button
from Thunderbird's "Subscribe" dialog box) ought to be able to sort out any
discrepancies between the subscription list and what actually exists in the
server directory path, but I have observed that, when refreshing, Thunderbird
also queries the server using the mangled path, so nothing is found, and so the
"refresh" function is quite useless.

So in summary, I have experienced a variety of problems resulting from TB
mangling the server directory path string by translating slash (/) characters to
pipe (|) characters, and I have described a detailed test scenario for observing
this bug in one of its various manifestations.

Richard Ketcham
Mail Administrator
U.W. Dept. of Chemistry

Comment 1

[Richard Ketcham]

I think I posted this bug to the wrong product.  I have reposted it to
Thunderbird bug report # 303125.

Richard K.