30.85 KB, text/plain
19.75 KB, text/plain
4.64 KB, text/plain
13.83 KB, text/plain
10.19 KB, text/plain
1.98 KB, patch
|Details | Diff | Splinter Review|
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0 Build ID: 20130404104324 Steps to reproduce: Upgraded version 17.0.5 to version 20.0b1 and then check for new mail on uw-imap-2007f server Actual results: Thunderbird keeps "looking for folders" and nothing happens Expected results: New mail should be displayed. Downgrading to version 17.0.5 solves this problem and new mail is displayed
I've seen something like this in seamonkey nightly build, but only for one account, vseerror, that I was trying to add. we also use uws-imap can you determine please whether the problem is also seen in earlier betas https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/18.0b1/ and https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/19.0b1/
Component: Untriaged → Networking: IMAP
Product: Thunderbird → MailNews Core
18.0b1 and 19.0b1 both work fine in my situation. The problem starts after upgrading to version 20.0b1
5004=Looking for folders… is IMAP_STATUS_LOOKING_FOR_MAILBOX > http://mxr.mozilla.org/comm-central/source/mail/locales/en-US/chrome/messenger/imapMsgs.properties#36 > http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapStringBundle.h#21 IMAP_STATUS_LOOKING_FOR_MAILBOX is used by LSUB and LIST/XLIST, > http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#7544 > http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#7577 and is also used for LIST of Drafts, Sent, & Templates folder, in case of not-subscribed. > http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapProtocol.cpp#5818 Is setting of Drafts, Sent, Templates in Copies&Folders normal? Are they normally subscribed? Anyway, get IMAP log and check log around LSUB, LIST/XLIST, please. See bug 402793 comment #28 for etting log.
If I compare the two log-files I notice a difference in line 26, where version 19.0b1 has "lsub" and version 20.0b1 has "list" After this "list" version 20.0b1 starts browsing all my files instead of only my inbox in /var/spool/mail
Jan Kees, Thanks for the logs. Haven't had time to look at them yet, but it sounds related to some bugs I have worked on. Just want to let others know that I will have a look at this soon so they don't need to dig deeper unless they just really want to.
The reason that LIST is coming before LSUB in TB 20 is is bug 799821. See bug 799821 comment 2. The question now is why does the server hang when we send LIST "" "*". I have a virtual machine with uw-imap2007e, but cannot reproduce the problem. From the logs, I see that Jan Kees is using uw-imap2007f. Also in the back of my mind, I very vaguely remember ready something (from Timo of Dovecot maybe?) that maybe LIST "" "*" is not a good idea if there are shared folders or other certain conditions on the server. Could be remembering wrong though. So, to proceed fixing this, we need to figure out if this is just a server bug or if we need to fix something in Thunderbird. Wayne, Jan, can you tell more about your servers? From the log, it looks like Jam uses this server as a desktop client too. Since it looks like the LIST command is listing every file in your user profile, perhaps there is some file/directory that is not readable by the uw-imapd that it is getting stuck on.
Assignee: nobody → david
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
(In reply to David Lechner (:dlech) from comment #7) > Also in the back of my mind, I very vaguely remember ready something (from > Timo of Dovecot maybe?) that maybe LIST "" "*" is not a good idea if there > are shared folders or other certain conditions on the server. Could be > remembering wrong though. I think this is it: > 10. Thou shalt use the LIST "*" wildcard only with great care. If > thou doth not fully comprehend the danger of "*", thou shalt use only > "%" and forget about the existance of "*". from https://www.washington.edu/imap/documentation/commndmt.txt.html
didn't mean to cancel the info needed
Following is bug 858062 comment #5. > uncheck "show only subscribed folders" and problem goes away Jan Kees(bug opeer), is your IMAP server usable with the option unchecked? (Account Settings/Server Settings/Advanced) (In reply to Jan Kees from comment #5) > log from session 20.0b1 (session is stopped by killing thunderbird) Is following in sample of UW-IMAP intentionally kept? > * NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/")) This is sample which shows that "delimiter of /" and "delimiter of ." can be mixed, multiple kinds of namespace are usable and multiple namespaces is possible, etc. Do you actually need "#ftp/" and "#news." as namespace? See bug 697789 and bug 720911 for funny phenomena with namespace = "#news." in the sample setting of shipped UW-IMAP.
Jan Kees, in your log, I noticed that it hangs when accessing files in the .thunderbird/ folder in your home directory. Are you running thunderbird on the same machine as the uw-imap server? Perhaps both are trying to access a file at the same time.
(In reply to WADA from comment #10) > > Is following in sample of UW-IMAP intentionally kept? > > * NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/")) > This is sample which shows that "delimiter of /" and "delimiter of ." can be > mixed, multiple kinds of namespace are usable and multiple namespaces is > possible, etc. Sometimes I see this in my settings. Typically after first creating the account. Along these lines I sometimes have trouble making subfolders because of bug 720911
With "Show subscribed folders only" disabled.(Thunderbird/22.0a1, 20130325031104) Gmail IMAP. XLIST is supported. > :imap.gmail.com:A:SendData: 25 xlist "" "%" > :imap.gmail.com:A:SendData: 26 xlist "" "%/%" Yahoo! IMAP. XLIST is not supported. > :imap.mail.yahoo.com:A:SendData: 4 list "" "%" > :imap.mail.yahoo.com:A:SendData: 5 list "" "%/%" Tb already stops using the dangerous LIST "" "*", and Tb requests LIST for two level lower only, so bug 710507 occurred by bug of Yahoo! IMAP. 'LIST "" "*" with "Show subscribed folders only" Checked' is fault of bug 799821.
(In reply to Wayne Mery (:wsmwk) from comment #12) > Along these lines I sometimes have trouble making subfolders because of bug 720911 Wayne, you still love and enjoy phenomenon of bug 720911? :-)
Needless to say, failure in your UW-IMAP server by LIST "" "*" is bug of your UW-IMAP, and LIST response like following is apparently by your improper configuration. > * LIST (\NoInferiors \Marked) "/" .thunderbird.org/kfonxn7d.default/panacea.dat > * LIST (\NoInferiors \Marked) "/" .thunderbird.org/kfonxn7d.default/global-messages-db.sqlite
For phenomenon with LIST "" "*". Followin is a part of RFC 3501 6.3.8. LIST Command section. > http://tools.ietf.org/html/rfc3501#section-6.3.8 (breakout character) > Note: The interpretation of the reference argument is > implementation-defined. It depends upon whether the > server implementation has a concept of the "current > working directory" and leading "break out characters", > which override the current working directory. (accessing directory/file of file sytem by IMAP server) > For example, here are some examples of how references > and mailbox names might be interpreted on a UNIX-based > server: > Reference Mailbox Name Interpretation > ------------ ------------ -------------- > ~smith/Mail/ foo.* ~smith/Mail/foo.* > archive/ % archive/% > #news. comp.mail.* #news.comp.mail.* > ~smith/Mail/ /usr/doc/foo /usr/doc/foo > archive/ ~fred/Mail/* ~fred/Mail/* Because Tb requests LIST "" "*" without preseeding "select MboxName", UW-IMAP server perhaps interprets it as <home_directory>/*, <current_directory>/*. 'the danger of "*"' in comment #8 from David Lechner is perhaps for this behavior of UW-IMAp and some IMAP servers. Above is apparently merely a convention in server, merely a local rule in server, from perspective of protocol named IMAP. Because protocol named IMAP permits LIST "" "*" as valid IMAP command, server side configuration should exclude directory/file which is not used as Mbox from directory which is accessd as directory for Mbox by himself, if IMAP server blindly accesses directory/file derived from string in IMAP command which referes to Mbox in IMAP. Please note that; I know that 'LIST "" "*" without preseeding select MboxName by Tb' is never good behaviour. If IMAP client use LIST "" "*", IMAP client should issue select before it, or should use LIST "MboxName" "*" , according to warning or caution written in some RFCs, in oorder not to produce unwanted problems. As seen in Tb's behavior with "Show subscribed folder only" disabled, Tb already uses wildcard of "%" in LIST(LIST Mbox/%/%), instead of wildcard of "*", altough reason is performance reason in case of very many Mboxes. Wildcard of "*" is limited to LSUB "" "*". Because purpose of "LIST command when Show subscribed folder only is enabled" is to complement "lack of important attribute such as \Noselect in LSUB response from server", LIST should be issued to "Mbox returned to LSUB" only. I believe following is never good behavior; Because LSUB "" "*" is used, issue LIST "" "*" before LSUB, in order to get \Noselect, \Noinferior etc. in LIST response.
Guess on why LIST is issued before LSUB by the patch. It's LSUB for folder rediscovery, so at end of LSUB response parsing, folder pane is refreshed using LSUB response without \Noselect, or with wrong(misspelled) \NoSelect. LIST after it is too late to reflect \Noselect in LIST response. Therefore, LIST before LSUB was needed. (i) Change like "postpone folder rediscovery execution, 'after LSUB' to 'after LSUB+LIST'" may be needed. (ii) Because folder rediscovery is invoked by "account collapse/expand" too, change around it may be needed. (ii) Because "subscribe" is currently following, LSUB "" "*" LIST "" "%" LIST "" "%/%" (not *, two level only, for performance reason) "LIST for all Mbox in LSUB response" by Subscribe may also be needed. (vi) Because "No \Noselect in LSUB response, \Noselect in LIST respnse only" is problem in limited servers only, "make it optional" may be most appropriate quick solution for this kind of regression. Why can change by patch be mandatory for normal IMAP server users? (In reply to Jan Kees from comment #4) > Created attachment 735168 [details] > log from session with version 19.0b1 which works fine Following is seen in IMAP log. > * LIST (\NoInferiors) NIL INBOX Correct one defined by RFC 3501 is as follows. > \Noinferiors > \Noselect Your case looks "bug 799821 is mandatory" case.
I am not running TB on the same machine as the uw-imap server. But the machine running uw-imap can access my home-directory which is shared among many machines. TB is not stopping at my .thunderbird directory. I just killed TB so the log would not be that big. I know that I should not uncheck "show only subscribed folders", because then TB starts indexing my whole home-dir and I only want my see my mailbox.
(In reply to Jan Kees from comment #18) > I know that I should not uncheck "show only subscribed folders", > because then TB starts indexing my whole home-dir and I only want my see my mailbox. Your UW-IMAP server looks to return all your .thunderbird/.../<anystring>/<anystring> also to LIST "" "%", LIST "" "%/%"... For IMAP server like your UW-IMAP server, I believe best solution is "making change like bug 799821 optional always", because LSUB "" "*" is used as circumvention of funny phenomena due to not-proper configuration because of personal use of UW-IMAP.
I believe server should be properly configured. - If .../Mail/Mbox is mapped to top level Mbox folder and if .../Mail/Sub is mapped to Mbox/Sub, put string like .../Mail at somewhere of UW-IMAP configuration file, in order that "*", "%" is correctly mapped to .../Mail, unless your are testing Tb's behaviour with wrongly configured sever. - Remove "#news." in namespace setting, which is merely example to show that "delimiter of /" and "delimiter of ." can be mixed, which is never "default what is practically good and meaningful", unless you are eager to enjoy bug 697789 and bug 720911. - If you actually want to use namespace, and if you don't want problem like bug 517054, correct pretty old bug of wrong \NoInferiors \NoSelect in UW-IMAP, (proper one is \Noinferiors \Noselect) configure server to return \Noselect in LSUB response as Tb expects, and configure server properly to return \Noselect for Mbox used for namespace as Tb expectsts.
(In reply to Jan Kees from comment #18) > I know that I should not uncheck "show only subscribed folders", > because then TB starts indexing my whole home-dir and I only want my see my mailbox. If undersand correctly, Tb issues LIST "" "%" and LIST "" "%/%" by "Subscribe" in both "Show subscribed folders only" Checked and Unchecked, in order to show all Mboxes in subscription list. This is same as first LIST command when "Show subscribed folders only" is Unchecked. In this case, does server freeze like problem happen in your UW-IMAP server? I guess No, because returned Tb's directory/files are .thunderbird.org directory and .thunderbird.org/kfonxn7d.default only in your case. I guess freeze like problem in your UW-IMAP server occurs only when wild card of "*" is used to know all available Mboxes. Your UW-IMAP server tries to return all directories and files under .thunderbird directory and tries to check all directories and files in order to return \NoInferiors(correct one is \Noinferiors in RFC3501), \NoSelect((correct one is \Noselect in RFC3501), \HasChildren, \HasNoChildren, \Marked, \UnMarked etc. Because \Noinferiors, \Noselect, \HasChildren, \HasNoChildren, are set by directory/file existent check only, cause of freeze like phenomenon is perhaps file content check of all Tb's filee for mail folder for \Marked and \UnMarked. > http://tools.ietf.org/html/rfc3501#section-6.3.8 > The LIST command SHOULD return its data quickly, without undue > delay. For example, it SHOULD NOT go to excess trouble to > calculate the \Marked or \Unmarked status or perform other > processing; if each name requires 1 second of processing, then a > list of 1200 names would take 20 minutes!
Jan Kees(bug opener), when did you start to use Tb with your UW-IMAP server? IIRC, change from "*" to "%/%"(2 level only) in LIST was by not-so-old Tb release, so, if you are using Tb for long time, I think same problem as this bug surely happened by Subscribe...
Summary: upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server → upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change of 'LIST "" "*"' use' by bug 799821)
Summary: upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change of 'LIST "" "*"' use' by bug 799821) → upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change to 'LIST "" "*"' use again' by bug 799821)
Summary: upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change to 'LIST "" "*"' use again' by bug 799821) → upgrading to 20.0b1 hangs on "looking for folders" on uw-imap server (Bad configuration of an UW-IMAP server was unfortunately exposed by unplesant change to 'LIST "" "*" use again' by bug 799821)
I'm using this uw-imap setup for many years. First with mozilla and later with thunderbird. I never changed anything in the setup for years. It always worked fine until version 20.0b1
You look to have Inbox and Trash only. So, there is no need of subscribe/unsubscribe. Did you try "Subscribe" in the past(Mozilla era, when swtchedd to Tb)? If not, and as far as "Show subscribed only" was Checked, I think 'LIST "" "*" without preceding select nor Mox name' didn't happen even in old Mozilla/Tb who issues 'LIST "" "*" upon subscribe. By the way, Tb prior to 20 used wildcard of "*" in normal situation, but it's by "select Trash" and 'LIST "" "Trash/*"' at connection where Trash is selected(seen in your log).
I've never used "Subscribe" but "Show subscribed only" needs to be Checked to avoid listing my whole home-dir in TB-versions prior to 20
ran thunderbird 21.0 after the following: export NSPR_LOG_MODULES=imap:5 export NSPR_LOG_FILE=/tmp/imap.log killed thinderbird after a while, and deleted most of the lines from the log (seems to be listing my home directory).
Sorry, I meant to preceed Comment 26 by the following (which I had added to bug 834705). I upgraded to Thunderbird 21.0 and now get "Looking for folders" when I startup. I am unable to access new messages. If I uncheck "Show only subscribed folders" and restart I no longer get the messages. The problem is that I have accumulated a huge number of folders over that last few decades and this makes saving email to the appropriate folder unwieldly. The server is uw-imap which is running more or less unchanged soince 2009. If I can supply uselful logs, etc., please let me know and I will do my best.
Further apologies for noise: Comment 26 should read 'part of log from thunderbird'. Also, uw-imap has no configuration files as such (other than changing the code and recompiling).
(In reply to Joe H from comment #26) > part of log from uw-imap server Log data. (1) Namespace response. It lokks your UW-IMAP server is also used with no customization in namespace setting... Why namespace of "#news."(delimiter=="."!) is needed? > * NAMESPACE (("" "/")("#mhinbox" NIL)("#mh/" "/")) (("~" "/")) (("#shared/" "/")("#ftp/" "/")("#news." ".")("#public/" "/")) (2) LIST command used by Tb. > 5 list "" "*" (3) LIST command response. > * LIST (\NoSelect \HasChildren) "/" .icedteaplugin > * LIST (\NoSelect \HasChildren) "/" .icedteaplugin/cache > * LIST (\NoSelect \HasChildren) "/" .icedteaplugin/cache/http ... (3-A) Your UW-IMAP server also returns all sub directories as Mbox... It's perhaps true for LIST %, LIST %/% which is used by Tb when "Show subscribed folders only" is Unchecked... (3-B) \NoSelect... You also don't look to enable fix of UW-IMAP's bug... Look into already attached log data by bug opener and compare with your log by yourself before attaching same log data to bug, please.
(In reply to WADA from comment #29) > (3-B) \NoSelect... You also don't look to enable fix of UW-IMAP's bug... I'm using the stock uw-imap server from before 2009 or so. It has no configuration other than reompilation. I have changed nothing other than upgrading Thunderbird to a later version. I am happy to enable a fix of uw-imap's bug, but I'm not sure what you are referring to?
(In reply to Joe H from comment #30) > > (3-B) \NoSelect... You also don't look to enable fix of UW-IMAP's bug... > I am happy to enable a fix of uw-imap's bug, but I'm not sure what you are referring to? Sorry for misleading comment. It may not be actual "bug" of IMAP sever and it may be Tb side bug, because \Noselect, \Noinferior etc. may be case insensitive. IIRC, UW-IMAP or Dovecot already shipped with optional change. e.g. "string change from \NoInferiour to \Noinferiour" etc. in his optional code or optional setup file. Even if it's Tb side bug, problem around \NoSelect, \NoInferior can be easily bypassed by such optional change. This and "#news." is merely an evidence that you use UW-IMAP with minimum customization to use some Mboxes, if UW-IMAP shipped such optional change. If you don't have problem like "Mbox of \NoSelect(can't hold messages in it) is not shown as grayed out folder in Tb", there is no need to force \Noselect, \Noinferior etc., unless you are suffering from problem like bug 317597 comment #22. By the way, bug 799821, which produced this bug, is a solution for issue of bug 317597 comment #22 what is produced by UW-IMAP. And, "LIST *" by bug 799821 is never RFC violation by Tb. "Who is bad" is apparently server who returns any sub directories of current directory as Mbox in IMAP server. What is bad in this bug is; after patch of bug 799821, "Show subscribed folders only" can't be used any more as bypass of "Server returns any sub directories of current directory as Mbox in IMAP server to LIST *, LIST %/% etc.", in bug opener's environment and your environment. As seen in bug 799821, developer is trying to backout change by that bug and is tryng to find better solution for issue of bug 317597 comment #22 what is produced by UW-IMAP.
This patch prevents this bug in cases where the user has set "IMAP Server Directory" in Advanced Server Settings.
This is an IMAP log before applying the patch from :irving. You can see that it lists all of the files in the users home directory.
This is the IMAP log after applying the patch. It only lists the mail/ directory.
In reply to Joe Pruett (Bug 799821 Comment 38) >why isn't it just as simple as making sure to pass the folder prefix to the list command? The patch I just posted does just this. However, when you first create an account, the folder prefix does not necessarily exist - you may have to add it manually. So, we still would see the effects of this bug when setting up an account for the first time.
that has always been the case. my way around that is i don't give the password until after i have configured the advanced settings.
I was going to attempt something more sophisticated, but I see now that this really is just an unfortunate side effect of UW-IMAP (and possibly other servers) exposing the users entire home directory. There is really not anything else to do since the folks with these servers already know the issue and the workarounds. Also, I am at a loss as to what how we could make a unit test that handles this - at least not without adding a huge amount of complexity to fakeimap.js. Would this patch be acceptable without a test, or do I need to spend some more time on it?
(In reply to David Lechner (:dlech) from comment #39) > Also, I am at a loss as to what how we could make a unit test that handles > this - at least not without adding a huge amount of complexity to > fakeimap.js. Would this patch be acceptable without a test, or do I need to > spend some more time on it? I think I'm happy for this to go in without a test. I think it would be useful though, to add some comments in both situations so that future devs know why the parameters are there.
Is this the type of comment you were suggesting?
This patch includes a useful commit message.
Comment on attachment 8473103 [details] [diff] [review] patch-v3 Yep, that's great. One minor nit "dirctory" is spelt wrong.
Attachment #8473103 - Flags: review?(standard8) → review+
Attachment #8473103 - Attachment is obsolete: true
Checkin-needed for patch-v4.
Comment on attachment 8473752 [details] [diff] [review] patch-v4 [Approval Request Comment] Regression caused by (bug #): User impact if declined: Testing completed (on c-c, etc.): Risk to taking this patch (and alternatives if risky): Don't we want this in comm-aurora too since we backed out Bug 799821 only in beta and release?
Attachment #8473752 - Flags: approval-comm-aurora?
It has been a while since I've done this. Was I supposed to whiteboard checkin-needed instead of keyword?
No, the tree has just been (more or less) closed :(
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 34.0
Attachment #8473752 - Flags: approval-comm-aurora? → approval-comm-aurora+
Great to see this going into thunderbird 31.1.0 because many users report issues. (probably moreso after version 24 came out than they report now) But I do not have many examples ready at hand, and searching https://support.mozilla.org/ is a serious PITA. Maybe christ1 and matt have examples they can cite, and help us get feedback from users who have reported issues. On probable example of Bug 1046287 (duped to this bug) - imap server directory setting not working anymore - is https://support.mozilla.org/en-US/questions/1012613 where the user gets 'too many connections' if "imap server directory" is set.
You need to log in before you can comment on or make changes to this bug.