376192 - Thunderbird crashes immediately upon accessing IMAP server (duplicate entries in .mailboxlist) [@ nsImapServerResponseParser::mailbox] - imap protocol log "Internal Syntax Error

Status: RESOLVED FIXED

(MailNews Core :: Networking, defect)

Description

[Steuard Jensen]

Crash seen in Thunderbird 2.0rc1 (supposedly, at least: downloaded from ftp.mozilla.org 1 Apr. 2007 for Vista; on 31 Mar for Mac OS 10.3.9).

I have a particular IMAP account that seems to immediately kill Thunderbird when it is accessed.  As soon as I log in to the account, there is a brief flicker in the folder list (associated with the "searching for folders" message) which almost looks like the Inbox disappears and reappears (or some folder up near the top), and then the program crashes.

This IMAP account is hosted on some friends' Linux computer; I haven't heard back yet on what software they're using.  It may be relevant that they are using a self-signed SSL certificate that is technically expired.  Whether or not that is related to the problem (I doubt that it is), it does give me the opportunity to isolate the problem: my other IMAP account works perfectly fine as long as I refuse to accept the expired certificate from the problem server.  The moment I accept that certificate, Thunderbird crashes as described.

I've included Talkback IDs for my crashes on Windows Vista below; for whatever reason, Talkback doesn't launch on my Mac.  (Does Talkback actually run on the Mac?)  I really don't know which bits from the Talkback reports are most useful to include here; I'll just include the stack signature, trace, and the Talkback IDs, and leave the rest to the experts.

Talkback IDs: TB30794811X, TB30794134M, TB30794065E, TB30794051E, TB30794038X, TB30793169G, TB30793040M.

Stack signature: nsImapServerResponseParser::mailbox e7d28eeb

Stack trace:
nsImapServerResponseParser::mailbox  [mozilla/mailnews/imap/src/nsImapServerResponseParser.cpp, line 964]


I have also created an IMAP protocol log, but I'm a little nervous about posting it publicly since it contains details of my mail accounts and folder structure.  However, it does include the following rather interesting lines (with server name removed):

3480[3a8f7c0]: ReadNextLine [stream=39612c8 nb=39 needmore=0]
3480[3a8f7c0]: 39c0d28:<***server name***>:A:CreateNewLineFromSocket: * LIST () "/" Mail/IMAP/Sent Messages
3480[3a8f7c0]: 39c0d28:<***server name***>:A:PARSER:Internal Syntax Error on line: %s: * LIST () "/" Mail/IMAP/Sent Messages

This sequence is repeated a total of three times near the end of the log.  It's especially odd to me since I don't actually have any folder named "Sent Messages" in this account (all of the other folder names in the log were accurate, though, apart from the mention of a "Trash" folder which may just be due to not having the account fully configured yet).  The final ten lines of the log read as follows:

3480[3a8f7c0]: ReadNextLine [stream=39612c8 nb=33 needmore=0]
3480[3a8f7c0]: 39c0d28:<***server name***>:NA:CreateNewLineFromSocket: * LIST (\NoInferiors) NIL INBOX
3480[3a8f7c0]: ReadNextLine [stream=39612c8 nb=21 needmore=0]
3480[3a8f7c0]: 39c0d28:oasis.slimy.com:NA:CreateNewLineFromSocket: 6 OK LIST Completed
3008[3a8fa50]: 39c0498:<***server name 2***>:S-INBOX:SendData: 13 IDLE
3480[3a8f7c0]: 39c0d28:<***server name***>:NA:SendData: 10 logout
3480[3a8f7c0]: ReadNextLine [stream=39612c8 nb=43 needmore=0]
3480[3a8f7c0]: 39c0d28:<***server name***>:NA:CreateNewLineFromSocket: * LIST (\NoInferiors) "/" Mail/IMAP/Trash
3008[3a8fa50]: ReadNextLine [stream=3961368 nb=10 needmore=0]
3008[3a8fa50]: 39c0498:<***server name 2***>:S-INBOX:CreateNewLineFromSocket: + idling

Comment 1

[Steuard Jensen]

Two more Talkback reports, this time with the latest nightly build, a fresh install, and a fresh profile.  The Talkback IDs are TB30929755 and TB30929934.

Comment 2

[Steuard Jensen]

Looking through talkback reports with the same stack signature (specifically, looking for e7d28eeb), it looks like related crashes go back to the earliest reports still in the database.  The earliest build showing this crash signature that I found there was "MozillaOrgThunderbird15Win322006071912"; there are also quite a few reports from users of Thunderbird 1.5.*.  The other crash descriptions there sound at least generally related to the issue that I'm seeing.

For what it's worth, I'm connecting to the IMAP server with SSL; a number of the other Talkback reports that included comments mentioned SSL or TLS as well.  (I'm not sure whether that's directly related, but it can't hurt to mention it.)

Comment 3

[Steuard Jensen]

Interesting.  Looking at some other IMAP bugs, I was referred to the .mailboxlist file governing IMAP folders in my account.  For some reason, that file included /three/ copies of a folder "Mail/IMAP/Sent Messages": one which I have never actually created (not intentionally, at least: I use "Sent" for outgoing mail).  Strangely, there was no entry in that list for "Mail/IMAP/Sent" at all.

After I edited the file by hand to eliminate the unwanted "Sent Messages" folders and to include "Sent" instead, Thunderbird no longer crashes (at least on Mac; I haven't had a chance to try on Windows yet).  Thus, the crash seems to be related to having multiple copies of the same folder name in the .mailboxlist file.  (I also deleted the empty "Sent Messages" file that had appeared in the directory itself.)

The question of how those duplicate folder names got there is another issue (particularly with a folder name that I've never used).  It may be worth noting that Thunderbird did not crash the first time I installed it and gave it a spin, but I had some other odd behavior (probably server related) that led me to uninstall and then reinstall.  So I'm pretty confident that something in the process of installing Thunderbird on my Mac and on my Windows Vista computer caused those duplicate entries in .mailboxlist in the first place.  Unfortunately, I have absolutely no idea how to track down that problem (which is probably a separate bug anyway).

Comment 4

[Steuard Jensen]

Someone wisely just cleared the "blocking-thunderbird2" request that I'd made when filing the bug.  I'm scaling back the severity of the bug as well: I still think it's a significant problem that ought to be fixed (and the fix would probably be pretty simple to someone who knows the code: just a simple sanity check when reading .mailboxlist could probably avert the crash), but I also suspect that in practice the issue will only arise with a tiny fraction of mail servers (probably old and buggy ones).  In particular, most of my IMAP issues with Thunderbird have wrapped up neatly since the server I use was upgraded (I'm not sure what the buggy server software was).

Comment 5

[Wayne Mery (:wsmwk)]

severity in this case is based more on how it acts more than how it impacts you.

Is this highly reproducible for you on that one account?
Does it happen with a trunk build?
 ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/

Do any of the other open bugs with "Internal Syntax Error" contain other similarities with your situation? ...
Bug 133689 – problems loading messages after getting "Internal Syntax Error"
Bug 112161 – InterMail vM.4.01.03.23: Getting "PARSER:Internal Syntax Error" when mail contains NULL char (\x0)
Bug 336619 – Crash if IMAP literal contains data after chunk boundary

Comment 6

[Steuard Jensen]

This bug /was/ 100% reproducible on that one account, but dropped immediately to 0% reproducible after I fixed my .mailboxlist file there to remove the duplicate entries.  (And ever since my IMAP server was upgraded, I haven't had /any/ IMAP problems with Thunderbird.)  I am very hesitant to re-break my mail account settings just to test this bug: I don't have a "test account" to do it with, and I really hate running risks with my primary mail.  (Also, it's possible that the upgraded server would be smart enough to correct the error itself; I don't know.)

If /you/ have a test account and want to try this out, my belief is that you can test it by hand easily: just open .mailboxlist manually in a text editor and duplicate a folder name (in my case, I had three identical entries for a folder named "Mail/IMAP/Sent Messages").  I'm pretty sure that just doing that would have made Thunderbird start crashing again (it could conceivably be relevant that no such folder actually existed, but my impression at the time was that the duplication was the main problem).  But as I said, it's possible that newer IMAP servers would correct this themselves; I have no data one way or the other.

I still don't know why Thunderbird created any of those three folders to begin with (much less three of them, which must have been a bug); I had a "Sent" folder already.

Finally, at least at first glance none of those other "Internal Syntax Error" bugs are related to this one.  Only one of them actually caused a crash, and they all seem to be related to problems with downloading individual messages whereas the issue in this bug occurred pretty much the instant the account was accessed at all.  But I would emphasize my guess that a simple "no duplicates" sanity check while parsing .mailboxlist would at least avoid the crash!  (The best behavior would probably be to just silently remove any duplicate entries, and ideally re-write .mailboxlist with them removed.)


As for severity, I realize that severity isn't just a "does it affect me" thing, but I guess I also thought that it was related to "does it affect any substantial fraction of the current user base".  I'm guessing that only people getting mail from very antiquated (and buggy) IMAP servers would be likely to encounter this.  But now that you mention it, I can see that you're right: "severity" ought to be just based on the actual effect observed, and the bug could be marked INVALID or WONTFIX if it was no longer relevant.

Comment 7

[Greg]

Hi there,
(This is my first bug comment, so please bear with me.)
I've got an account that crashes consistently when I log in with Thunderbird.  It's really quite annoying.
Here's one example crash report: TB 49082955

OS: Windows 2003 Enterprise Edition. Also seen on Windows XP Pro
Thunderbird: All versions of 2.0 (2.0Beta1 through 2.0.0.16) and no versions of 1.5
I'm connecting to an IMAP server.  The problem was when my office upgraded Exchange email servers from 2003 (i think) to 2008 (i think). I don't really know that part.

I'll take a look at the .mailbox for duplicate entries.

Comment 8

[Greg]

ok, I couldn't figure out how to see this ".mailboxlist" file.  Is it supposed to be on the imap server, or my client computer, or inside the thunderbird display?

Comment 9

[Wayne Mery (:wsmwk)]

David, should component be networking:imap?

Greg, TB49082955 no longer on the server. If you still see the crash, can you post another crashid?

Comment 10

[Greg]

Hi Wayne,
I haven't tried the test case recently.  I will try to install the right version for a reproduction tomorrow.

Thanks,
Greg

Comment 11

[Greg]

Hi Wayne,
I installed Thunderbird 2.0.0.19 and logged into my work email. It crashed again.  

See:
http://talkback-public.mozilla.org/search/start.jsp?search=2&type=iid&id=TB52513413W

Comment 12

[Wayne Mery (:wsmwk)]

topcrash for 3.0b1 (~top 15 if you exclude cruft)

TB52513413W entire stack is 
nsImapServerResponseParser::mailbox  [mozilla/mailnews/imap/src/nsImapServerResponseParser.cpp, line 964]

TRUNK
bp-a517e1c8-31e4-46ec-b466-439142090119

nsImapServerResponseParser::mailbox	nsImapServerResponseParser.cpp:960
nsImapServerResponseParser::mailbox_list	nsImapServerResponseParser.cpp:885
nsImapServerResponseParser::mailbox_data	nsImapServerResponseParser.cpp:813
nsImapServerResponseParser::response_data	nsImapServerResponseParser.cpp:577
nsImapServerResponseParser::ParseIMAPServerResponse	nsImapServerResponseParser.cpp:238
nsImapProtocol::ParseIMAPandCheckForNewMail	nsImapProtocol.cpp:1757
nsImapProtocol::Logout	nsImapProtocol.cpp:6032
nsImapProtocol::TellThreadToDie	nsImapProtocol.cpp:1175
nsImapProtocol::ProcessCurrentURL	nsImapProtocol.cpp:1718
nsImapProtocol::ImapThreadMainLoop	nsImapProtocol.cpp:1284
nsImapProtocol::Run	nsImapProtocol.cpp:1034
nsThread::ProcessNextEvent	xpcom/threads/nsThread.cpp:510
NS_ProcessNextEvent_P	nsThreadUtils.cpp:227
nsThread::ThreadFunc	xpcom/threads/nsThread.cpp:254
_PR_NativeRunThread	nsprpub/pr/src/threads/combined/pruthr.c:436
pr_root	nsprpub/pr/src/md/windows/w95thred.c:122
msvcr80.dll@0x29ba	

common comments on talkback:
subscribing
starting up
import
Emptying the deleted mailbox
- Expanding folders. I see an extra folder that does not belong. It sometimes will go away by colapsing and expanding the folder group.
unsubscribe a folder.
- I was logging into my Exchange email account. see [Bug 376192] Thunderbird crashes immediately upon accessing IMAP server (duplicate entries in .mailboxlist) [@ nsImapServerResponseParser::mailbox] - imap protocol log "Internal Syntax Error
- clicked on inbox and email client closed. This started happening as soon as I configured two IMAP accounts. I did not have this problem in the Mozilla Suite with multiple IMAP accounts.

Comment 13

[Magnus Melin [:mkmelin] (away, back Aug 4)]

http://hg.mozilla.org/comm-central/annotate/ae57fee4ac62/mailnews/imap/src/nsImapServerResponseParser.cpp#l960

Comment 14

[David :Bienvenu]

taking, plussing. If someone has a test account where I can reproduce this, that would be extremely helpful. A little null check bulletproofing might help here, but it would be great to reproduce the problem.

Comment 15

[Greg]

>
> If someone has a test account where I can reproduce this,
that would be extremely helpful. 
>

I don't have a test account for you, but I'd be willing to give you more info to debug (not sure what, though), or I could try out fixes for you.

Comment 16

[David :Bienvenu]

Greg, can you try a TB 3.0 b2 build and see if it still crashes for your test case? If so, an imap protocol log might be useful: https://wiki.mozilla.org/MailNews:Logging 

thx!

Comment 17

[Greg]

I will try out 3.0 beta 2 soon and let you know.

Comment 18

[Greg]

Yes, it works!!!  (My test setup: Windows Server 2003 Enterprise Edition, Thunderbird version 3.0b2)
Thank you!

-Greg

Comment 19

[David :Bienvenu]

I still see this crash in crash stats for b2, so it's not fixed in general, unfortunately (Greg, I wonder if your server stopped generating the responses that was crashing us). I think what's happening here is that we're getting a syntactically invalid protocol response, and trying to drop the connection, but we're still in the middle of receiving the mailbox list, so when we try to logout and parse the server response, we're still getting mailbox list stuff, and our parser crashes. We may be able to fix the crash, but the user is still not going to be able to use the server, because of the syntax error. So it would be helpful if someone who still has this issue could generate a protocol log. Greg, on the off chance that you can still crash with a previous build, a protocol log would be very helpful.

Comment 20

[Greg]

Aw, that's too bad.  I can give an older build a try.  Do you have a preference? v3.0 beta 1? the latest v2.0?

Comment 21

[David :Bienvenu]

Greg, as long as the build crashes in that same way, a protocol log from either one is fine.

Comment 22

[David :Bienvenu]

Attachment: proposed fix

essentially add a null check instead of an assertion. This is a somewhat speculative fix but judging from crash-stats, we're dereferencing a null pointer, and I think we've already cleared the url pointer in this case.

Comment 23

[Greg]

Ok, I tested several more versions- no luck.
If I'd known a protocol log would have been useful, I could have taken one a long time ago...

Here are the ones I tried:
1.5.0.9 did NOT crash.
2 beta 1 did NOT crash.
2.0.0.4 did NOT crash.
2.0.0.19 did NOT crash.
3 beta 2 did NOT crash.

Comment 24

[Mark Banner (:standard8)]

Comment on attachment 365572 [details] [diff] [review]
proposed fix

This looks to me like it could regress bug 211793, but was the problem in that bug the fact that we were doing boxSpec->hostName = nsnull, not the fact we were doing the null checks?

Comment 25

[David :Bienvenu]

that's a good question, but I don't think this would regress bug 211793 - if the connection or current url were null, we would crash - otherwise we execute the same code we would before.

Greg, I think your server stopped generated the invalid protocol that was causing us to stop the mailbox list and crash.

Comment 26

[Mark Banner (:standard8)]

Comment on attachment 365572 [details] [diff] [review]
proposed fix

Ok, let's give this a try.

Comment 27

[David :Bienvenu]

fix checked in - we'll have to check talkback for this crash in builds with this fix.

Comment 28

[Wayne Mery (:wsmwk)]

~#50 crasher for 3.0b1 and 3.0b2

#185 crash for TB20019

to assess how this patch might match up with reports, here are comments from TB2 MozillaOrgThunderbird2Win322008120920 crashes

Attepting to subscribe to an IMAP folder on a micorsoft exchange server.

clicked on inbox and email client closed. This started happening as soon as I configured two IMAP accounts. I did not have this problem in the Mozilla Suite with multiple IMAP accounts.

Clicked on Manage Folder Subscriptions

Draft save

Emptying the deleted mailbox

I was logging into my Exchange email account. see [Bug 376192] Thunderbird crashes immediately upon accessing IMAP server (duplicate entries in .mailboxlist) [@ nsImapServerResponseParser::mailbox] - imap protocol log "Internal Syntax Error

Made configuration change to Account Settings, changed secure connections from: TLS, when available to None.

Opening Thunderbird

Past leaving the account to "local folder" , or changing to an other mail account Thunderbird just closes itself. No message, no notice

Setting up New account to use Izymail protocols

Setting up security for SMTP for AOL mail

Tried to subsribe to additional folders

trying to connect to an exchange server

trying to subscribe to exchange(via IMAP) folders

trying to subscribe to IMAP folders

Watching IMAP subscribe view and closing the window.

Comment 29

[Wayne Mery (:wsmwk)]

mark, bienvenu, what do you think?
not sure how to read this - mostly gone from trunk according to crash-stats as of 2/23 build*. but one crash with 04/08 build bp-f99753a2-bbb3-45ea-a6bf-5b26e2090423

* http://crash-stats.mozilla.com/report/list?product=Thunderbird&version=Thunderbird%3A3.0b2&version=Thunderbird%3A3.0b2pre&version=Thunderbird%3A3.0b3pre&version=Thunderbird%3A3.1a1pre&query_search=signature&query_type=contains&query=nsImapServerResponseParser%3A%3Amailbox&date=&range_value=24&range_unit=days&do_query=1&signature=nsImapServerResponseParser%3A%3Amailbox%28nsImapMailboxSpec*%29

Comment 30

[David :Bienvenu]

I'm not sure that Eudora had the fix (I'm not sure how we map eudora executable versions to our TB versions in talkback)

Comment 31

[David :Bienvenu]

Comment on attachment 365572 [details] [diff] [review]
proposed fix

simple null check fix...

Comment 33

[Daniel Veditz [:dveditz]]

Comment on attachment 365572 [details] [diff] [review]
proposed fix

Approved for 1.8.1.24, a=dveditz for release-drivers

Comment 34

[David :Bienvenu]

Attachment: fix as landed

patch needed a bit of tweaking for 2.0

Comment 35

[Al Billings [:abillings - ex-MoCo]]

Anyone encountering this could check to see if it is fixed with the 1.8 nightly builds at ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-mozilla1.8/.

Currently, there is no way for QA to verify this fix.

Comment 36

[Greg]

(In reply to comment #35)
> Anyone encountering this could check to see if it is fixed with the 1.8 nightly
> builds at ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-mozilla1.8/.
> 
> Currently, there is no way for QA to verify this fix.

Sorry, I no longer have access to the account where I was seeing this.