926467 - imap error: "Could not parse command" started after upgrade to Version 24 (≪Body≫ or * is wrongly stored in mailnews.customHeaders by someone, then "fetch ... HEADER.FIELD (... ≪Body≫ * ...)" is sent by Tb 24, thus IMAP server returns BAD)

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Cliff Lewis]

Attachment: Log file of imap commands and responses

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20130910160258

Steps to reproduce:

As soon as Thunderbird upgraded from version 17.0.8 to 24.0.1, I started to get error messages when I tried to access mailboxes in any of my four imap accounts that are hosted by gmail. pop mail accounts worked normally with no problems. I used my backup program to roll back the Thunderbird program files and profile to the day before the problem started, and that stopped the problem. I allowed Thunderbird to upgrade automatically again, and the same problem resulted. I am now back to using Version 17.0.8 and preventing automatic upgrades.  This problem only occurs with the one profile; my wife's profile does not have the same problem. I have also tried creating a new profile with one of the gmail accounts, and it worked without problems. But when I tried to copy pop mail files and prefs.js from the old profile to the new one I started having the same problem, even though I deleted lines related to the imap accounts from prefs.js. My computer uses Windows 7 Professional, 64Gb, SP1.

I also posted a message about this 3 days ago at https://getsatisfaction.com/mozilla_messaging/topics/imap_could_not_parse_command?utm_content=reply_link&utm_medium=email&utm_source=reply_notification#reply_13109734.

It has received no replies except two other people saying they had the same problem.


Actual results:

The message reads "The current operation on 'Inbox' did not succeed. The mail server for account ********** responded: Could not parse command." The message occurs only when the mailbox contains a new message that has not been downloaded. I was not able to download the new messages, and I had to retrieve them through webmail. I also had a problem with one account not accepting the password. A log file is attached.


Expected results:

New messages should have been retrieved without errors.

Comment 1

[WADA:World Anti-bad-Duping Agency]

Confusing but multiple IMAP log lines(no new line char) in attached log file is merged to single line which terminates with CRLF.

Line contains "14 UID fetch" in attached log file, for Inbox.
> 6616[9b864f0]: 96a1800:imap.googlemail.com:S-INBOX:CreateNewLineFromSocket: 13 OK Success
> 6616[9b864f0]: 96a1800:imap.googlemail.com:S-INBOX:SendData: 14 UID fetch 1913:1915 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To ≪Personality≫ ≪Body≫)])
> 6616[9b864f0]: ReadNextLine [stream=5827e80 nb=32 needmore=0]
Line contains "14 BAD" in attached log file, for Inbox.
> 6616[9b864f0]: 96a1800:imap.googlemail.com:S-INBOX:CreateNewLineFromSocket: 14 BAD Could not parse command
> 6616[9b864f0]: 96a1800:imap.googlemail.com:S-INBOX:SendData: 15 IDLE
> 6616[9b864f0]: ReadNextLine [stream=5827e80 nb=10 needmore=0]

Line contains "10 UID fetch" in attached log file, for [Gmail]/All Mail.
> 1048[c11600]: 7e74800:imap.googlemail.com:S-[Gmail]/All Mail:CreateNewLineFromSocket: 9 OK Success
> 1048[c11600]: 7e74800:imap.googlemail.com:S-[Gmail]/All Mail:SendData: 10 UID fetch 4475:4476,4478:4491 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To ≪Personality≫ ≪Body≫)])
> 5532[9b86640]: ReadNextLine [stream=cc3fe28 nb=60 needmore=0]
Line contains "10 BAD" in attached log file, for [Gmail]/All Mail.
> 1048[c11600]: 7e74800:imap.googlemail.com:S-[Gmail]/All Mail:CreateNewLineFromSocket: 10 BAD Could not parse command
> 1048[c11600]: 7e74800:imap.googlemail.com:S-[Gmail]/All Mail:SendData: 11 IDLE
> 1048[c11600]: ReadNextLine [stream=cf9640 nb=10 needmore=0]

Following is UID fetch command issued by your Tb.
>  UID fetch a1:a2, ... ,z1:z2 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS
>      BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID
>      Priority X-Priority References Newsgroups In-Reply-To
>      Content-Type Reply-To ≪Personality≫ ≪Body≫)])

Why "≪Personality≫" and "≪Body≫" is contained in requested HEADER.FIELDS?
Is "message header named ≪Personality≫" or "message header named ≪Body≫" actually used in messages you receive and/or in messages sent by your Tb?

You intentionally defined custom header named ≪Personality≫ and ≪Body≫ at message filter definition panel?
(mailnews.customHeaders   = ...: ≪Personality≫: ≪Body≫: ...)
Or You intentionally added "≪Personality≫" and "≪Body≫" to mailnews.customDBHeaders? (mailnews.customDBHeaders = ...: ≪Personality≫: ≪Body≫: ...)
Or it's by addon?

Why log lines are merged?
If server is actually Gmail's server and Tb on Windows, AFAIK, CRLF exists at end of each IMAP log.
Local proxy server of anti-virus software is used for Gmail IMAP servre(SSL is used) access? (Avast! style)
Or You intentionally enabled "Port scna/mail scan of SSL port" of antivirus software", which is usually disabled by default? (ESET Smart Security style)

Comment 2

[Cliff Lewis]

OK, I may be in way over my head on this site. None of the above makes the slightest bit of sense to me. I apologize if I have overstepped my bounds by posting here, but the Thunderbird support site encouraged submitting a bug report if I couldn't find an answer.

I found in a forum a suggestion to generate a log file following the instructions at https://wiki.mozilla.org/MailNews:Logging. This was done by running a batch file consisting of:

set NSPR_LOG_MODULES=imap:5
set NSPR_LOG_FILE=%USERPROFILE%\Desktop\imap.log
"%ProgramFiles(x86)%\Mozilla Thunderbird\thunderbird.exe"

Why newline characters or CRLF would be missing from the log file is beyond me. I did edit the file with Notepad to hide account names, but I think the LFs were already missing when I opened the file. And I have not intentionally done anything with my antivirus software. In fact I tried to check mail with AV and firewall disabled, and still had the same result.

No filters are defined for any of my gmail accounts. I see that <<Personality>> and <<Body>> are defined as custom headers in the filter definition panel but I do not remember doing that myself. In fact I don't even know what they mean, or how to get << or >> as single characters. Maybe they were imported from Eudora? I can remove them if you think it has anything to do with the problem.

In short, I would not know how to customize Thunderbird to do any of these things. I have to assume these things were created automatically as part of Thunderbird's installation or setup or came from gmail. Sorry I can't be of more help.

Comment 3

[WADA:World Anti-bad-Duping Agency]

> I see that <<Personality>> and <<Body>> are defined as custom headers
> in the filter definition panel but I do not remember doing that myself.
Is <<Personality>> and <<Body>> used in filter rules etc.?
  - Check msgFilterRules.dat file under all mail directories by Text Editor.
    "Mail directory" is set in mail.server.server#.directory.
  - Check virtualFolders.dat file under Tb's profile directory by Text Editor.

> Maybe they were imported from Eudora?
If you imported mail related settings from Eudora, ≪Personality≫ and ≪Body≫ may be "special keyword" or "special notation of header name" in filter rule etc. of Eudora.
However, if so, I can't guess reason why your problem doesn't occur in Tb 17 but your problem occurs in Tb 24. Reason I can imagine is only that;
- Add-on who set and used ≪Personality≫ and ≪Body≫ was disabled upon upgrade to Tb 24
  due to incompatibility.
- Gmail IMAP started to reject ≪Personality≫ and ≪Body≫ just after you upgraded to Tb 24,
  because problem in Gmail IMAP was produced by such unplesant IMAP command parameter value
  which was sent from some IMAP clients. 
What did you do just before upgrade from Tb 17 to Tb 24 or just after upgrade to Tb 24?

Anyway, if ≪Personality≫ and ≪Body≫ is not your intentional setting and is not required setting, remove them.
(i)  Delete them via filter definition panel/Customize.
     Or, via Tools/Options/Advanced/General, Config Editor,
     remove them from mailnews.customHeaders, restart Tb.
(ii) If they are set in mailnews.customDBHeaders too,
     remove them from mailnews.customDBHeaders via Config Editor, restart Tb.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In addition to comment #3)
Needless to say, before Tb's setting change for recovery from problem, "keeping backup of Tb's profile directory" is almost mandatory step.
When non-ascii character is saved as UTF-8 in Tb's prefs.js, if you edit prefs.js by Notepad.exe of Win with charset=UTF-8, Notepad.exe perhaps writes BOM for UTF-8. This BOM may produce problem in Tb's prefs.js access. Please avoid prefs.js editing by Notepad.exe.

Comment 5

[WADA:World Anti-bad-Duping Agency]

FYI.
If ≪Personality≫ and ≪Body≫ is requested at "Custimize..." of Message Filter UI, Tb 24.0.1 doesn't accept it, with showing following alert.
> The header you entered contains an invalid character, such as ':',
> a non-printable character, a non-ascii character, or an eight bit ascii character.
> Please remove the invalid character and try again.
i.e. ≪Personality≫ and ≪Body≫ can not be set in mailnews.customHeaders via UI at least in Tb 24.0.1.

Comment 6

[Cliff Lewis]

The custom headers do seem to be the cause of this problem. I removed them and Tb 24.0.1 is now functioning without problems.

Comment 7

[jah]

Attachment: imap.log

Comment 8

[jah]

(In reply to jah from comment #7)
> Created attachment 824604 [details]
> imap.log

I'm having a similar "Could not parse command" problem whenever I try to sync with gmail imap.  It started happening sometime between 4 and 13 October which likely coincided with the update from 23.x to 24.0.  The most recent email I was able to "download" was on the 4th.

I followed https://wiki.mozilla.org/MailNews:Logging#Generating_a_Protocol_Log to generate the attached log. Let me know what else I can do to help.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to jah from comment #8)
> (In reply to jah from comment #7)
> > Created attachment 824604 [details]
> > imap.log

Your case is similar to original bug opener's case, but is different.
  Cause is "*"(no quote) in HEADER.FIELDS.
> 7964[123c8360]: 11c6e800:imap.gmail.com:S-INBOX:SendData:
>   14 UID fetch 1209:1217 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS
>      BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority
>      References Newsgroups In-Reply-To Content-Type Reply-To x-send-later-at
>      x-send-later-uuid x-send-later-recur X-Envelope-To
>      * x-open-relay X-Daemon-Classification Reply-To X-Facebook Received X-Sender
>      X-Original-To Return-Path)])
> 7964[123c8360]: 11c6e800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket:
>   14 BAD Could not parse command

Same question as question to bug opeenr :
  What is set in mailnews.customHeaders and mailnews.customDBHeaders?
  Who had set "*"(no quote) in mailnews.customHeaders and/or mailnews.customDBHeaders?
  When did someone put "*"(no quote) in mailnews.customHeaders and/or
  mailnews.customDBHeaders?
 
> I'm having a similar "Could not parse command" problem whenever I try to
> sync with gmail imap.  It started happening sometime between 4 and 13
> October which likely coincided with the update from 23.x to 24.0.  The most
> recent email I was able to "download" was on the 4th.

If "*"(no quote) in mailnews.customHeaders and/or mailnews.customDBHeaders was set while you were running Tb 17, the "*"(no quote) was perhaps sent in HEADER.FIELDS, and BAD should be returned from Gmail IMAP if Gmail IMAP rejects such bad parameter since initial of Gmail IMAP.
When did you upgrade from Tb 17 to Tb 23?
When did you upgrade from Tb 23 to Tb 24?

Comment 10

[Cliff Lewis]

The thing that everyone experiencing this problem seems to have in common is that they have custom headers defined for message filters. In my case I do not know when or how or why the custom headers were defined, or why they suddenly became a problem with version 24--they were present before the upgrade, and my best guess is that they were imported from Eudora when I switched to TB back in 2010. But deleting the headers solved the problem. The same thing worked for another person who followed my advice on another forum.

Try going to Tools->Message Filters... Click the "New" button. Under "Match all of the following," click the drop-down list that begins with "Subject". Click "Customize..." and see if any custom headers are defined. If any are, select them and click the "Remove" button. Exit all of the dialogs and see if you can retrieve your gmail.

Comment 11

[jah]

(In reply to WADA from comment #9)
> Your case is similar to original bug opener's case, but is different.
>   Cause is "*"(no quote) in HEADER.FIELDS.
> > 7964[123c8360]: 11c6e800:imap.gmail.com:S-INBOX:SendData:
> >   14 UID fetch 1209:1217 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS
> >      BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority
> >      References Newsgroups In-Reply-To Content-Type Reply-To x-send-later-at
> >      x-send-later-uuid x-send-later-recur X-Envelope-To
> >      * x-open-relay X-Daemon-Classification Reply-To X-Facebook Received X-Sender
> >      X-Original-To Return-Path)])
> > 7964[123c8360]: 11c6e800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket:
> >   14 BAD Could not parse command
> 
> Same question as question to bug opeenr :
>   What is set in mailnews.customHeaders and mailnews.customDBHeaders?

mailnews.customDBHeaders; x-send-later-at x-send-later-uuid x-send-later-recur
mailnews.customHeaders;X-Envelope-To: Envelope-To: *: x-open-relay: X-Daemon-Classification: Reply-To: X-Facebook: Received: X-Sender: X-Original-To: Return-Path

>   Who had set "*"(no quote) in mailnews.customHeaders and/or
> mailnews.customDBHeaders?
>   When did someone put "*"(no quote) in mailnews.customHeaders and/or
>   mailnews.customDBHeaders?

I didn't set "*", but I remember it being present when I set other custom headers for message filters. I don't know what version of Thunderbird it was when I set them, but I've been using Thunderbird since forever.

> When did you upgrade from Tb 17 to Tb 23?
> When did you upgrade from Tb 23 to Tb 24?

I use auto update on the release channel and pretty much update each time there is a new version.

Good news is that I removed just the "*" from mailnews.customHeaders and all is well again.  Here's an excerpt from the imap.log now:

8160[67e8ef0]: d3ab800:imap.gmail.com:S-INBOX:SendData: 13 UID fetch 1219 (UID X-GM-MSGID X-GM-THRID X-GM-LABELS RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To x-send-later-at x-send-later-uuid x-send-later-recur X-Envelope-To x-open-relay X-Daemon-Classification Reply-To X-Facebook Received X-Sender X-Original-To Return-Path)])

8160[67e8ef0]: ReadNextLine [stream=51a9700 nb=474 needmore=0]
8160[67e8ef0]: d3ab800:imap.gmail.com:S-INBOX:CreateNewLineFromSocket: * 82 FETCH (X-GM-THRID 1450338805047763972 X-GM-MSGID 1450338805047763972 X-GM-LABELS ("\\Important" "\\Sent") UID 1219 RFC822.SIZE 2433 MODSEQ (209796) FLAGS () BODY[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To x-send-later-at x-send-later-uuid x-send-later-recur X-Envelope-To x-open-relay X-Daemon-Classification
8160[67e8ef0]:  Reply-To X-Facebook Received X-Sender X-Original-To Return-Path)] {752}

Yay. \o/

Thanks for the assistance!

Comment 12

[WADA:World Anti-bad-Duping Agency]

This is not Gmail IMAP only phenomenon.
Yahoo! IMAP also returns BAD to "*" in HEADER.FIELDS whis is sent by Tb 24.
> 32 UID fetch 615 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject
>    Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type
>    Reply-To status * received)]) 
> 32 BAD [CLIENTBUG] UID FETCH Command arguments invalid

Conditions:
(a) Tb 24
(b) mailnews.customHeaders = status: *: received
    mailnews.customDBHeaders is not set
(c) Message filter rule which refers to "*" is not defined for any account.
IIRC, header in customHeaders was requested only when the custom header is referred by a filter rule(no need of "enabled status") in former Tb versions.
Phenomenon may be;
  Tb 24 now always requests all headers in customHeaders, regardless of (c).

Comment 14

[jimmy]

Opening the config editor and resetting the key "mailnews.customHeaders" fixed the IMAP problem for me (bug: 934179)

Comment 15

[Wayne Mery (:wsmwk)]

several reports of "could not parse command" at https://getsatisfaction.com/mozilla_messaging/tags/bug_926467

Comment 16

[rsx11m]

Bug 363238 - saved searches fail for searches on x-headers - added some code complementing the "BODY.PEEK[HEADER.FIELDS (%s)]" part with headers from mailnews.customHeaders, checking in for Thunderbird 19.0; this might be a starting point to narrow down the regression, but wouldn't quite explain the question how those bogus entries got into that preference in the first place.

http://hg.mozilla.org/comm-central/diff/5cca127781c1/mailnews/imap/src/nsImapProtocol.cpp

Comment 17

[mark]

Is this bug still open for repair? Noted that the bug still exists in 52.4.0 64bit

Comment 18

[cunio]

The bug is still present here TB 102 beta.
I see it in GMail IMAP account.

Comment 19

[Wayne Mery (:wsmwk)]

Thanks for updating the bug. We wouldn't track this old bug that does not have major functional implications.

(In reply to cunio from comment #18)

The bug is still present here TB 102 beta.
I see it in GMail IMAP account.

this only shows up in an imap protocol log?

Comment 20

[cunio]

No, the problem is that folder tree from imap.gmail.com is loaded, but when I try to see what is in any GMail folder I get "...Could not parse command..." via system notification window (Fedora 36 Linux/KDE). Deleting all "custom headers" from filers settings cures the problem.