153251 - error when getting new msgs

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[sairuh (rarely reading bugmail)]

sometime using today's 2002.06.20.06 branch comm bits (on linux rh7.2), i
started getting the following error when i tried to get new msgs or switch to a
folder which had unread msgs:

The current command did not succeed. The mail server responded: Invalid
field-name in UID Fetch BODY.PEEK[HEADER.FIELDS.

i'm not sure what would've caused this to occur, but it seems i can reproduce
this with a new profile... Karen and i are still trying to narrow this down to
the smallest number of steps, so pls pardon the complication of the recipe.
(it'll likely be modified in future comments.)

1. moved ~/.mozilla to another name, and make sure there aren't other
netscape/mozilla instances running (just to start out clean).

2. create a new profile (via wizard), launch it.

3. went into to Prefs and turned off the password mgr.

4. started mail and created an imap account.

5. logging onto mail was fine. i also closed my sidebar (out of habit).

6. quit entire app.

7. i save copies of my previous prefs.js and rules.dat (for convenience), so now
i do the following:

  a. copy old rules.dat into the new profile's ImapMail/<server>/ dir

  b. add the following line to prefs.js so that my custom header will be recognized:

      user_pref("mailnews.customHeaders", "Received");

8. restart the app: browser comes up (i close the sidebar out of habit), then
restart and login to mail. so far okay...

9. in the thread pane (still viewing Inbox), i get rid of several columns, so
that i only have thread, subject, sender and date.

10. click on a folder (eg, "bugmail") containing unread msgs.

results: get the error dlg mentioned above.

after i quit the entire app, and look at my prefs.js, i noticed that the line i
had added for the custom header has somehow gotten corrupted:

      user_pref("mailnews.customHeaders", "Received: ä&");

i'll attach the troublesome prefs.js, along with the rules.dat...

Comment 1

[sairuh (rarely reading bugmail)]

refinement of steps: i didn't need to change the thread pane columns at all --i
had mentioned that step since i didn't remember the exact order of every task i
did (or whether they were needed). anyhow, so far the recipe above minus steps 9
and 10 --when i login to mail for the second time, i get the same error. (nor am
able to view folders with unread msgs, or get msgs, still.)

i have an imap log which was created during these steps (again, minus steps 9
and 10), which i'll attach soon.

Comment 2

[sairuh (rarely reading bugmail)]

more observations: okay, the last time i tried this, i didn't get the error when
i logged on (ie, no new msgs for the Inbox folder), but i did get it once i
clicked on a folder with unread/new msgs (eg, "bugmail" which does get filtered
mail). however, step 9 is unnecessary --it's the process of getting new msgs
that seems to matter, i think (so step 10 would be optional if the Inbox had
gotten anything new since the last login).

okay, really attaching those files!

Comment 3

[sairuh (rarely reading bugmail)]

Attachment: IMAP log

Comment 4

[sairuh (rarely reading bugmail)]

Attachment: corrupted prefs.js

Comment 5

[sairuh (rarely reading bugmail)]

Attachment: rules.dat file

Comment 6

[Karen Huang]

From above IMAP log, the following lines does not display correctly:
----------------------------------------------------------------------------------
7176[8d9d868]: nsmail-2.mcom.com:S-bugarama:SendData: 7 UID fetch 1:4,6,8:9 (UID
RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Subject Date Message-ID
Priority X-Priority References Newsgroups ä& Received Received Received Received
Received Received Received Received)])
7176[8d9d868]: nsmail-2.mcom.com:S-bugarama:CreateNewLineFromSocket: 7 BAD
Invalid field-name in UID Fetch BODY.PEEK[HEADER.FIELDS
----------------------------------------------------------------------------------

Ccing Cavin -- Cavin, this is the bug that I mentioned to you today....
Don't know why Sarah got this error from the Server after above changes.
This can be reproduced by Sarah and she mentioned that she didn't encounter this
problem before.....

Comment 7

[Navin Gupta]

If you look at this filter it is asking for this header ä&. Do you remember 
how you got here? What was this intended to be originally?

name="prefs list"
enabled="yes"
type="1"
action="Move to folder"
actionValue="imap://[e-mail]/news-stuff"
condition="OR (ä&,contains,[e-mail]) OR (to or
cc,contains,[e-mail])"
name="wanted news 1"
enabled="yes"
type="1"
action="Move to folder"
actionValue="imap://[e-mail]/news-stuff"
condition="AND (\"Received\",contains,gila.mozilla.org) AND (subject,contains,menu)"

Comment 8

[laurel]

When you input the custom header, how did you enter it -- did you put a colon
":" or space or something other than the exact header into the ui?  Gary
reported similar problem and he had input the header with a colon -- see
commments in bug 155496.

Comment 9

[Karen Huang]

When I worked with Sarah. She didn't put the colon for custom header.
But, error will occur after she saved copies of her previous prefs.js and 
rules.dat files.
For some reason, user_pref("mailnews.customHeaders", "Received") has been 
overwritten with user_pref("mailnews.customHeaders", "Received: ä&") 
automatically.....

Comment 10

[David :Bienvenu]

I think Navin worked on a bug that might be related...I don't remember the details.

Comment 11

[Navin Gupta]

How can it happen automatically ? Can you reproduce it every time if you start
with no garbage ?

Comment 12

[Navin Gupta]

There was a related bug 141354 which got fixed on branch recently 6/19. Could it
be the case that you added a new filter using a branch build older than that and
didn't notice the problem until today. One more question could you have started
with corrupted rules.dat always (unknowningly) ? 

Comment 13

[David :Bienvenu]

I would think it was just bug 141354, except that Gary very recently reproduced
this, I believe. Gary, is it possible you ran a branch build that didn't have
this fix? I was not able to reproduce this problem myself with a trunk build.

Comment 14

[grylchan]

Ok this is the info I have.

I filed bug 155496 based on trunk builds. I had a
incorrect custom header as I added the ':' to first
drop down filter item which Laurel mentioned I should
not (that's bug 107620). So when I added colon
and tried to filter my mesg, when I did a 'get mesg',
I would get Body.Peek error mesg.

I tried again on linux trunk build 2002070808 but
with out ':' and it worked and I didn't see the error mesg.

I tried custom filter on 20020708 commercial branch on NT 4.0
but with my incorrect custom header (with the ':') and I
was able to produce the same error mesg. If take out the colon,
every thing works fine.

I hope this helps

Comment 15

[David :Bienvenu]

can you say exactly what header you added? If I try to add one with a ":" in it,
it just fails to add and doesn't show up in the dropdown.

Comment 16

[grylchan]

Here are the headers I added:
 -'X-Accept-Language:' contains 'en-us, en' 
    move to folder sent on local folders. 
  -'User-Agent:' contains 'Mozilla/5.0'
    move to folder sent on local folders. 
  -'Received:' contains 'linzilla'
    move to folder 'x' on local folders

Comment 17

[David :Bienvenu]

OK, the filter corruption bug has nothing to do with this bug. The problem is
that the custom headers UI can't allow ":" in the custom headers. In fact, the
custom headers UI needs to do a lot more validation of the header specified by
the user, because if the user does enter an invalid header, the imap server will
likely complain when we try to fetch it. The custom header ui should ensure that
the header is a valid rfc822 msg header  (there's probably some function around
to do that). The imap code might also need to throw out invalid custom headers too.

Comment 18

[James Pattie]

I just ran across this yesterday in Mozilla 1.1a when I switched from WU imap to
Cyrus Imap and had a filter created with a custom header of 'Reply-To:' that I
had created back around 0.9.2 or maybe earlier.  When I select from the
available actions (Sender, To, etc.) I have about 40 instances of Reply-To:
listed which doesn't make sense.  If I remove the rule that was referencing
Reply-To: as the header, then I no longer get the error.  I haven't checked to
see if there is any corruption in prefs.js, but eyeballing my rules.dat file I
couldn't see anything wrong with it.

Comment 19

[Navin Gupta]

How does "mailnews.customHeaders" pref look like ?

Comment 20

[Navin Gupta]

How does "mailnews.customHeaders" pref look like in prefs.js ?

Comment 21

[David :Bienvenu]

the problem is probably that you used "Reply-To:" and not "Reply-To" - the ':'
is what's causing the problem. We should not be allowing the ':' in the custom
headers UI, for starters.

Comment 22

[Navin Gupta]

Attachment: proposed fix

I have added a check if there is ':' in header name then we throw an alert and
bail out.

Comment 23

[Navin Gupta]

cc robinf for alert wording review. Cavin, David Can you review the fix ? thx
 

Comment 24

[Navin Gupta]

I need to put the alert string in the dtd file. 

Comment 25

[Cavin Song]

Comment on attachment 94071 [details] [diff] [review]
proposed fix

r=cavin.

Comment 26

[David :Bienvenu]

Comment on attachment 94071 [details] [diff] [review]
proposed fix

I'll wait for the final fix to sr - that error message is not going to fly -
you at leat need to tell the user how to fix the problem, and as you say, it
needs to be localizable.

Comment 27

[robinf]

Suggested wording for error message: "The colon character (:) is not allowed in
a custom header name. Remove the colon and try again."

Comment 28

[Navin Gupta]

Attachment: patch

made it localizable. Alert will now ask them to remove colon and try again.

Comment 29

[Navin Gupta]

I have changed the alert text as per robinf. 

Comment 30

[David :Bienvenu]

this fix is OK - however, it only fixes the most common cause of this problem.
As I've mentioned before, any invalid rfc822 header entered in this dialog will
cause this problem. Do you want to spin up a new bug for the general problem, or
address the general problem in this bug? The general fix will require a
completely different error message, and different code - up to you.

Comment 31

[Navin Gupta]

I think it is ok for now. I'll log a bug about rfc822 header. BTW, what does it
allow, alphanumeric ?
 

Comment 32

[David :Bienvenu]

you'd have to look at the rfc. Let me go find it for you. (it's rfc # 822 :-) )

Comment 33

[Navin Gupta]

ok, I found out what rfc says.. 

2.2. Header Fields

   Header fields are lines composed of a field name, followed by a colon
   (":"), followed by a field body, and terminated by CRLF.  A field
   name MUST be composed of printable US-ASCII characters (i.e.,
   characters that have values between 33 and 126, inclusive), except
   colon. 

I'll attach a new patch. good that my patch already covers colon part. 

Comment 34

[Navin Gupta]

Attachment: patch following rfc2822

I will make changes to final alert as per robinf review before checking in.

Comment 35

[Navin Gupta]

will remove the dump too.

Comment 36

[David :Bienvenu]

I would suggest something like "the header you have entered is not a valid
character. It contains a ':' or other invalid character(s). Please enter a valid
header"

Comment 37

[robinf]

Besides the colon, what are other invalid characters (ones that fall outside the
33-126 range)? I'd like the error message to tell users about the invalid
characters. 

"The header you entered contains an invalid character, such as ':',...(include
list of other invalid characters). Please remove the invalid character and try
again."

Comment 38

[David :Bienvenu]

all the non-printable characters (which we can't display easily in error
message), any non-ascii character (e.g., a Japanese character). Also, all the 8
bit ascii characters like è. But, as I said before, ':' is the most likely
character.

Comment 39

[robinf]

Suggested error message text: "The header you entered contains an invalid
character, such as ':', a non-printable character, a non-ascii character, or an
eight bit ascii character. Please remove the invalid character and try
again."

Comment 40

[David :Bienvenu]

that seems fine - I would suggest "character(s)" but it's not important.

Comment 41

[David :Bienvenu]

Comment on attachment 94111 [details] [diff] [review]
patch following rfc2822

sr=bienvenu, if you use robin's new proposed wording, and change the following
comment:
 //if we don't have rfc2822 header field name bail out
to say
// if user entered an invalid rfc822 header field name, bail out.

it's just slightly clearer

Comment 42

[Navin Gupta]

fixed

Comment 43

[Karen Huang]

*** Bug 162184 has been marked as a duplicate of this bug. ***

Comment 44

[Navin Gupta]

*** Bug 107620 has been marked as a duplicate of this bug. ***

Comment 45

[laurel]

OK using oct14 commercial trunk builds: win98, mac OS 10.1, linux rh6.2
UI now has said error handling for adding an invalid character in custom header.

Comment 46

[laurel]

*** Bug 177808 has been marked as a duplicate of this bug. ***