205789 - Bad HTML SPAM mostlly from comcast.com hangs IMAP communication

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[Andrew Brady]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3.1) Gecko/20030425
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3.1) Gecko/20030425

Certain SPAM emails, mostly coming from comcast.com, are hanging the IMAP
reading process. I have a few examples of these emails which will be
attached. The emails are HTML content-type with non-html tags.

I have tcpflow'd the IMAP and once it has hung no communication with
the IMAP servers occurs.

Also turned on the IMAP protocol debugging in Mozilla and after the first
attempt to read one of the offending emails, the only debug I get,
repeatedly, when I click on a new message in the same folder, is:

6151[8bb5198]: 886a860:host.ecmwf.int:S-bad_spam:ProcessCurrentURL: entering
6151[8bb5198]: 886a860:host.ecmwf.int:S-bad_spam:ProcessCurrentURL: entering
6151[8bb5198]: 886a860:host.ecmwf.int:S-bad_spam:ProcessCurrentURL: entering
6151[8bb5198]: 886a860:host.ecmwf.int:S-bad_spam:ProcessCurrentURL: entering
6151[8bb5198]: 886a860:host.ecmwf.int:S-bad_spam:ProcessCurrentURL: entering


Reproducible: Always

Steps to Reproduce:
1. Click on spam email
2. Email views ok
3. Click on next email in same folder
4. Nothing
5. Click on another email in same folder
6. Nothing
7. Click on a differnt folder
8. Ok again

Comment 1

[Andrew Brady]

Attachment: A unix mbox mail file containing two emails that give moz IMAP indigestion

Comment 2

[David :Bienvenu]

OK, I tried this as best I could, and didn't have the problem you describe.
However, the mbox file contains some embedded nulls, which are not allowed by
the IMAP protocol (i.e., an IMAP server MUST not send embedded nulls to a
client) and I can't convince our servers to accept messages with embedded nulls,
or give them back to me. Embedded nulls cause us to drop the connection with the
server, I believe, because it's a fatal error as far as we're concerned.
However, we should re-establish the connection when you click on the next
message. Can you try a newer 1.4b/final build? I'll try to see if I can get a
server to serve up embedded nulls (what IMAP server are you using? and can you
attach a more complete log, or e-mail it to me? thx)

Comment 3

[David :Bienvenu]

it would also be helpful to get the full imap protocol log 

Comment 4

[Andrew Brady]

Thanks for the pointers. I was using UW imapd IMAP4rev1 2000.287
and also having same problem with UW imapd IMAP4rev1 2001.315. I tried
the latest Mozilla release and found the same effect.

I stripped out the nulls in the folder and this cleared the problem.

I upgraded the imapd to the latest IMAP4rev1 2003.337 and the problem
is also solved, even when the nulls exist in the folder. Checking the
imap trace with this new version shows that the null characters have
been eliminated.

Comment 5

[David :Bienvenu]

Attachment: proposed fix

if we get a syntax error, close the connection (this patch also removes some
unused code and turns logging back when m_runningUrl is null so we can log
logouts). This has r/sr=sspitzer over aim. 

Ideally, we would not throw away the connection in this situation, but that's
the way the parser is designed to work. I will investigate making it recover
more gracefully.

Comment 6

[David :Bienvenu]

this fix was checked in. I should say that I believe the problem is NOT that the
imap server is sending the embedded nulls, but that it is stripping them out but
not adjusting the message size, so it confuses our parser.

Comment 7

[Blu3]

FYI, there are 'embedded NULLs' bug entries for a couple years now.  Supposedly
it was fixed.  I was one of the first reporters since my printer has an awful
habit of postfixing a NULL at the end of all it's status reports.  For quite
some time I was getting a spam, almost daily, with an embedded NULL.

I understand that it's "not in the spec" for IMAP, but ignoring the problem and
all the broken software out there isn't really the best answer IMO.  A simple
stripping of the NULL (as below) is fine.  (Or s/NULL/space/)

I'll fetch a nightly and test it w/ my printer's reports to make sure things are
spiffy.

Comment 8

[David :Bienvenu]

I just discovered that our line buffering code is stripping out nulls, which
will throw off the size of the message, and generate a syntax error. We should
be just converting nulls to spaces, IMO. I'm going to try that change as well.

Comment 9

[David :Bienvenu]

Attachment: proposed fix

OK, the general line reading code was just removing nulls, instead of replacing
them with spaces, which throws off the byte count for imap and pop. This patch
makes it so we just replace nulls with spaces.

Comment 10

[David :Bienvenu]

fix checked in, r/sr=sspitzer.

Comment 11

[David :Bienvenu]

tomorrow's build should be much better behaved with your printer reports.

Comment 12

[(not reading, please use [e-mail] instead)]

david, is this needed on the 1.4 branch, for 1.4.1?

Comment 13

[David :Bienvenu]

yes, I think so. the last patch.