224900 - ill-formatted messages cause message download hangs

Status: RESOLVED INVALID

(MailNews Core :: Networking: POP, defect)

Description

[Dan Kegel]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007

Ever since I switched to Mozilla 1.5 from 1.4, I've been having
trouble fetching email via POP.  This is a bit like bug 223193, but for POP.
Once the problem happens, it seems quite reproducible; it might be 
associated with downloading particular messages, as watching the network 
traffic with ethereal showed it was always happening at the same message;
restarting netscape and reconnecting yielded the same hang reliably. 
The UI is fine, and everything's working
except for the message download.  Switching to IMAP and downloading
all messages works around the immediate problem, and POP then works again
for a while.  I turned on pop3 logging, and it took
three days for the problem to recur.  The last four lines of the 57 MB log file
are:
1077635936[8101410]: Entering NET_ProcessPop3 81
1077635936[8101410]: POP3: Entering state: 19
1077635936[8101410]: RECV:
PQHZl32ZT8CijYkT6TpA83o1hbyP4Ab62DuN2h79yd0n5FHU20nmCKT3+LkbPWzK4EKdusmZodZA
1077635936[8101410]: RECV: (null)
I'll clear the log, restart netscape, and post the full (and hopefully
short) log at http://kegel.com/mozpophang.txt.gz

Reproducible: Sometimes

Steps to Reproduce:
1. Fetch mail via POP.

Actual Results:  
Message downloading gets stuck at a particular message.

Expected Results:  
Downloaded all messages.

Comment 1

[Dan Kegel]

OK, I posted the log.  It's short and sweet and repeatable.

And for what it's worth, here's what netstat -tp shows during the hang:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
PID/Program name   
tcp        0      0 dual:33999              secretcow.com:pop3      ESTABLISHED
2670/mozilla-bin 

In other words, the pop3 connection is still open, with no unread bytes.

Comment 2

[Dan Kegel]

Aha.  IMAP gets stuck on the same message.
It got unstuck after I viewed my inbox with my ISP's webmail interface
(which creates a new "internal placeholder" message for some reason, btw).
Next time I'll see if I can get an IMAP log, too.

Comment 3

[Dan Kegel]

OK, it's happened two more times; nice short logs are at
http://kegel.com/mozpophang2.txt.gz and
http://kegel.com/mozpophang3.txt.gz
In all three cases, it's a particular kind of malformed message that
hangs pop, it seems.
IMAP does hang if I try to move the message to a local folder,
but it's ok if I just try to delete the message.  After that,
I seem to be able to restart mozilla and get pop to resume downloading.

I do have one log of imap getting hung; it's quite large (though I did
try to get a minimal case), and is at
http://kegel.com/mozimaphang.txt.bz2

The malformed messages appear to be sent by the Sober virus;
see http://www.f-secure.com/v-descs/sober.shtml

Since I'm on Linux, I'm nominally immune to these viruses, but it's
a bit annoying that download hangs.

I suppose it could be a server-side problem, too...

Comment 4

[Dan Kegel]

Definitely not a server side problem.  A good workaround when Pop gets stuck
in moz1.5 is to quit, start moz1.2.1, and download all messages; then quit
and start moz1.5 again.

Comment 5

[Christian Eyrich]

Dan, as you're on Linux, you're able to test a recent 1.6 trunk nightly. Your
debugging work looks like the fix I did for bug 223062 should help you too.
Please let me know if this helps.

Comment 6

[Dan Kegel]

OK, it finally happened again!  Is 1.6 alpha recent enough,
or do I need a different build to get your fix?  Please
paste the URL into a comment here, as I won't be able to
receive email until I fix this.

Comment 7

[Christian Eyrich]

No, 1.6a is recent enough as the patch was to risky for it.
I recommend using a nightly from
http://ftp.mozilla.org/pub/mozilla.org/mozilla/nightly/latest/ like
mozilla-i686-pc-linux-gnu-sea.tar.gz.

Comment 8

[Dan Kegel]

The Nov 10th build seems to work for me.  Thanks; I'll reopen this
if I notice the problem recur.

Comment 9

[Christian Eyrich]

Oh, I'm glad to read this. Let's hope it's really dead and gone.

Comment 10

[Dan Kegel]

It's baaaack!  This time I can reproduce it on Windows XP with
Mozilla 1.6beta, so I'm setting OS to 'all'.  (Or should I open
a new bug?)  I'll attach packet logs.  Perhaps this is
a slightly different but related problem.  

See also bug 187828 (not sure why I didn't see that one before).

Comment 11

[Dan Kegel]

requesting blocking1.6

Comment 12

[Dan Kegel]

Attachment: Log file (both ascii and tcpdump) of win xp POP session that gets stuck with mozilla 1.6b

This showed up on my wife's XP box in Mozilla-1.5 today; upgrading
to Mozilla-1.6b didn't fix it (unlike on my Linux box, where
1.6b fixed the hang).  Maybe this stimulus is different, who knows.
I guess I could try retrieving this message under Linux,
or reverting to mozilla-1.4.

Comment 13

[Dan Kegel]

Attachment: Mozilla POP3 log of problem happening

this file was generated by setting NSPR_LOG_MODULES=POP3:5

Comment 14

[Christian Eyrich]

It's bad you see the problem again. And from your tcpdump it looks very like the
same cause (hundreds of 0x0 in the mail data).

But I've no idea why using 1.6b doesn't help. I did a perl script that simulates
a server that sends exactly the same data as in the dump and have no problems
(tested on Win and Linux).

It would be nice if you could try retrieving such a faulty mail with your Linux
system that Moz/Win doesn't get. But I've no idea what to do - if it works or not.

Comment 15

[chris hofmann]

this would be a good one to pick up for 1.6 if a low risk fix appears.  1.6- for
now, but renominate if we get it figured out and come up with a patch.

Comment 16

[Elizabeth Fuller]

I've since found a couple different hang conditions.
Some hangs appear to be caused by antivirus software; turning that off helps.
Others aren't.  See also bug 220225

Comment 17

[Dan Kegel]

I think I got bit by both this bug and bug 234315 at the same time --
at moment, I can't download via either pop or imap on
Windows XP with mozilla 1.6!  The logs are largish, so I've
saved them at http://www.kegel.com/bug224900-logs.zip

With pop, the bad message appears to be 
Message-ID:.<[e-mail]\r>
From: "Lesa Madison" <[e-mail]\r>
Reply-To: "Lesa Madison" <[e-mail]\r>
To: [e-mail]
Subject: ciaglis at 3$ per dose
...

where the \r's are actual 0x0d's in the message (a bit odd),
and the body of the message has a couple nulls in it
right after the =='s that probably terminate a base64 message part:

...
23:09:45.177491 IP zilf.org.110 > zilf.1178: P 9131:9213(82) ack 402 win 65535 
(DF)
0x0000   4500 007a ab9b 4000 3006 5f85 d85c a5f4        E..z..@.0._..\..
0x0010   c0a8 0064 006e 049a 7e1c 2323 7e87 0ec4        ...d.n..~.##~...
0x0020   5018 ffff b842 0000 7559 3239 744c 334e        P....B..uY29tL3N
0x0030   320d 0a4c 326c 755a 4756 344c 6e42 6f63        2..L2luZGV4LnBoc
0x0040   4439 7761 5751 395a 5842 6f4d 6a59 324d        D9waWQ9ZXBoMjY2M
0x0050   413d 3d00 000d 0a0d 0a2d 2d2d 2d30 3536        A==......----056
0x0060   3530 3330 3131 3435 3735 3137 3436 3734        5030114575174674
0x0070   352d 2d0d 0a0d 0a2e 0d0a                       5--.......

Comment 18

[Christian Eyrich]

Dan, you still see this only on Windows, not on Linux?

Comment 19

[Dan Kegel]

It's my wife who has the problem, and she uses XP.
Maybe she gets different spam than I do;
I have not recieved the "ciaglis 3$" spam on Linux, so
I don't know what it'd do.

Comment 20

[Elizabeth Fuller]

Happened again tonight.  Once again, there's a null byte
after the = padding on the end of a base64 message:

0x02f0   7661 4852 7462 4434 3d00 480d 0a0d 0a2d        vaHRtbD4=.H....-
0x0300   2d2d 2d35 3233 3338 3739 3439 3230 3630        ---5233879492060
0x0310   3637 3838 3737 2d2d 0d0a 0d0a 2e0d 0a          678877--.......

I'll attach the log.  Note that SpamAssassin has
quite a bit to say about this message.  I could
probably have my server discard spam instead of
sending it to us, and work around this problem
(though of course it'd be better if mozilla didn't hang!)

Comment 21

[Elizabeth Fuller]

Attachment: A windump log of an ill-formed message hanging mozilla 1.6 on windows xp

Comment 22

[Elizabeth Fuller]

By the way, we did confirm both yesterday and today
that deleting the one message containing the NULL bytes
via webmail, and restarting mozilla, cleared the hang,
so it's pretty certain the message in each log is the
one that caused the hang.

Comment 23

[Christian Eyrich]

Thanks for all the logs so far, I don't think we need more. I recognize this
null bytes as the reason for the hangs and believe you, your Mozilla is
suffering from it.

My problem is, the Mozillas I tested (different on Linux and also Win98) aren't
bothered by null bytes in any mail. There already is some anti-null-byte loop in
the ReadNextLine() function and I can't see a bug in it.
Trying to compile a verbose logging Windows build for you, failed yesterday
because of some mysterios compile error.
So I'm sorry for the moment.

Comment 24

[Elizabeth Fuller]

Attachment: Log of problem with both windump and extra-special verbose mozilla

This log was made using
http://www.eyrich-net.org/tests/mozilla-i686-pc-cygwin_dbg.zip
which has special debugging log statements.

Comment 25

[Christian Eyrich]

In comment #16 you say a antivirus is running on your computer. Is this still
the case?
Although you wrote turning it off doesn't help, is it really off? Not only in
some disabled mode but shut down, removed from the tasklist and everything?

Comment 26

[Elizabeth Fuller]

It's disabled, but the control applet is still in the taskbar
(showing disabled status).  In the past, that was enough to
prevent trouble.  Next time I have the hang I'll try reproducing
with the virus checking applet exited and no longer in the taskbar,
is that enough?

Comment 27

[Christian Eyrich]

> is that enough?

I hope it.
It's not that I'm saying "it's the anti-virus". But the things I found out
through the new log pointing this way.

Comment 28

[Elizabeth Fuller]

That was it.  Annoyingly, McAfee VirusScan 8.0 *doesn't* turn
off several of its modules when you turn everything off via
the UI; you have to go and kill the programs via the task manager.
The offending program is McVSCSCN.exe; kill it, and everything's 
fine.  Supposedly, the problem happens with any client, even telnet,
connecting via pop3.  See
http://forums.mcafeehelp.com/viewtopic.php?t=17380&start=40
http://forums.mcafeehelp.com/viewtopic.php?t=21154
Thus this is NOTABUG for mozilla.

Comment 29

[Christian Eyrich]

Uff, I'm eased. 
In your case it seems McAfee VS is proxying the communication and simply stopps
passing data to Mozilla when it encounters the \0 ...

So that's No.3 - I had two other different communication problems with different
AV in the last days. I swear I'll ask for an installed AV on the next
communication bug first.

So can we close this again as RESOLVED/FIXED?

Comment 30

[Dan Kegel]

I'll close it myself... dunno why I couldn't last night.

Comment 31

[George]

*** Bug 234315 has been marked as a duplicate of this bug. ***