196584 - a dot is visible at the end of each (mail)message for servers that don't dot-stuff

Status: RESOLVED FIXED

(MailNews Core :: Networking: POP, defect)

Description

[Ruurd Koons]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030304
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030304

Each message that is received shows the protocol defined dot to end the message.
This causes mozilla not to catch up signatures so they are not removed when
replying messages in HTML-format.
The 'dot on a single line' should not counted in as the message. As far as I
know this bug has been around since 1.3b, versions before acted correct.

Reproducible: Always

Steps to Reproduce:
1. compose a message to yourself
2. send it
3. receive it and see the terminating dot

Actual Results:  
got a message ending with a dot on a single line, which I didn't wrote during
composition, it is also not escaped (..) in the source of the message, so it's a
real terminating dot.

Expected Results:  
strip the terminating dot from the message, so for it isn't visible to the user
and doesn't interfere with the signature in HTML-format.

On windwoes I didn't found the problem with 1.3b, so it seems MacOSX only to me.

Comment 1

[Tom Sommer]

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030530

I can confirm this in 20030530

Using POP3, and Windows 2000

Comment 2

[Ruurd Koons]

I did some additional research and found out on Windows using 1.4rc1 the same
happens but ONLY for my account at my ISP. For other family members using
Mozilla as well it is the same, all mails at my ISP's adressess end with a dot.
I have several other e-mail accounts, including the one of my very own Linux
server running Sendmail and it doesn't have that dot. My other accounts are all
run by Sendmail so I guess my ISP (HCCnet in the Netherlands) does use some
**** mailserver or they have some misconfiguration causing the dot to appear.

Mozilla has absolutely no problems and perfect normal behaviour on all other
accounts, so I think I can be sure that the problem is limited to the way my ISP
configured their mailserver.
M$ Outloek doesn't have the dot from the same account at my ISP.

Comment 3

[Tom Sommer]

Try setting user_pref("mail.server.default.dot_fix", false); in your user.js
(this will fix only newly received mails)

It seems to have solved the problem for me...

Why is this not disabled by default?

Comment #2: What mail software does your host use?
Mine uses: X1 NT-POP3 Server example.com (IMail 6.06 116683-2)

Comment 4

[Ruurd Koons]

Sorry, I didn't meant Sendmail, of course I meant the POP3 program which is in
case of other accounts the UW POP3 deamon.
The deamon of my ISP talks like this:
+OK POP3 server at pop.hccnet.nl starting.
quit
+OK POP3 server signing off.

maybe that someone can identify it this way... I know a UW talks different, it
ends with Sayonara.

Sorry again for the double post

Comment 5

[Ruurd Koons]

The pref setting does the trick!

Dot is gone in my few tests...
Don't understand why it defaults to true while it _causes_ a dot.

Comment 6

[Tom Sommer]

I don't understand it either... I'm looking into it

Comment 7

[bugger]

(In reply to comment #6)
> I don't understand it either... I'm looking into it

Any word on this?

It is now the year 2004 and I also have this problem in Thunderbird 0.7 and
Mozilla 1.7rc3 where the trailing dot appears on every single incoming message
from the pop3 server and it is fixed by that setting, but why oh why is that
setting not the default?  No other client (ex: kmail, eudora, etc...) appends
this dot at the end but Mozilla Mail and Thunderbird.

Comment 8

[Rory McGuire]

As per the description, I believe this problem is an incompatibility with the
POP3 spec.  Specifically, RFC1939 states that the 'RETR' command gets a
multi-line response (ostensibly because the actual message will contain CRLF
sequences).  Multi-line responses are terminated with '.CRLF' (0x2d 0x0d 0x0a),
and that sequnce is NOT considered part of the response.  It should therefore
not be included as part of the message.

Mozilla Thunderbird 0.7.2 (Windows/20040707) on WinXP Pro 2002 sp1

Comment 9

[Rory McGuire]

(In reply to comment #8)

Typo in previous comments:
'.CRLF' (0x2d 0x0d 0x0a)
shoud read:
'.CRLF' (0x2e 0x0d 0x0a)

Comment 10

[drewamato]

I also had this problem, and was pointed to this bug. However, now I'm not so
sure it is a "bug", as such.

I was having the "extra dot" problem with our in-house, low-quality,
roll-your-own POP3 server. TB was adding the "end-of-multiline-response" dots to
messages received via POP3.

When I looked at the logs, though, it seemed that the POP3 server was reporting
an inaccurate message length, in the "LIST <n>" responses; the octet-lengths
reported by the LIST command were larger than they should have been (due to an
encoding "problem".) 

I changed the POP3 server to not return the message length in response to LIST
(which breaks conformance to the protocol, I know) and TB suddenly worked
exactly as expected; the messages no longer had the extra dot. 

I'm suspecting that TB has some sort of way of trying to hedge its bets on
guessing when a message ends; and if you send "CRLF.CRLF" before the end of the
message (as given by the message length response to LIST) it includes it as part
of the message. (That's why I wouldn't call it a bug; what else could you expect
it to do in the circumstances?) 

So, I'm guessing that in some cases at least, this is actually a broken POP3
implementation, and not TB's fault. (e.g., I've found the exact same problem
with Hotmail -> POP gateways. Hotmail will give you a supposed length for each
message (the <D:getcontentlength> prop), which is (I don't know why) longer than
the actual message is. Some Hotmail->POP gateways report this length in response
to LIST, and hence will give you the "dot at the end" problem in TB.)

Comment 11

[Marco Cimmino]

Ok I've seen comment #10 and I've discovered that infact in my accounts "dot"
appears only when I use FreePOPS (http://www.freepops.org/en/) and not when I
use another server.

This because this kind of programs that receive mails via HTTP and not POP3 they
cannot know exact dimension of messages, because they take the dimension from
the host web pages that is express in KB or even in MB and not in bytes.
So using this kind of programs that take mails from web and not from POP3 server
(in certain situation you cannot use POP3, because you have closed port or
something like that) you will never have the exact dimension of the message at
the point of the LIST request from client.

SO, the question is:
Why Thunderbird doesn't calculate the dimension of the message by own when
receive the mail and without taking care of dimension contained in the LIST
response?

Comment 12

[Mike Robinson]

(In reply to comment #3)
> Try setting user_pref("mail.server.default.dot_fix", false); in your user.js
> (this will fix only newly received mails)

I did a file search on my computer and I don't have a user.js file anywhere. How 
can I acquire one and where should I place it?

Comment 13

[Mike Cowperthwaite]

*** Bug 282401 has been marked as a duplicate of this bug. ***

Comment 14

[David Balažic]

RFC 1939 is cristally clear :
When all lines of the response have been sent, a
final line is sent, consisting of a termination octet (decimal code
046, ".") and a CRLF pair.


If the received data is longer or shorter than expected, then that should be
dealt with AFTER it is received.

If the message is "too short", why stuff it with the dot ? Does that make any
sense ? Of course not. If any dot were part of the message, it would be
byte-stuffed anyway. So this is doubly broken.

Also: the standard does not seem to assign any meaning to the size listed in
LIST command. It is there only to help the user to decide, whether he downloads
a certain message or not.

Comment 15

[John Serris]

I'm using tb1.0.7 and seeing this dot too. Setting
user_pref("mail.server.default.dot_fix", false) fixes the issue. My host is ugin
Cpanels pop (cppop).

So is this a bug in cpop or TB?

Comment 16

[John Serris]

I also found this:
https://bugzilla.mozilla.org/show_bug.cgi?id=97912 which is marked as
WORKSFORME. It's still not clear if it's a TB bug. Something to note though, is
that every other email client I've tried doesn't show the extra carriage returns
and dot.

Comment 17

[Dave Latham]

I also witnessed this issue with a POP3 server giving imprecise size responses for the LIST command.  It happens when the actual message size is shorter than the LIST response.

Going by the POP3 spec in RFC 1939, this is a issue that manifests because of bugs in both the POP3 server and Thunderbird.  The POP3 server gives an incorrect response to LIST.  Thunderbird incorrectly interprets the response to RETR by considering the final line part of the message, while the rfc states clearly, "the line containing ".CRLF" is not considered part of the multi-line response".

The problem would not appear using either a POP3 email server that is fully compliant with the RFC or if Thunderbird was fully compliant with the RFC.

Comment 18

[Martin]

I agree with everything in Dave Latham's comment 17. I'm seeing "." appended to the message body under 2.0.0.6 collecting from an MTA that overstates the octet count. The non-default 'mail.server.default.dot_fix=false' does fix the problem. I'd expect an MUA to ignore the LIST response and wait for a line that reads ".". If logs or a test-account on the pop3 server are useful, let me know. 

Comment 19

[jwq]

 I see this problem with some (not all) messages retrieved from a Microsoft Exchange POP3 server version 5.5.2657.74. It appears to be restricted to some (not all) messages which I receive from users internal to the same server and does not affect messages received from users external to the server.

Comment 22

[Alfred Peters [:infofrommozilla]]

I can confirm that this bug is caused by the combination of an incorrect POP3 LIST response and the Pref: mail.server.default.dot_fix=TRUE

In the program code, mail.server.default.dot_fix=TRUE ensures that TB ignores the end point.

I think this fix was introduced to enable the reception of POP3 servers that did not do dot-doubling in the email body.

Since such servers are rarely used today, I think it makes sense to change the default setting of that pref to FALSE.

What do you think?

Comment 23

[Wayne Mery (:wsmwk)]

Good question. I don't have expertise in this area, so defer to others.

Comment 24

[gene smith]

Attachment: tb.log.moz_log

Setting dot_fix false definitely fixes the problem for me for emails sent to outlook365 in pop3 mode. With it set at the default true I see a blank line followed by a dot at the end of each text email.
With the same message sent to another server (my isp's server) true or false doesn't matter, don't see the blank line and dot. So seems OK to default it to false.
The problem at the server must be caused by the length given to tb in the in the RETR response since both servers (outlook and isp) return the same final few strings per POP3:5 log. In the log, isp returns a length but outlook doesn't return a length at all.
For the record, I'm attaching the log. Shows first outlook fetch and then isp's.

Comment 25

[Alfred Peters [:infofrommozilla]]

(In reply to gene smith from comment #24)

The problem at the server must be caused by the length given to tb in the in the RETR response since both servers (outlook and isp) return the same final few strings per POP3:5 log. In the log, isp returns a length but outlook doesn't return a length at all.

No.
https://searchfox.org/comm-central/source/mailnews/local/src/nsPop3Protocol.cpp#3074

        int32_t nsPop3Protocol::RetrResponse(nsIInputStream* inputStream,

[...}
https://searchfox.org/comm-central/source/mailnews/local/src/nsPop3Protocol.cpp#3106

        m_pop3ConData->cur_msg_size =
            m_pop3ConData->msg_info[m_pop3ConData->last_accessed_msg].size;

If there is no <size> in the RETR response, we use msg_info[].size
And this is set in: nsPop3Protocol::GetList():
https://searchfox.org/comm-central/source/mailnews/local/src/nsPop3Protocol.cpp#2453

        m_pop3ConData->msg_info[m_listpos - 1].size = atol(token);

Created attachment 9197599 [details] tb.log.moz_log

2nd email:
| SEND: LIST
| [...]
| RECV: 6 1516
| [...]
| SEND: RETR 6
| [...]
| RECV: +OK 1516 octets
| RECV: [=> 1516 octets]

CHECK - Size fits.

1st email:
| SEND: LIST
| [...]
| RECV: 15 45032
| [...]
| SEND: RETR 15
| [...]
| RECV: (null)
| RECV: [=> 7245 octets]

ERROR: 45032 ≠ 7245

Comment 26

[Alfred Peters [:infofrommozilla]]

But the question is: Should we change the default of mail.server.default.dot_fix to FALSE

Comment 27

[gene smith]

If it's only to work around servers that don't dot-stuff, I say just default it to FALSE and avoid the complaints about a final dot appearing in some text emails. If the server doesn't dot-stuff like it should, then it's a server bug (which should now be quite rare but can be worked-around by setting dot_fix back to True).

Comment 28

[Wayne Mery (:wsmwk)]

Do you agree with comment 27?

Comment 29

[Magnus Melin [:mkmelin]]

Sound like we should yes.
(I don't see a dot with o365 nor any other servers I use.)

Comment 30

[Magnus Melin [:mkmelin]]

Attachment: Bug 196584 - default dot-stuffing to false. r=rnons

Comment 31

[Alfred Peters [:infofrommozilla]]

Is the Pref still relevant at all?
Or maybe it died with the POP3-JS module?

https://searchfox.org/comm-central/search?q=dot_%3FFix&path=&case=false&regexp=true

Comment 32

[Magnus Melin [:mkmelin]]

Ping noted the same in the review. Indeed it's not implemented, and I don't know of any complaints so far. We should probably just remove the leftovers.

Comment 33

[Magnus Melin [:mkmelin]]

Attachment: Bug 196584 - remove leftovers of pop3 dot-stuffing (dotFix, dot_fix). r=rnons

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4b25111bcf1e
remove leftovers of pop3 dot-stuffing (dotFix, dot_fix). r=rnons