126672 - publishing doesn't put in both cr/lf; it does only platform-specific setting

Status: VERIFIED FIXED

(SeaMonkey :: Composer, defect)

Description

[sujay]

Kathy has a fix for this...

Comment 1

[Kathleen :Brade]

Attachment: patch to use persist flags and always publish with cr/lf

Note: specifications indicate that http should be able to accept all cr/lf
variations but that's only http.  4.x sends both cr and lf so we should do this
for 6.x.

Comment 2

[kinmoz]

Comment on attachment 70543 [details] [diff] [review]
patch to use persist flags and always publish with cr/lf

sr=[e-mail]

It would nicer if we could identify the platform we are going to and adjust the
output flags accordingly, but I understand that part of the problem is that we
have to set these flags before the channels are opened, which means that info
isn't available yet.

CR/LF is readable on all platforms so I'll give a thumbs up on this.

Could we add a comment notating why we are using CR/LF?

Comment 3

[Akkana Peck]

Kathy asked me to review, but I can't in good conscience.  This will make me
avoid using publishing for my own pages; it would bother me very much if my
files all ended up with Windows linebreaks when they'd never been touched by a
Windows machine, so that I had o run a script on them before I edited them with
something other than composer.  Mapping linebreaks to something that doesn't
match the platform native line break on either the user's machine or the server
is evil.

Sniffing the server first to determine its platform would be worth it, IMO, to
avoid mis-mapping (mapping to the linebreaks that are correct for the server
makes a lot of sense to me).

Kathy and I discussed it at some length and I said I would clarify that I'm not
trying to block checkin of this, just make sure everybody understands that it's
evil and it's going to piss some people off.  If the only audience we care about
is people who use composer exclusively and will never try to edit the source
themselves, then perhaps it doesn't matter.

Kathy asked if I would prefer if files on the server were mismatched, some with
mac linebreaks, some with windows, etc. depending on who edited the file -- and
yes, that doesn't bother me, since it's what normally would have happened before
publishing, when multiple people edit files locally and ftp them.

And yes, if you check this in, please do add a comment explaining it (perhaps
referring back to this bug) so someone like me doesn't come along later and try
to "fix" it.

Comment 4

[Charles Manske]

What does a standard FTP program do? I bet it simply moves the bits and doesn't
touch CR/LF, thus the server gets files with different OS cr/lf styles.
So I'm leaning toward Akkana's opinion to not to any substitutions during 
publishing. 

Comment 5

[Charles Manske]

Kathy says FTP programs do appropriate cr/lf conversions, so I think that is what
we should do. But for first release, it seems to be non-trivial to figure out
server OS, so I'm ok with just a particular convention for now. I don't have a
strong opinion on which one.

Comment 6

[Akkana Peck]

Here are the results of my testing:
ftp file with LF from linux to linux: linebreaks remain LF
ftp file with CRLF linebreaks from linux to linux: linebreaks remain CRLF
ftp file with CR linebreaks from mac OS9.04 to linux using fetch: linebreaks
change to LF.
LF linebreaks from linux to OSX: remain LF
CRLF linebreaks from linux to OSX: remain windows
CR linebreaks from linux to OSX: remain CR
CRLF linebreaks from Windows (ftp in DOS prompt under Win ME) to linux:
linebreaks change to LF

I don't know how to test ftp to windows or Mac OS9 -- how does one run a server
on those OSes?  Maybe Charley can help.

Conclusion: Windows and mac ftp translate linebreaks appropriately for the
server (i.e. they don't translate blindly to Windows).  Linux ftp doesn't
translate at all.  None of the clients tested mapped to CRLF when uploading to a
linux server. 

Comment 7

[Kathleen :Brade]

Can you clarify what ftp client software you were using?
What about http?

important note about ftp:  ftp should do translation (mozilla doesn't work
correctly because it always sends data in binary format instead of text)  This
issue is a separate bug from what http should do.

Comment 8

[Akkana Peck]

Note that this means that windows/mac clients, when downloading a file with LF
from a unix server, will also do appropriate substitution back to the current
platform's line breaks (verified with mac, not tested with windows).  One
relevant question is: what happens if you have a file that uses CRLF residing on
a Unix server?  Obviously it will be right on windows and wrong on unix.  Will
the mac translate this file correctly, or will it end up with double linebreaks
(CR translated from the Unix LF, plus the extra CR that didn't belong on Unix)?
 Likewise, what happens if you have a file with linux line breaks on a windows
server, and you ftp download it?  It'll be right on linux, but what will it look
like on windows and mac?

Last study I read, something like 22% of servers were Windows, and that number
may be declining (or so the news sites say) due to fears over security issues. 
So if we're going to guess based on what type of server is more likely, Unix is
a better guess than Windows.  Looks like both aol.com and netscape.com are both
running solaris, so Unix linebreaks are consistent with our own servers.

What's the argument for using CRLF?

Comment 9

[Akkana Peck]

Here's the list of ftp clients again: mac=fetch, windows=ftp from dos shell,
linux=ftp from command line.

Comment 10

[Kathleen :Brade]

The arguments for CRLF include:
  4.x parity  (tested with Linux 4.x Composer publishing to Linux server)
  text/html line ending canonical form is CRLF
  ftp clients who send text (which we don't at the moment) should send CRLF
    (servers "fix" line endings if necessary)

Relevant rfc's include:
  text/html (http://ietf.org/rfc/rfc2854.txt)
    section 4: 'As with all MIME text subtypes, the canonical form of 
               "text/html" must always represent a line break as a sequence 
               of a CR byte value (0x0D) followed by an LF (0x0A) byte value.'

  http 1.1 (http://ietf.org/rfc/rfc2616.txt)
    section 3.7.1: 'When in canonical form, media subtypes of the "text" type 
                   use CRLF as the text line break. HTTP relaxes this 
                   requirement and allows the transport of text media with 
                   plain CR or LF alone representing a line break when it is
                   done consistently for an entire entity-body.'
    
  FTP (http://ietf.org/rfc/rfc0959.txt)
    3.1.1.1: "In accordance with the NVT standard, the <CRLF> sequence should 
              be used where necessary to denote the end of a line of text."

My conclusions:
  FTP issue is non-issue at the moment due to bug where ftp data is always
uploaded in binary form.  If we ever fix that bug, we need to always send CRLF
(unless I misunderstand what I've read).
  HTTP doesn't really care what format you use as long as the whole document
uses the same technique.  CRLF is the canonical form for text/html files.
  I prefer we go with the patch in this bug for now.  It is the same as 4.x
behavior so our users won't be surprised.
  I'd be ok with just sending LF based on percentage argument Akkana posed but
we'll be wrong some day if/when ftp upload is fixed to send only binary data as
binary.

Comment 11

[Adam Lock]

I believe that CRLF is the correct way to publish files though it would be nice
to see an ascii mode for uploading text on ftp channels.

Comment 12

[Kathleen :Brade]

ftp binary (always) issue is now in bug 127292

Comment 13

[Kathleen :Brade]

fixed

Comment 14

[sujay]

verified.