131677 - VFC versus Stream-LF file conflicts

Status: VERIFIED FIXED

(SeaMonkey :: Composer, defect)

Description

[Dirk Munk]

This is a OpenVMS specific problem !

The composer is written in C, and most likely it uses standard C file routines 
instead of RMS routines. Hence it produces standard C Stream-LF files with a 
max. record length of 32676 bytes.

Other editors and FTP ASCII file transfers produce VFC files.

The composer can open a VFC file, and all looks well. It is possible to edit 
the file (on screen), and it is even seemingly possible to save the file. 
However in reality the new contents will not be written, and the save operation 
will result in a empty file.

In my view there are several solutions to this problem:

1. The composer will create a new version of the file in Stream-LF format. 
After all this is the standard VMS way of doing these things. So we will get 
index.html;1 , index.html;2 , index.html;3 and so on.

2. The composer can handle VFC files. I assume that this can be quite a problem 
if it is necessary to use RMS file handles.

3. The composer will not open the file, but will give a warning instead that 
the file is not in Stream-LF format.

I came across this problem when I used FTP to move a HTML file from my PC to 
the VMS system, and that is a rather normal operation.
Apache (sorry, Compaq Secure Web Server) can use both type of files, so Stream-
LF is not mandatory for the web server.

Comment 1

[Colin Blake]

You are absolutely right. At the very least Mozilla needs to say it can't handle 
these files. I need to figure out what can be done to solve this problem.

Comment 2

[Colin Blake]

Gee, I don't know how to fix this one. The real solution would be to check each 
file before we open() it and see if its VFC (or any other unsupported format), 
and then refuse to open() it. But that would mean adding a stat() call before 
each open() and that would have a significant performance impact. I really don't 
want to do that.

Or I could add some code to the composer component to check the file format 
before opening it (assuming I'd be allowed to check in this platform specific 
code). But that would only fix the problem for Composer, and I suspect there's 
other places where we'd like to exclude unsupported file formats (eg on a file 
upload from a form).

I could trap each open() and only do the stat() for files being opened for write 
access. That would reduce the extra stat() overhead, but it would mean you'd 
still be able to edit a VFC file, you just wouldn't be able to save it 
afterwards. Not a very nice solution.

Any other ideas?

Comment 3

[Dirk Munk]

IMHO the problem lies in the fact that a existing file is rewritten. This normal
for most platforms I assume. But isn't it possible to open a new version of the
file? So a open-for-write would in fact result in a open-for-read of the
existing file, plus a open-for-write for a new version with the same name. (That
version can have the Stream-LF characteristics.)

This is the normal beheaviour for any VMS editor etc. And even without this
problem, this is the way I would expect the editor to work anyway. I would like
to be able to open a existing file, do some editing, save in between, and when I
close the composer, I still would like to be able to go back to the original
file by just deleting the last version(s). 

VMS users will expect this in my view, and doing things this way will
automatically resolve this problem. 

Comment 4

[Colin Blake]

Unfortunately the way Mozilla writes out a modified file is to open the file, 
rewind it, and then start writing again. This is the behaviour which leads to 
"no new version". I agree its very unVMSlike, but that's the way it is.

I'm looking at ways of having Mozilla refuse to open files which are of a format 
it can't read correctly. I think this is the correct fix. Otherwise we're going 
to be displaying the meta-data in the composer window, or someone will complain 
that we switched file types on them, or something. Just not opening the file 
seems like the simplest, cleanest, approach.

Comment 5

[Dirk Munk]

I understand the validity of your arguments, but the no-new-version situation in
general doesn't make me happy. 

Maybe it's difficult for this version, but in general terms, wouldn't it be
possible to setup OS specific file handling modules? This would make
error-handling etc. a lot easier. Or maybe it could be a feature of the
C-compiler, I don't know. 

I quite often come accross C-programs without any (file) error-handling what so
ever, or we get a useless error status. So as support people we can search for
hours, days or weeks for problems that could be solved in minutes if we had
normal status checks and error codes in these programs.

Sorry for this lament, but I think it is such a shame that it isn't possible (or
done) to combine the virtues of a great multi-platform project like this with
the  virtues of the operating systems involved. I suppose that should be
possible with a good modular setup. The result could be great sofware, instead
of "yeah, it looks great but ......"

No offense of course, I think you're doing a great job. Else I wouldn't bother
making error reports :-)


   

Comment 6

[Colin Blake]

Mozilla does have platform-specific file handling modules, and currently OpenVMS 
uses the Unix modules. There's no technical reason why we couldn't (shouldn't?) 
have our own OpenVMS file module and fix some of these problems, and make 
Mozilla behave a little more VMSlike. But the reality of the situation is how 
much effort this would take. For the past 2-3 years Mozilla has been evolving 
and changing very rapidly. The ONLY reason we've been able to get OpenVMS ports 
out in a timely manner is BECAUSE we are building off of the UNIX code. If we 
were to introduce large hunks of OpenVMS specific code then we (OpenVMS) would 
have to support it, and that would include keeping it up to date and in sync 
with the rest of the Mozilla code base. To give you some idea of how rapidly 
Mozilla is evolving, during one week of January alone Mozilla had over 1500 
checkins from 91 different people, resulting in over 30,000 lines of code being 
added/changed. If we had to duplicate some of these changes into OpenVMS 
specific modules you wouldn't see an OpenVMS release of Mozilla for a long time!

Maybe once 1.0 is out development will slow down (but I wouldn't bet on it!) and 
we can start to look at some of these issues. But for now our only choice is to 
port and ship the UNIX code. And that means from time to time you see some 
undesirable behaviour. Sorry, but that's life.

Comment 7

[Dirk Munk]

OK, point taken !!

Sometimes I wish VMS wasn't so open. We can use Unix sources, WIN32 sources, SUN
Unix sources (shortly), and most likely Nintendo sources shortly too when this
goes on. (Just kidding :-) )

Comment 8

[Colin Blake]

Assigning to default owner because I need some information. Where is the correct 
place to check for file size and report an error if we don't have a size (this 
is the right test since the nsLocalFile::GetFileSize routine on OpenVMS will 
return an error for unsupported file types). I tried in nsFileIO::Open and 
nsFileTransport::Process but in both those cases returning an error just hung 
Composer.

Comment 9

[Colin Blake]

Dirk, I've been thinking about this one a lot, and have a possible solution.

First off though, I was unable to find a way to prevent Composer (in fact 
Mozilla in general) from opening a file of VFC format. The only way would be to 
stat() the file first and that would create too much overhead. But when a VFC 
file is displayed it doesn't look right, because the record format control 
characters have been interpreted as data. This should be a BIG clue that 
something is wrong!

The real problem before was that if you tried to save a file that was VFC format 
you ended up with an empty file and effectively lost all your data. This problem 
I believe I have fixed. All files created by Mozilla are now of stmlf format, 
even files which already exist and are of a different format. 

So now, if you load a VFC file into Composer, you see some funny characters at 
the start of each line (the record format data). If you choose to ignore this 
and save the file, you will now have a stmlf format file which contains the HTML 
for exactly what you saw on the screen (ie. it will include the record format 
data). This is now exactly perfect, but a lot better than it was before.

Files of VAR format are read in successfully, displayed correctly, and saved (in 
stmlf format) correctly.

If you'd like to try this new fix (and I'd really like you to before 1.0 ships), 
let me know where I can ftp or mail a new shareable to.

Thanks, Colin.

Comment 10

[Colin Blake]

Dirk has tried the new image I sent hime and he reports that things are working
better. No more lost data.

Comment 11

[sujay]

verified per comments.