170911 - NSPR cannot deal with >2GB files on OS/2

Status: RESOLVED FIXED

(NSPR :: NSPR, enhancement, P3)

Description

[Julien Pierre]

Warp Server for E-business introduced support for >2GB files on JFS.
This support requires the use of special kernel APIs, post-fixed with "L".

Currently, the lack of support in NSPR means that odd things will happen when
trying to deal with such files.

a) In the browser, it has the following effect when browsing a JFS volume, for
example with the URL "file:///j:/". A file greater than 2 GB in size show as 1 byte.

This is because NSPR currently implements the GETFILEINFO, GETFILEINFO64 and
MD_STAT calls using the VAC++ C runtime library "fstat" call. That call does not
support >2GB file.

The correct way is to call DosQueryFileInfo and pass FIL_STANDARDL to it, which
will obtain the size as a 64-bit integer (LONGLONG). 
Fortunately, this can be done on all OS/2 kernels, even ones without the
support. If the call fails, one can always fall back to "FIL_STANDARD" which
will return a 32-bit size . Ideally we should check the OS/2 version and see if
we have a kernel that supports >2GB files & JFS in order to choose which  one to
try first.

b) When double-clicking on the >2GB file, one is told that "the file /j:/big
cannot be found". I believe this is because the large file needs to be opened
with DosOpenL rather than DosOpen. Otherwise it cannot be read.
I think this is a new API in Warp Server for E-business, and doesn't exist in
older kernels.
If we need to have a single version of NSPR that runs on all OS/2 kernels and
support large files if the kernel does, then we'll need to play tricks by
loading the symbol dynamically from DOSCALL1.DLL .

c) there may be other issues related to >2GB in OS/2 NSPR with seek . I haven't
checkd that yet.

Comment 1

[Julien Pierre]

Attachment: Program to create a >2GB file on JFS on OS/2

This can be used to demonstrate the problem with Mozilla & NSPR.
Compile and type "Fill j:\big 5120".
This will create a 5 GB file.
Then go to "file:///j:/" in Warpzilla.

Comment 2

[Mike Kaply [:mkaply]]

I know the APIs to do this, and Scott Garfinkle has been bugging me about it, 
but in reality it would not be terribly useful because the client uses C 
runtime file APIs all over the place in addition to NSPR...

Comment 3

[Scott Garfinkle]

I agree with Mike. I think the large file support should be limited to creating
large file via http or ftp.

Comment 4

[Julien Pierre]

Well, there are other possible uses for it.
For example you can use a streaming browser plugin to play a large video file
off the disk.

Comment 5

[Mike Kaply [:mkaply]]

No, you can't. Because the plugin doesn't support large files.

Comment 6

[Julien Pierre]

Mike,

In a streaming plug-in, the plug-in doesn't open a file. Typically, the entire
content or its size is not available when starting play. This is the reason for
streaming. The plug-in gets data delivered to it in individual small buffers by
Mozilla. It shouldn't matter to the plug-in where the data comes from (HTTP or
disk), or how much data there is. It should just keep playing until it gets to
the end of the stream.

In this case it would be up to the plug-in subsystem of Mozilla to read the disk
file and deliver the data progressively as a stream to the plug-in, and that's a
place that could easily take advantage of the 64-bit NSPR file APIs, if it
doesn't already. Note that these may not even be needed since you just need to
do a PR_Open and PR_Read() until you reach EOF to stream, which don't require
64-bit parameters since the data is streamed in small buffers that fit into RAM
by definition. What's neeeded is the ability to open and read from the file.
This is not possible currently because DosOpen fails on a large file.

Comment 7

[Scott Garfinkle]

It still seems to me like a lot of potential for problems and confusion for very
little return. Even the testing gets very ugly. 

Comment 8

[Julien Pierre]

Scott,

That's why the priority of this bug was demoted to P3. I still hope to finish
the patch for this some day though.

Comment 9

[Max Alekseyev]

Won't "large file support" (introduced in OS/2 GCC 3.2.2 beta 3) help to resolve
the issue?

Comment 10

[Mike Kaply [:mkaply]]

Not really because we use a lot of OS/2 APIs internally as well to deal with files.

I honestly have yet to find a compelling reason to do this work.

Comment 11

[Julien Pierre]

Max,

NSPR directly uses the OS/2 APIs to access files. It does not use the compiler's
runtime library. So the support for large files in the newest gcc does not help
NSPR support large files.
I think it would be really good to have this support. I agree with Mike that it
may not be necessary immediately today, but at some point people will be
downloading files larger than 2 GB. I know I'm already downloading CD-ROM ISO
files which are 650 MB. Often the products are compromised of more than one ISO
file. One could imagine that they would be packaged in a single file in the
future, that would exceed 2 GB. Also, DVD ISO files will exceed 2 GB so the need
will become more pressing (I just got a DVD burner but have not played with it
much yet).

Comment 12

[Julien Pierre]

in the previous comment, compromised should read comprised ...

One of the questions about the implementation of this patch is whether we want
to bother doing the dynamic loading of the optional 64-bit file APIs.
Those APIs are only available in WSEB kernel, and the newer UNI kernels.
They are not exported from DOSCALL1.DLL in older kernels such as Warp 3 and Warp 4.

Possible solutions include :
1) Call the new 64-bit file APIs directly. This means NSPR will require a newer
OS/2 Toolkit to build, and will only run on the newer kernels. This drops
compatibility for Warp 3 because there will be unresolved symbols when loading
NSPR4.DLL . It would be fine with me (after all, Warp 3 kernel is 10 years old
and even Warp 4 is 8 years old), but I don't know if it is acceptable to IBM.

2) Use conditional compilation to use the 32-bit or 64-bit file APIs, and make
two builds of NSPR, one for Warp 4 and earlier, another for WSEB and ACPx. This
means there would need to be two Mozilla OS/2 builds instead of one.

3) Try to dynamically load the 64-bit file APIs and store pointers to them if
available. Then the NSPR code can do the right thing at runtime. This is the
most complicated solution to implement of the three, but it will only require a
single build and will keep compatibility even with kernels from the dinosaur days.

I believe I originally tried to implement 3). I would like to know what gcc did
in this regard when they added OS/2 large file support. If they did 1), then the
C runtime already depends on the new kernels, so there is no sense in bothering
with 3) for NSPR. They didn't do 2) because I only saw one gcc runtime for OS/2.
If they did 3), then it makes sense to do the same for NSPR.

Comment 13

[Julien Pierre]

I downloaded the source code for the gcc3 OS/2 runtime . They implemented option
3 - ie. they open a handle to DOSCALLS, and try to obtain pointers to each of
the DOS*L kernel calls for large file support. That means they still support the
older pre-WSEB kernels.

We should do the same for Mozilla .

I'm having a much more pressing need for this feature than two years ago when I
first opened this bug . So, I will again be working on an NSPR patch, as soon as
I get my gcc3 NSPR environment up-and-running . Unfortunately,  I didn't keep a
copy of my old patch - but it wasn't completed and working so it's not a huge loss.

I don't have a working gcc3 build environment right now, and I managed to mess
up my VAC build environment too, so I can't make any patch right now. Once
written, the new NSPR OS/2 large file support code should compile and run with
either VAC or gcc compiler, though.

Comment 14

[Mike Kaply [:mkaply]]

It's yours

Comment 15

[Julien Pierre]

While working on the patch (I'm doing it on the 4.4.1 branch, since I'm still
using VAC++ and the NSPR tip won't compile), I noticed that there are plenty of
"#ifdef NO_LONG_LONG" in the OS/2 NSPR source code. I'm wondering if it would be
OK to get rid of all that garbage . Both VAC++ 3.6.5 and the current version of
gcc support "long long" . I don't think VAC 3.0 did, but that's not been used
for years for any Mozilla project that I know. Please correct me if I'm wrong.

Also, I plan to make the patch change the build to require the WSEB OS/2
toolkit, which has the large file APIs and structures defined in its headers,
even though the runtime NSPR code will be able to run on all kernels. Please let
me know if there is any problem with that new requirement.

Comment 16

[Julien Pierre]

I have found that the "Programming Guide and Reference Addendum" which describes
the >2GB OS/2 APIs is awful. The examples are supposed to be C, but aren't. For
example for DOSFindNext, there aren't any { } braces anywhere in the program
(for main, while loops, or if statements). And once the horrid syntax is fixed
to the point of compiling/running, there is a bug - the last file gets displayed
twice ;-)

Also, the text in the documentation isn't in agreement with the implementation.
For DosFindFirst, the doc says if you pass FIL_STANDARDL, info is supposed to be
returned in a FILESTATUS3L structure . In fact, it gets returned into a
FILEFINDBUF3L structure (and discovering that is the only thing the pseudo-C
code example in DOSFindNext is good for! They can't even get the loop right and
the last ).

I'm going to go with the behavior from the code, rather than what the doc says ...

Comment 17

[Julien Pierre]

Attachment: preliminary patch

This patch adds support for 2GB files. It is made on top of NSPR_4_4_1_RTM .
I'm not sure if it will apply to the NSPR tip - but the tip is broken with
VAC++ right now, so I can't work on the tip.

I have checked that this code compiles and links on my system. I did not test
if any of the new code ran. I may be able to test it against an old VAC++ build
of Mozilla .

Right now, I have defined a static variable "isWSEB" which is meant to tell
whether we are running on a kernel with large file support or not.

I still need to :
1) set the value of that variable dynamically at NSPR initialization time
2) dynamically query the pointers to the new functions (DosOpenL) and store
them in pointers (eg. _DosOpenL) . This would be done at NSPR initialization
time also
3) change the code that calls the new APIs to call them through the function
pointers

Comment 18

[Julien Pierre]

I did some testing with Mozilla 1.4, which was built with VAC++ . I just
replaced the NSPR DLLs with mine .

My first test was to display
file:///F|/ .

The output did not change. The files larger than 2 GB still got reported with a
"1 byte" size . This suggests that either my code did not work, or Mozilla/2 is
using the wrong NSPR function (32-bit instead of 64-bit).

Then, I clicked on one of the files, and tried to save it.

It brought up a window asking what to do with the file - either open it in an
application, or save it.

Without my patch, OS/2 denies the open of the file, and Mozilla ignores any
click on any of these files.

I chose the option to save the file to another volume (second JFS volume, on a
different physical disk).

Mozilla started reading the file at a very rapid rate. However, it did not write
anything to the target disk, unfortunately. I'm wondering if the "save" code in
Mozilla uses NSPR for writing the file or not when saving. If I open smaller
files from disk and save them to the other disk, it works fine.

This suggests that my patch might be OK, but there may be more than just NSPR to
make the browser handle those files - ie. browser code to be fixed.

Comment 19

[Julien Pierre]

In my previous test, Mozilla was saving the files to %TMP%, which pointed to a
different drive . So I pointed my TMP drive to the same drive as my target .
I tried to copy a 5 GB file from one JFS partition to another using Mozilla's
"save as" feature .
It appeared to work for a while - it sure broke the 2 GB barrier.
But, unfortunately, it stopped at 4GB ;-)) I'm pretty sure that has to do with
browser code.

I think the current browser code handles downloads much better (I know for sure
on unix platforms it does), so it would be worth trying my patch with the
current browser code. But the newest Mozilla browser builds only with gcc, and I
still don't have my gcc environment working, so I can't try it myself.

Comment 20

[Julien Pierre]

The reason the size still got reported as 1 is that I didn't implement
_PR_MD_GETFILEINFO64 and _PR_MD_GETOPENFILEINFO64 .

I think I would have to use a totally different approach than the current
implementation, which uses the libc _stat and fstat calls. I looked at the gcc
runtime source, and it doesn't implement 64-bit fstat, so this will have to be
brand-new code, and it will be compiler-independent, unlike the current code
which seems to depend on some features of VAC that the C RTL files are OS/2 file
handles.

For _PR_MD_GETOPENFILEINFO64, I can call DosQueryFileInfo with FIL_STANDARDL to
get 64-bit file info. But there is some mapping of fields to be done for
PRFileInfo64 which is somewhat nasty. However, it shouldn't be too bad given
what the gcc runtime does (which uses DosQueryFileInfo with FIL_STANDARD) .

_PR_MD_GETFILEINFO64 is similar, but using DosQueryPathInfo with FIL_STANDARDL .

Comment 21

[Julien Pierre]

Attachment: patch update

I believe this patch completes the large file support for NSPR. It is still
lacking the dynamic loading of large file API functions, however.

Comment 22

[Julien Pierre]

With this new patch, Mozilla 1.4 no longer shows file sizes of "1" for local
large files, which means the support is working . The values seem to wrap over 4
GB, but I believe that's a problem with Mozilla itself, which may have been
resolved since. Hopefully someone with a gcc build can test the patch and try it
with a current mozilla build.

Comment 23

[Julien Pierre]

I was finally able to build with gcc . My latest patch, as attached, works
unmodified with it. Note that I still need to do the dynamic loading of the API
calls before check-in.

When compiling this patch into NSPR, it allows viewing and saving files in the 2
GB - 4 GB range using URLs such as file:///f:/ .

Unfortunately, the browser bug for handling files above 4 GB seems to still be
present in Mozilla 1.6rtm built with gcc, which is what I tested with . I
haven't found a bug on it. I will try on other platforms with the latest code
and file one if appropriate.

Comment 24

[Julien Pierre]

Wan-Teh, what would be the next release of NSPR appropriate for taking this
change ? I assume I missed the train for the NSPR release going in to Mozilla
1.7, but that's OK. Please set the target. Thanks.

Comment 25

[Julien Pierre]

Adding Wan-Teh. Looks like he wasn't cc'ed on this. Doh!

Comment 26

[Wan-Teh Chang]

The next NSPR release is 4.6.  You can target
NSPR 4.6/Mozilla 1.8.  Please have someone
review your patch.

Comment 27

[Julien Pierre]

Attachment: completed patch

I cleaned up the previous patch, and added the dynamic loading of the 64-bit
file functions.  I tested it successfully on a WSEB kernel. Non-WSEB kernel
needs to be tested, but should work.

I changed my implementation for GetOpenFileInfo64 and GetFileInfo64 . Rather
than trying to fully implement stat64 using code borrowed from the GNU C
library as in the previous patch, I just call the 32-bit version first, then
make the extra necessary OS/2 call just to get the 64-bit file size. This may
not be as efficient but is a much cleaner patch.

Mike, Javier, please review.

Comment 28

[Julien Pierre]

Attachment: fix initialization problem

Somehow, the functions only load by ordinal, not by name. I'm not sure why
exactly I didn't catch this in my earlier test, but this is the correct patch.

Comment 29

[Mike Kaply [:mkaply]]

Do the two versions of the function take exactly the same parameters?

If so, rather than using a boolean to determine the path, why not just either
assign the new or old function name to your my* variables.

Comment 30

[Julien Pierre]

Mike,

No, the two sets of functions (L / non-L) don't take exactly the same
parameters. As you can see in the patch, DosOpenL and DosSetFilePtrL take some
LONGLONG arguments which aren't there in DosOpen and DosSetFilePtr .
DosSetFileLocksL has an extra peaop2 argument that's missing in DosSetFileLocks .

Comment 31

[Max Alekseyev]

Julien, shouldn't we load DOSCALLS module instead of DOSCALL1 ? 
I've heard that loading the latter is a bad style ;)

Comment 32

[Julien Pierre]

Max,

Loading DOSCALL1 works on the WSEB kernels, which is where it needs to. I have
no objection to loading DOSCALLS if that works also.

Comment 33

[Julien Pierre]

Comment on attachment 149836 [details] [diff] [review]
fix initialization problem

PR_GetFileInfo64 did not get fixed in this patch. During the simplification, I
forgot to rewrite that function.

Comment 34

[Julien Pierre]

Attachment: clean-up patch

- loaded DOSCALLS instead of DOSCALL1
- fixed PR_GetFileInfo64
- fixed PR_MD_UNLOCK_FILE

I have done very limited testing with Mozilla, and it appears to work with the
file browser (URL of file:///f:/, with a large file created in the root with
the provided test program).

Mike, Javier, please review. I'd like to get this into NSPR 4.6. Thanks.

Comment 35

[Scott Garfinkle]

The proposed changes look OK to me. As an unrelated aside, you might consider
creating md.blocked_sema with the DCE_AUTOREST flag. This would eliminate the
need to call DosResetEventSem after the wait (which call, as coded, code lead to
a race condition where you end up losing an event, anyway). AUTOREST tell the
kernel to wake up any waiters, then go ahead and reset the semaphore atomically.

Comment 36

[jhp (no longer active)]

Comment on attachment 179255 [details] [diff] [review]
clean-up patch

Also looks good to me.	The only nit I have is related to style: fix some of
the tabbing, and try to keep lines to 80 chars or less.

Comment 37

[Julien Pierre]

Scott, Javier,

Thanks for the review. The change you suggest with DCE_AUTOREST probably belongs
in a separate bug/RFE, since I didn't modify the semaphore code in this patch.
Is the DCE_AUTOREST flag supported on all OS/2 kernels ?

Wan-Teh, could you please check the patch in to the NSPR tip and mark this fixed ?

Thanks.

Comment 38

[Julien Pierre]

Comment on attachment 179255 [details] [diff] [review]
clean-up patch

There is something wrong with this patch. It corrupted my POP mailbox when
doing "compact folders". I don't know what the bug is yet. Wan-Teh, please
don't check in.

Comment 39

[Julien Pierre]

Attachment: fix bug in _PR_MD_LSEEK64

The problem was that it wasn't returning the value of newlocationL for the
64-bit case, but rather an uninitialized value ...

Comment 40

[Julien Pierre]

Comment on attachment 179423 [details] [diff] [review]
fix bug in _PR_MD_LSEEK64

I believe this one is finally good for checkin. Really.

Comment 41

[Mike Kaply [:mkaply]]

Comment on attachment 179423 [details] [diff] [review]
fix bug in _PR_MD_LSEEK64

sr=mkaply, a=mkaply - wtc, this is ready to go in.

Comment 42

[Julien Pierre]

I checked this patch in on the tip of NSPR :

Checking in pr/include/md/_os2.h;
/cvsroot/mozilla/nsprpub/pr/include/md/_os2.h,v  <--  _os2.h
new revision: 3.35; previous revision: 3.34
done
Checking in pr/src/md/os2/os2io.c;
/cvsroot/mozilla/nsprpub/pr/src/md/os2/os2io.c,v  <--  os2io.c
new revision: 3.17; previous revision: 3.16
done
Checking in pr/src/md/os2/os2sock.c;
/cvsroot/mozilla/nsprpub/pr/src/md/os2/os2sock.c,v  <--  os2sock.c
new revision: 3.17; previous revision: 3.16
done

As well as on NSPRPUB_PRE_4_2_CLIENT_BRANCH :

Checking in pr/include/md/_os2.h;
/cvsroot/mozilla/nsprpub/pr/include/md/_os2.h,v  <--  _os2.h
new revision: 3.21.2.14; previous revision: 3.21.2.13
done
Checking in pr/src/md/os2/os2io.c;
/cvsroot/mozilla/nsprpub/pr/src/md/os2/os2io.c,v  <--  os2io.c
new revision: 3.11.2.6; previous revision: 3.11.2.5
done
Checking in pr/src/md/os2/os2sock.c;
/cvsroot/mozilla/nsprpub/pr/src/md/os2/os2sock.c,v  <--  os2sock.c
new revision: 3.8.2.9; previous revision: 3.8.2.8
done