NSPR can't allocate high memory on OS/2



16 years ago
15 years ago


(Reporter: Julien Pierre, Assigned: Wan-Teh Chang)


Firefox Tracking Flags

(Not tracked)




16 years ago
Before Warp Server for e-business, OS/2 had a process address space limit of 512
MB. The reason for this was so that any block of memory could be shared between
16-bit and 32-bit OS/2 applications.

However, IBM lifted this limit in the Warp Server for e-business kernels. It is
possible with these kernels to tell OS/2 when allocating memory that it can be
above the 512 MB limit. I suggest that we do this by default in NSPR so that an
NSPR application is not subject to the limit. This is more than a theoritical
limit, I actually ran into it.

Comment 1

16 years ago
Did it go in a Merlin fixpak?

If not, we can't do it.

Most OS/2 users are running merlin with fixpak 14.

Comment 2

16 years ago

I'm not sure if it did - my guess is no. However, this may be a change that is
still compatible with older versions. It can easily be tested if you have access
to a machine running Warp 4.0. Both of my home PCs are running Warp server for
e-business, as well as the work system.

Unfortunately, I couldn't find the documentation on the >512 MB support in the
Warp Server for e-business documentation. There are only two documents that are
relevant - the base "Control Program Programming Guide and Reference" and the
"OS/2 Programming Guide and Reference Addendum". It's not listed in either place.
I know this is something that got added in Warp Server 3 Advanced SMP (ie.
before E-business and before Warp 4.0), and it was documented in the addendum
for its Toolkit. I no longer have it installed. I would have expected it to be
included in the WSEB doc, but it isn't. 

Regardless, I found it in the headers, and the flag for DosAllocMem to allocate
memory above 512 MB is OBJ_ANY, which is defined as 0x00000400U. This tells OS/2
that it can allocate memory "anywhere" according to the comment in bsememf.h .
The meaning is "anywhere including above the 512 MB limit". My guess (and hope)
is that on a Warp 4 system, this flag will be ignored, and DosAllocMem will
simply return memory below the 512 MB limit. But I'm not sure of that, I haven't
used Warp 4.0 in years. It needs to be tested. If the flag causes an error, a
specific error will probably be returned, and in the worst case we could try
again without OBJ_ANY.

Comment 3

16 years ago
There is another problem in the way of implementing this, which is potentially a
lot more serious. It is related to the browser's very relaxed memory management

A lot of the code in the browser currently mixes PR_Malloc() with a C-library
free() call, or malloc() and PR_Free(). This is alright today because PR_Malloc
currently maps to the C library malloc() call.

If however we starting mapping it to DosAllocMem - with or without the OBJ_ANY
flag - both of these cases will cause either a crash or a memory leak (depending
how relaxed the free() function in the C runtime is). We know about the problem
because we have another NSPR allocator called the zone allocator, which doesn't
use the C malloc() function directly. This zone allocator is only implemented on
NT and Unix currently. It is turned off by default and is enabled with an
environment variable. We know that it is incompatible with several Netscape
server applications that haven't bothered to straighten their allocation code,
and the client suffers from this problem too.

So if we go with this other allocator, all the code in the browser needs to be
reviewed. Basically, there should never be any calls to malloc() or free(), only
PR_Malloc() or PR_Free(). This is something that should be done regardless to
clean the browser code, as it stands in the way of implementing a more efficient
allocator on any platform.

A possible short-term transparent fix which doesn't involve fixing the incorrect
browser code would be to fix the C runtime library malloc() call to use OBJ_ANY
. I don't know of any way to do this with the C runtime from either VACPP or
EMX. Unlike on Unix, we can't provide new malloc() and free() functions and
override the existing functions in the C runtime library DLL by virtue of DLL
linking order in the executable.

However, on OS/2, there is no single shared C runtime library. Every application
ships with its own C runtime DLL - including Mozilla. So it should be possible
to do this. But it would require rebuilding the C runtime DLL from source, with
modified malloc and free calls. The C library source is not included with IBM's
compiler, so I don't think that's an option either.

Perhaps the best choice would be to couple this implementation of >512 MB memory
allocation on OS/2 with the implementation of the zone allocator, since the
later is already incompatible with the browser currently. Applications that need
large amounts of memory are also likely to be using this zone allocator, so this
may be a good compromise.

Comment 4

15 years ago
I wanted to give a quick shot at trying to run some NSS apps with a modified
NSPR allocator that would allow allocating >512 MB memory (ie. calls DosAllocMem
with OBJ_ANY).
However, I ran into the problem that I need an implementation of realloc() ...

Mike, is it at all possible to obtain the source code to the C runtime library ?
Short of that, a binary patch of the DLL is probably the easiest fix.

However, I'd much prefer the source because I'm investigating another problem
that I suspect involves locking in the allocator of the C runtime library (164512).

Comment 5

15 years ago
I'll try to get the source, but there is no way I can support a Mozilla that is
using a custom C runtime.

This is simply going to be a limitation of Mozilla on Os/2.

Note that this could be a nice opportunity to fix the issue that Mozilla has of
using a zillion different calls throughout that go to malloc rather than PR_Malloc.

Comment 6

15 years ago
I recommend against this. Many OS/2 APIs will cause crashes, traps or weird 
behavior when they get high memory buffers. We have a lot of experience on this 
issue and we ended up with many workarounds in our applications where we 
required high memory.

Comment 7

15 years ago
This can't be fixed.

Memory allocated above 512MB cannot be passed to all OS/2 APIs, so we cannot
change the default Mozilla allocators to allocate memory in this area.

The only way allocating above 512MB is useful is in cases where you have
complete control of what is happening with that memory which is simply not the
case in Mozilla.

the 512MB limit is going to be a permanent restriction of Mozilla on OS/2.

The GCC folks agree with me as well - GCC will not allocate memory above 512MB
for this same reason.

Last Resolved: 15 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.