108137 - enable build for SmartHeap

Status: RESOLVED INCOMPLETE

(SeaMonkey :: Build Config, defect)

Description

[Chris Waterson]

Allow mozilla to be built with the SmartHeap library.

Comment 1

[Chris Waterson]

Attachment: patch that doesn't work

Comment 2

[Chris Waterson]

Attachment 56222 [details] [diff] appears to create DLLs that have no reference to MSVC's malloc
(); however, the build doesn't run and gets heap corruption errors on startup.

Comment 3

[Chris Waterson]

Attachment: boioioiong. use DLL_LIBS, like francois & cathleen said

Comment 4

[Chris Waterson]

The above patch produces a working, but very unstable build. There are still 
several libraries that appear to be pulling in malloc et al. from MSVC, 
including:

  nspr4(_strdup)
  plds4(malloc, free)
  mozreg(malloc, free)
  smimestb(malloc, free)
  ISimpleDOMNodeMarshal(malloc, free)
  ISimpleDOMDocumentMarshal(malloc, free)
  nssckbi(malloc, free)
  MapiProxy(malloc, free)
  mozMapi32(malloc, free)
  oji(_strdup)
  xpacct32(_strdup)
  appcomps(_strdup)
  xpinstall(_strdup)
  xpcom(_strdup)
  js3250(_strdup)

I'm not sure why we're pulling in malloc and free; I suspect we're getting 
_strdup due to some MS header file funkiness. I'll dig more tomorrow.

Comment 5

[Chris Waterson]

Well, it turns out that shlw32m.lib _doesn't have_ strdup(). That's going to be 
a problem.

Comment 6

[Chris Waterson]

Attachment: link -verbose output from plds4.dll

Comment 7

[Chris Waterson]

Hey jband, rpotts - I'm trying to wedge SmartHeap into the build (again!), and 
am running into a few problems.

First, the library that I've got doesn't have strdup (I'm currently using a 
bootleg copy while I wait for AOL's fine procurement machinery to lumber into 
gear). I suspect this causes heap corruption when we use SmartHeap's free on 
memory allocated with MSVCRT's strdup.

That aside, I'm still getting some bizarre stuff where the linker seems to 
insist upon putting references to MSVCRT's malloc and free: see the above 
|link -verbose| output (attachment 56311 [details]) from an attempt to link plds4.dll 
using the following command:

  link -nologo -DLL -SUBSYSTEM:WINDOWS -PDB:NONE -OUT:"plds4.dll"
     -DEBUG -DEBUGTYPE:CV -MAP /BASE:0x30000000
     shlw32m.lib
     c:/waterson/NETSCAPE_6_2_RELEASE/mozilla/dist/WIN32_O.OBJ/lib/nspr4.lib
     user32.lib ./plarena.obj ./plhash.obj ./plvrsion.obj  ./plds.res

Surprisingly, a nearly identical command to link plc4.dll results in no 
references to the malloc in MSVCRT.DLL:

  link -nologo -DLL -SUBSYSTEM:WINDOWS -PDB:NONE -OUT:"plc4.dll" -DEBUG
    -DEBUGTYPE:CV -MAP /BASE:0x30000000
    shlw32m.lib
    c:/waterson/NETSCAPE_6_2_RELEASE/mozilla/dist/WIN32_O.OBJ/lib/nspr4.lib
    ./plvrsion.obj ./strlen.obj ./strcpy.obj ./strdup.obj ./strcat.obj
    ./strcmp.obj ./strccmp.obj ./strchr.obj ./strpbrk.obj ./strstr.obj
    ./strcstr.obj ./base64.obj ./plerror.obj ./plgetopt.obj  ./plc.res

I've tried re-linking plds4.dll by hand, removing user.lib, but to no avail.

Comment 8

[Chris Waterson]

cathleen wants to fix this.

Comment 9

[John Bandhauer]

Is this really supposed to be so easy? Waterson, do you know what happened to 
that dll entrypoint hooker code I passed off to mjudge? We were going to use 
this for tracemalloc support on Win32. That stuff had support for hooking all 
the allocation entry points (that we knew of at the time). Some of them are 
mangled in interesting ways.

Comment 10

[John Bandhauer]

Attachment: nsDebugHelpWin32.zip that I gave to mjudge last year (in case anyone cares)

Comment 11

[Chris Waterson]

(*shrug*) We're flying a bit blind right now (somebody mailed us a single static
library, our copy of SmartHeap is in that wonderful procurement black hole
called ``on order''). The static lib does appear to provide operator new,
operator delete, the array versions of these, malloc, free, calloc, and realloc,
and AFAICT, _most_ of the time they're being used.

I'll mail you the SH library if you want to take a crack at this.

Comment 12

[rpotts (gone)]

hey chris,

why does the linker think that 'MSVCRT' is a default lib?  have you tried adding 
/NODEFAULTLIB:MSVCRT and then explicitly adding it *after* the smartheap lib (if 
necessary)...

-- rick

Comment 13

[Jonadab the Unsightly One (Nathan Eady)]

Looks like this won't make it in for 0.9.6.

Comment 14

[Martin Poirier]

This didn't make it in for 0.9.8, will it be 0.9.9 ?