Closed Bug 677501 Opened 13 years ago Closed 13 years ago

Build a mozutils library containing jemalloc and other things

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: glandium, Assigned: glandium)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: fixed-in-bs)

Attachments

(1 file, 2 obsolete files)

From bug 662814 comment 26:

So, with the constraints from this API, our build system and our various ways of linking things for the various platforms we target, I think the least worse thing to do would be to have a directory where we'd build a libmozutils library that:
 - includes jemalloc when jemalloc is enabled
 - includes the event log infrastructure from this bug
 - includes what currently is libmozutils on android when building for android
 - is a DSO on win/osx and a static lib on linux (i.e., is whatever jemalloc currently is)

I think doing so as a separate patch (i.e. without the event log infrastructure) first would be better.
An advantage of this approach is that whenever we'll need to add something else that has the same kind of constraints, it will be much easier to handle.
I'm wondering if we shouldn't fold libmozalloc in there as well.
If that works (I know libmozalloc is tricky), then yes, I'd say do it.
I don't think we want to include libmozalloc into something everything links against.  IIRC it's intentional that only libxul uses it.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #3)
> I don't think we want to include libmozalloc into something everything links
> against.  IIRC it's intentional that only libxul uses it.

Components use it as well.
On android, jemalloc is exposed through mozalloc, and everything links against that.
Assignee: nobody → mh+mozilla
Depends on: 678195
Depends on: 680373
Depends on: 680440
Attached patch imported patch (obsolete) — Splinter Review
This is WIP.
Attachment #554479 - Attachment is obsolete: true
Comment on attachment 555811 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>diff --git a/config/config.mk b/config/config.mk
>--- a/config/config.mk
>+++ b/config/config.mk
>@@ -233,23 +233,23 @@ endif # NS_TRACE_MALLOC
> 
> endif # MOZ_DEBUG
> 
> # We don't build a static CRT when building a custom CRT,
> # it appears to be broken. So don't link to jemalloc if
> # the Makefile wants static CRT linking.
> ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> # Disable default CRT libs and add the right lib path for the linker
>-MOZ_MEMORY_LDFLAGS=
>+MOZ_UTILS_LDFLAGS=
> endif

I wonder if any of this is relevant anymore ...

This actually looks completely reasonable.  I expected something much scarier ;-)
Attachment #555811 - Flags: review?(khuey) → review+
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #8)
> > # We don't build a static CRT when building a custom CRT,
> > # it appears to be broken. So don't link to jemalloc if
> > # the Makefile wants static CRT linking.
> > ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> > # Disable default CRT libs and add the right lib path for the linker
> >-MOZ_MEMORY_LDFLAGS=
> >+MOZ_UTILS_LDFLAGS=
> > endif
> 
> I wonder if any of this is relevant anymore ...

It still is, unfortunately. Some places where USE_STATIC_LIBS is defined may not actually require it anymore, though. But some definitely do (I tried a build without that hack. lots of fail)
Blocks: 414946
Refreshed against m-c, only context changes.
Attachment #555811 - Attachment is obsolete: true
Depends on: 683875
The following tinderbox win32 build is missing jemalloc.dll and will not start:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1314999564/

cset: http://hg.mozilla.org/mozilla-central/rev/e5815c156b6c

I assume this is the bug responsible.
The whole b-s merge backed out of m-c for causing bustage:
http://hg.mozilla.org/mozilla-central/rev/472716252ea3

https://tbpl.mozilla.org/?usebuildbot=1&rev=e5815c156b6c
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: fixed-in-bs
Target Milestone: mozilla9 → ---
A clobber would have fixed it. The problem is that our build system doesn't know everything needs to be relinked, since the libraries to link changed :(
Ok fair enough. Unfortunately at 4am when I'd already spent an hour sorting out the mess that was m-i, then retriggered a load on m-c and waited to see how that went, I didn't really have time to pick through and work out what had broken. I asked on #pymake but everyone else was away apart from Callek who agreed that just backing out the lot was the easiest solution.
The backout was backed out on b-s
Whiteboard: fixed-in-bs
Blocks: 685480
Depends on: 685554
Status: REOPENED → RESOLVED
Closed: 13 years ago13 years ago
Resolution: --- → FIXED
Worse still, the backout lost the rename from jemalloc.def to mozutils.def.in which meant that I lost my local changes to jemalloc.def :-(
Depends on: 689049
Depends on: 690856
Comment on attachment 557330 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>+  dnl On Windows, OSX and OS2, we want to link all our binaries against mozutils
>+  MOZ_UTILS_LDFLAGS='$(call EXPAND_LIBNAME_PATH,mozutils,$(LIBXUL_DIST)/lib)'
So, one of my Windows jemalloc builds has a problem. We set these flags that use two variables only known to the mozilla-central build system.

>     if grep -q '__imp__\{0,1\}free' crtdll.obj; then
But my crtdll.obj doesn not contain __imp__free...
>+      MOZ_UTILS_LDFLAGS='-LIBPATH:$(DIST)/lib -NODEFAULTLIB:msvcrt -NODEFAULTLIB:msvcrtd -NODEFAULTLIB:msvcprt -NODEFAULTLIB:msvcprtd -DEFAULTLIB:mozcrt'
So the flags don't get updated,

>     dnl Also pass this to NSPR/NSS
>+    DLLFLAGS="$DLLFLAGS $MOZ_UTILS_LDFLAGS"
And broken flags get passed to NSPR/NSS...
Depends on: 693286
Depends on: 695989
Depends on: 706042
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.