Build a mozutils library containing jemalloc and other things

RESOLVED FIXED

Status

defect
RESOLVED FIXED
8 years ago
a year ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

(Depends on 1 bug, Blocks 1 bug)

Trunk
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: fixed-in-bs)

Attachments

(1 attachment, 2 obsolete attachments)

Assignee

Description

8 years ago
From bug 662814 comment 26:

So, with the constraints from this API, our build system and our various ways of linking things for the various platforms we target, I think the least worse thing to do would be to have a directory where we'd build a libmozutils library that:
 - includes jemalloc when jemalloc is enabled
 - includes the event log infrastructure from this bug
 - includes what currently is libmozutils on android when building for android
 - is a DSO on win/osx and a static lib on linux (i.e., is whatever jemalloc currently is)

I think doing so as a separate patch (i.e. without the event log infrastructure) first would be better.
An advantage of this approach is that whenever we'll need to add something else that has the same kind of constraints, it will be much easier to handle.
Assignee

Comment 1

8 years ago
I'm wondering if we shouldn't fold libmozalloc in there as well.
If that works (I know libmozalloc is tricky), then yes, I'd say do it.
I don't think we want to include libmozalloc into something everything links against.  IIRC it's intentional that only libxul uses it.
Assignee

Comment 4

8 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #3)
> I don't think we want to include libmozalloc into something everything links
> against.  IIRC it's intentional that only libxul uses it.

Components use it as well.
On android, jemalloc is exposed through mozalloc, and everything links against that.
Assignee

Updated

8 years ago
Assignee: nobody → mh+mozilla
Assignee

Updated

8 years ago
Depends on: 678195
Assignee

Updated

8 years ago
Depends on: 680373
Assignee

Updated

8 years ago
Depends on: 680440
Assignee

Comment 6

8 years ago
Posted patch imported patch (obsolete) — Splinter Review
This is WIP.
Assignee

Updated

8 years ago
Attachment #554479 - Attachment is obsolete: true
Comment on attachment 555811 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>diff --git a/config/config.mk b/config/config.mk
>--- a/config/config.mk
>+++ b/config/config.mk
>@@ -233,23 +233,23 @@ endif # NS_TRACE_MALLOC
> 
> endif # MOZ_DEBUG
> 
> # We don't build a static CRT when building a custom CRT,
> # it appears to be broken. So don't link to jemalloc if
> # the Makefile wants static CRT linking.
> ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> # Disable default CRT libs and add the right lib path for the linker
>-MOZ_MEMORY_LDFLAGS=
>+MOZ_UTILS_LDFLAGS=
> endif

I wonder if any of this is relevant anymore ...

This actually looks completely reasonable.  I expected something much scarier ;-)
Attachment #555811 - Flags: review?(khuey) → review+
Assignee

Comment 9

8 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #8)
> > # We don't build a static CRT when building a custom CRT,
> > # it appears to be broken. So don't link to jemalloc if
> > # the Makefile wants static CRT linking.
> > ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> > # Disable default CRT libs and add the right lib path for the linker
> >-MOZ_MEMORY_LDFLAGS=
> >+MOZ_UTILS_LDFLAGS=
> > endif
> 
> I wonder if any of this is relevant anymore ...

It still is, unfortunately. Some places where USE_STATIC_LIBS is defined may not actually require it anymore, though. But some definitely do (I tried a build without that hack. lots of fail)

Updated

8 years ago
Blocks: 414946
Assignee

Comment 10

8 years ago
Refreshed against m-c, only context changes.
Assignee

Updated

8 years ago
Attachment #555811 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Depends on: 683875
http://hg.mozilla.org/mozilla-central/rev/e2697c06468f
http://hg.mozilla.org/mozilla-central/rev/8e01bc314326
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla9

Comment 13

8 years ago
The following tinderbox win32 build is missing jemalloc.dll and will not start:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1314999564/

cset: http://hg.mozilla.org/mozilla-central/rev/e5815c156b6c

I assume this is the bug responsible.
The whole b-s merge backed out of m-c for causing bustage:
http://hg.mozilla.org/mozilla-central/rev/472716252ea3

https://tbpl.mozilla.org/?usebuildbot=1&rev=e5815c156b6c
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: fixed-in-bs
Target Milestone: mozilla9 → ---
Assignee

Comment 15

8 years ago
A clobber would have fixed it. The problem is that our build system doesn't know everything needs to be relinked, since the libraries to link changed :(
Ok fair enough. Unfortunately at 4am when I'd already spent an hour sorting out the mess that was m-i, then retriggered a load on m-c and waited to see how that went, I didn't really have time to pick through and work out what had broken. I asked on #pymake but everyone else was away apart from Callek who agreed that just backing out the lot was the easiest solution.
Assignee

Comment 17

8 years ago
The backout was backed out on b-s
Whiteboard: fixed-in-bs
Assignee

Updated

8 years ago
Blocks: 685480

Updated

8 years ago
Depends on: 685554
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
Worse still, the backout lost the rename from jemalloc.def to mozutils.def.in which meant that I lost my local changes to jemalloc.def :-(
Depends on: 689049
Depends on: 690856
Comment on attachment 557330 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>+  dnl On Windows, OSX and OS2, we want to link all our binaries against mozutils
>+  MOZ_UTILS_LDFLAGS='$(call EXPAND_LIBNAME_PATH,mozutils,$(LIBXUL_DIST)/lib)'
So, one of my Windows jemalloc builds has a problem. We set these flags that use two variables only known to the mozilla-central build system.

>     if grep -q '__imp__\{0,1\}free' crtdll.obj; then
But my crtdll.obj doesn not contain __imp__free...
>+      MOZ_UTILS_LDFLAGS='-LIBPATH:$(DIST)/lib -NODEFAULTLIB:msvcrt -NODEFAULTLIB:msvcrtd -NODEFAULTLIB:msvcprt -NODEFAULTLIB:msvcprtd -DEFAULTLIB:mozcrt'
So the flags don't get updated,

>     dnl Also pass this to NSPR/NSS
>+    DLLFLAGS="$DLLFLAGS $MOZ_UTILS_LDFLAGS"
And broken flags get passed to NSPR/NSS...
Depends on: 693286
Assignee

Updated

8 years ago
Depends on: 695989

Updated

8 years ago
Depends on: 706042

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.