Build a mozutils library containing jemalloc and other things

RESOLVED FIXED

Status

()

Core
Build Config
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: fixed-in-bs)

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

6 years ago
From bug 662814 comment 26:

So, with the constraints from this API, our build system and our various ways of linking things for the various platforms we target, I think the least worse thing to do would be to have a directory where we'd build a libmozutils library that:
 - includes jemalloc when jemalloc is enabled
 - includes the event log infrastructure from this bug
 - includes what currently is libmozutils on android when building for android
 - is a DSO on win/osx and a static lib on linux (i.e., is whatever jemalloc currently is)

I think doing so as a separate patch (i.e. without the event log infrastructure) first would be better.
An advantage of this approach is that whenever we'll need to add something else that has the same kind of constraints, it will be much easier to handle.
(Assignee)

Comment 1

6 years ago
I'm wondering if we shouldn't fold libmozalloc in there as well.
If that works (I know libmozalloc is tricky), then yes, I'd say do it.
I don't think we want to include libmozalloc into something everything links against.  IIRC it's intentional that only libxul uses it.
(Assignee)

Comment 4

6 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #3)
> I don't think we want to include libmozalloc into something everything links
> against.  IIRC it's intentional that only libxul uses it.

Components use it as well.
On android, jemalloc is exposed through mozalloc, and everything links against that.
(Assignee)

Updated

6 years ago
Assignee: nobody → mh+mozilla
(Assignee)

Updated

6 years ago
Depends on: 678195
(Assignee)

Updated

6 years ago
Depends on: 680373
(Assignee)

Updated

6 years ago
Depends on: 680440
(Assignee)

Comment 6

6 years ago
Created attachment 554479 [details] [diff] [review]
imported patch

This is WIP.
(Assignee)

Comment 7

6 years ago
Created attachment 555811 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things
Attachment #555811 - Flags: review?(khuey)
(Assignee)

Updated

6 years ago
Attachment #554479 - Attachment is obsolete: true
Comment on attachment 555811 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>diff --git a/config/config.mk b/config/config.mk
>--- a/config/config.mk
>+++ b/config/config.mk
>@@ -233,23 +233,23 @@ endif # NS_TRACE_MALLOC
> 
> endif # MOZ_DEBUG
> 
> # We don't build a static CRT when building a custom CRT,
> # it appears to be broken. So don't link to jemalloc if
> # the Makefile wants static CRT linking.
> ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> # Disable default CRT libs and add the right lib path for the linker
>-MOZ_MEMORY_LDFLAGS=
>+MOZ_UTILS_LDFLAGS=
> endif

I wonder if any of this is relevant anymore ...

This actually looks completely reasonable.  I expected something much scarier ;-)
Attachment #555811 - Flags: review?(khuey) → review+
(Assignee)

Comment 9

6 years ago
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #8)
> > # We don't build a static CRT when building a custom CRT,
> > # it appears to be broken. So don't link to jemalloc if
> > # the Makefile wants static CRT linking.
> > ifeq ($(MOZ_MEMORY)_$(USE_STATIC_LIBS),1_1)
> > # Disable default CRT libs and add the right lib path for the linker
> >-MOZ_MEMORY_LDFLAGS=
> >+MOZ_UTILS_LDFLAGS=
> > endif
> 
> I wonder if any of this is relevant anymore ...

It still is, unfortunately. Some places where USE_STATIC_LIBS is defined may not actually require it anymore, though. But some definitely do (I tried a build without that hack. lots of fail)

Updated

6 years ago
Blocks: 414946
(Assignee)

Comment 10

6 years ago
Created attachment 557330 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

Refreshed against m-c, only context changes.
(Assignee)

Updated

6 years ago
Attachment #555811 - Attachment is obsolete: true
(Assignee)

Comment 11

6 years ago
http://hg.mozilla.org/projects/build-system/rev/e2697c06468f
http://hg.mozilla.org/projects/build-system/rev/8e01bc314326
Whiteboard: fixed-in-bs
(Assignee)

Updated

6 years ago
Depends on: 683875
http://hg.mozilla.org/mozilla-central/rev/e2697c06468f
http://hg.mozilla.org/mozilla-central/rev/8e01bc314326
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla9

Comment 13

6 years ago
The following tinderbox win32 build is missing jemalloc.dll and will not start:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1314999564/

cset: http://hg.mozilla.org/mozilla-central/rev/e5815c156b6c

I assume this is the bug responsible.
The whole b-s merge backed out of m-c for causing bustage:
http://hg.mozilla.org/mozilla-central/rev/472716252ea3

https://tbpl.mozilla.org/?usebuildbot=1&rev=e5815c156b6c
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: fixed-in-bs
Target Milestone: mozilla9 → ---
(Assignee)

Comment 15

6 years ago
A clobber would have fixed it. The problem is that our build system doesn't know everything needs to be relinked, since the libraries to link changed :(
Ok fair enough. Unfortunately at 4am when I'd already spent an hour sorting out the mess that was m-i, then retriggered a load on m-c and waited to see how that went, I didn't really have time to pick through and work out what had broken. I asked on #pymake but everyone else was away apart from Callek who agreed that just backing out the lot was the easiest solution.
(Assignee)

Comment 17

6 years ago
The backout was backed out on b-s
Whiteboard: fixed-in-bs
(Assignee)

Updated

6 years ago
Blocks: 685480

Updated

6 years ago
Depends on: 685554
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Worse still, the backout lost the rename from jemalloc.def to mozutils.def.in which meant that I lost my local changes to jemalloc.def :-(
Blocks: 688999
Depends on: 689049
Depends on: 690856
Depends on: 691876
Comment on attachment 557330 [details] [diff] [review]
Build a mozutils library containing jemalloc and other things

>+  dnl On Windows, OSX and OS2, we want to link all our binaries against mozutils
>+  MOZ_UTILS_LDFLAGS='$(call EXPAND_LIBNAME_PATH,mozutils,$(LIBXUL_DIST)/lib)'
So, one of my Windows jemalloc builds has a problem. We set these flags that use two variables only known to the mozilla-central build system.

>     if grep -q '__imp__\{0,1\}free' crtdll.obj; then
But my crtdll.obj doesn not contain __imp__free...
>+      MOZ_UTILS_LDFLAGS='-LIBPATH:$(DIST)/lib -NODEFAULTLIB:msvcrt -NODEFAULTLIB:msvcrtd -NODEFAULTLIB:msvcprt -NODEFAULTLIB:msvcprtd -DEFAULTLIB:mozcrt'
So the flags don't get updated,

>     dnl Also pass this to NSPR/NSS
>+    DLLFLAGS="$DLLFLAGS $MOZ_UTILS_LDFLAGS"
And broken flags get passed to NSPR/NSS...

Updated

6 years ago
Depends on: 693286
(Assignee)

Updated

6 years ago
Depends on: 695989

Updated

6 years ago
Depends on: 706042
You need to log in before you can comment on or make changes to this bug.