Closed Bug 854535 Opened 11 years ago Closed 6 years ago

'master' B2G builds fail to link libxul.so (arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 892575148 bytes for output file: Cannot allocate memory)

Categories

(Firefox OS Graveyard :: GonkIntegration, defect)

Product:
Firefox OS Graveyard
Component:
GonkIntegration
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: jwatt, Unassigned)

References

Details

(Whiteboard: [workaround:64bit toolchain in comment 21,22,26]) 

I'm guessing that mozilla-central libxul has now grown big enough that it can't be linked with the B2G tools. If I run:

BRANCH=master ANDROIDFS_DIR=$HOME/mozilla/android-backup ./config.sh unagi
./build.sh

the build fails with:

../../../prebuilt/darwin-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 892575148 bytes for output file: Cannot allocate memory
collect2: ld returned 1 exit status

I'm on OS X 10.8.3 with 16 GB of RAM (over 8 GB free when building) and with ccache set to have 5 GB of cache.
I started an earlier discussion here:

https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.b2g/ZTuNtPk2aho

but it doesn't seem like I'm doing anything obviously wrong, which is why I think we've hit a limit with 'master'.
Blocks: 853808
This bug is preventing Mac developers working on v.next features on mozilla-central from linking B2G.
FWIW I cannot reproduce this bug on my macbook air (OSX 10.7). I did a full build with the latest code from m-c and everything worked out.
Ehsan's busy at GDC this week but he runs a Mac so hopefully he can take a look next week.  Please let us know if you can't, Ehsan.  Thanks.
Flags: needinfo?(ehsan)
Is it easily possible to build our own android toolchain these days?  If yes, then we could potentially solve this by building a 64-bit toolchain.
Flags: needinfo?(ehsan)
(In reply to Michael Wu [:mwu] from comment #3)
> FWIW I cannot reproduce this bug on my macbook air (OSX 10.7). I did a full
> build with the latest code from m-c and everything worked out.

I'm on 10.8.

My .userconfig contains:

export CC=gcc-4.6 CXX=g++-4.6
export B2G_DEBUG=1
export B2G_NOOPT=1
export NOFTU=1
GECKO_PATH=/Volumes/case-sensitive-journaled/mozilla/trees/i-b2g
GECKO_OBJDIR=/Volumes/case-sensitive-journaled/mozilla/trees/obj/i-b2g-debug

Today I updated and rebuild by doing the following, and I still get the same error linking libxul:

cd B2G
rm -rf out
rm -rf /Volumes/case-sensitive-journaled/mozilla/trees/obj/i-b2g-debug
git pull
git checkout master
git show HEAD  # compare to https://github.com/mozilla-b2g/B2G/
BRANCH=master ./config.sh unagi
./build.sh

arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 892809028 bytes for output file: Cannot allocate memory
Oh, and I updated /Volumes/case-sensitive-journaled/mozilla/trees/i-b2g too, which is a clone of mozilla-inbound.
Can you try again, without:

export B2G_DEBUG=1
export B2G_NOOPT=1
export NOFTU=1
export B2G_DEBUG=1
export B2G_NOOPT=1

Removing these does indeed allow libxul to link. (Obviously developers will want/need to keep them though.)
FWIW, just B2G_NOOPT=1 is enough to prevent libxul from linking. :/
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #5)
> Is it easily possible to build our own android toolchain these days?  If
> yes, then we could potentially solve this by building a 64-bit toolchain.

It's not too hard. The build/tools/DEV-SCRIPTS-USAGE.txt file (section II) in the NDK describes how to download the relevant sources and rebuild the NDK.
Severity: normal → critical
Do we know how to fix this? I can't debug on trunk any more so this is a blocker for me.
I'm assuming the problem is that the linker is running out of address space.

I've modified the Linaro Android toolchain builder to be able to build a B2G ICS toolchain, as part of work on profiling.  I know it can host on x86_64 Linux; I haven't tried OS X, but it might be a way to get 64-bit toolchain.
https://github.com/jld/b2g-toolchain-prebuilt is the repo, if anyone wants to try to port it to Darwin.
Note that Fennec is running into the same problem while building with the stock Android NDK r8c. I'm building with --enable-debug and --disable-optimize (which presumably corresponds to the B2G_NOOPT=1 jwatt was talking about) and on a Mac libxul fails to link because:

ld(66719) malloc: *** mmap(size=1060532224) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
/Users/kats/android/android-ndk-r8c/toolchains/arm-linux-androideabi-4.6/prebuilt/darwin-x86/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 1060528980 bytes for output file: Cannot allocate memory
collect2: ld returned 1 exit status

In bug 859341 Adrian switched to using NDK r8e with the x86_64 toolchain which worked for him, so I might try that. I might also just turn off my --disable-optimize mozconfig option, but it seems like only a matter of time before libxul grows to the point where it will happen again.
Yeah, we need to find a solution here, and soon. Not being able to create a real (non-opt) debug build means debugging is much more difficult. We can't continue like this.
Severity: critical → blocker
Component: General → Builds
I'm running into this problem on Linux as well, when doing debug/no-opt builds. I could build an opt build fine but that was really annoying to debug so I tried the no-opt build and it failed.

.userconfig:
GECKO_PATH=/Users/kats/zspace/mozilla-git
GECKO_OBJDIR=/Users/kats/zspace/mozilla-git/obj-gonk
B2G_DEBUG=1
B2G_NOOPT=1

BRANCH=master ./config.sh emulator
./build.sh
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #19)
> I'm running into this problem on Linux as well, when doing debug/no-opt
> builds.

Interesting — there was an attempt to see if this was reproducible on Linux about a week ago, and the build succeeded.

(In reply to Jed Davis [:jld] from comment #14)
> I've modified the Linaro Android toolchain builder to be able to build a B2G
> ICS toolchain, as part of work on profiling.  I know it can host on x86_64
> Linux

Problem: it's actually building 32-bit GCC and binutils in that case (and I didn't notice; oops), but I think this is a simple matter of removing the "-m32" from relevant places.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #19)
> I'm running into this problem on Linux as well, when doing debug/no-opt
> builds.

This might work for x86_64 Linux:

Clone git://github.com/jld/b2g-toolchain-prebuilt.git somewhere
In .userconfig, set TARGET_TOOLS_PREFIX=/path/to/b2g-toolchain-prebuilt/toolchain-4.4.3/x86_64-linux-gnu/bin/arm-linux-androideabi-  (replace /path/to with the actual path)
./build.sh

The GCC is modified, but it just adds a new compiler flag and shouldn't behave any differently if that flag isn't explicily used.
Flags: needinfo?(bugmail.mozilla)
For anybody else who runs into this bug: when setting TARGET_TOOLS_PREFIX in the .userconfig file, you have to export it too (i.e. "export TARGET_TOOLS_PREFIX=/path/to/...") or it won't take effect. Building with this toolchain does in fact link libxul without any problems, even if B2G_DEBUG and B2G_NOOPT are set.
Flags: needinfo?(bugmail.mozilla)
(In reply to comment #22)
> For anybody else who runs into this bug: when setting TARGET_TOOLS_PREFIX in
> the .userconfig file, you have to export it too (i.e. "export
> TARGET_TOOLS_PREFIX=/path/to/...") or it won't take effect. Building with this
> toolchain does in fact link libxul without any problems, even if B2G_DEBUG and
> B2G_NOOPT are set.

If you have a build of this toolchain around, can you please host it somewhere?
It's the toolchain jld is hosting in github. See comment 21. His repo has the binaries for linux, no building required on your end if you want to use it.
(In reply to comment #24)
> It's the toolchain jld is hosting in github. See comment 21. His repo has the
> binaries for linux, no building required on your end if you want to use it.

Great, thanks!
Btw following jld's instructions and code I built a 64-bit toolchain for OS X that I successfully used to build debug/no-opt build of B2G emulator. The toolchain has been merged to jld's github repo in comment 21, just do a s/linux-gnu/apple-darwin/ on the TARGET_TOOLS_PREFIX.
This happens with B2G_NOOPT=1 on Linux as well.
Summary: 'master' B2G builds fail to link libxul.so on OS X (arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 892575148 bytes for output file: Cannot allocate memory) → 'master' B2G builds fail to link libxul.so (arm-linux-androideabi/bin/ld: fatal error: libxul.so: mmap: failed to allocate 892575148 bytes for output file: Cannot allocate memory)
Just a note. NOOPT does work on 4.3 builds for the nexus 4 on linux, not that it's much help since few people are using that platform at the moment.
(In reply to comment #28)
> Just a note. NOOPT does work on 4.3 builds for the nexus 4 on linux, not that
> it's much help since few people are using that platform at the moment.

Interesting!  Do you know what's specific about that build?
New toolchain I believe?
Still busted for me. Requesting 960-odd MB for mmap in ld fails with NOOPT, when building for Keon. Lin64 with about 18GB of free RAM. This is with the default toolchain.
I ran into this and I'm trying the 64-bit toolchain linked to in comment #21 now.

Meanwhile, a friend suggests that adding "-Wl,--no-keep-memory" to LDFLAGS may possibly solve the issue for 32-bit hosts; they ran into a similar issue building WebKit. They also found that the flag sped up linking on 64-bit machines with lots of RAM too, so perhaps we should try that?
Whiteboard: [workaround:64bit toolchain in comment 21,22,26]
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.