984515 - final linking of firefox takes huge amount of RAM due to bad factoring, breaks build

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[maillist-mozilla]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 (Beta/Release)
Build ID: 20140220020851

Steps to reproduce:

trying to compile firefox 27.0.1 from source on an ARM netbook with 512 MB RAM (linux) and 1 GB swap.



Actual results:

the build itself works fine (if we disregard the fact that it takes more than 2.5 days due to massive C++ header bloat), but as last step firefox invokes the linker and passes a list of roughly 3000 object files to link together. this step results in an OOM kill.

```
Executing: c++ -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wsign-compare -Wno-invalid-offsetof -D_GNU_SOURCE -I/src/build/firefox/mozilla-release/include -Os -g0 -fno-unwind-tables -fno-asynchronous-unwind-tables -Wa,--noexecstack -Wl,--no-keep-memory -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -std=gnu++0x -pthread -pipe -DNDEBUG -DTRIMMED -fomit-frame-pointer -fPIC -shared -Wl,-z,defs -Wl,-h,libxul.so -o libxul.so /src/build/firefox/mozilla-release/firefox-build-dir/toolkit/library/tmpGmY4c3.list -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -Wl,-rpath-link,/src/build/firefox/mozilla-release/firefox-build-dir/dist/bin -Wl,-rpath-link,//lib ../../dist/lib/libxpt.a -Wl,-version-script,symverscript -L../../dist/bin -L../../dist/lib ../../dist/lib/libjs_static.a -L/lib -lffi -L../../dist/bin -L../../dist/lib -lcrmf -lsmime3 -lssl3 -lnss3 -lnssutil3 -lXrender ../../dist/lib/libmozsqlite3.so -lz -L/lib -levent -L/lib -lpixman-1 -L/lib -lasound -L../../dist/bin -L../../dist/lib -L/src/build/firefox/mozilla-release/firefox-build-dir/dist/lib -lnspr4 -lplc4 -lplds4 ../../dist/lib/libmozalloc.so -lgtk-x11-2.0 -lgmodule-2.0 -pthread -latk-1.0 -lgio-2.0 -lpangoft2-1.0 -lfreetype -lfontconfig -lgdk-x11-2.0 -lpangocairo-1.0 -lpango-1.0 -lcairo -lgdk_pixbuf-2.0 -lgobject-2.0 -L/lib -lglib-2.0 -lX11 -lXext -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lgobject-2.0 -lglib-2.0 -L/lib -lcairo -L/lib -lfontconfig -lXt -lgthread-2.0 -L/lib -lfreetype -lrt 

/src/build/firefox/mozilla-release/firefox-build-dir/toolkit/library/tmpGmY4c3.list:
    INPUT("nsBidiUtils.o")
    INPUT("nsSpecialCasingData.o")
    INPUT("nsUnicharUtils.o")
    INPUT("nsUnicodeProperties.o")
[ insert list of 3000 object file names here ]
    INPUT("../../gfx/2d/SourceSurfaceCairo.o")
    INPUT("../../gfx/2d/SourceSurfaceRawData.o")

collect2: error: ld terminated with signal 9 [Killed]



Expected results:

it should not link every single object in a single step, but rather do it in steps, for example put every 100 objects in a separate .a(rchive). or make some DSO's. or do something else that makes it possible to link the stuff together.
btw i had to --disable-elf-hack, this seems incompatible with armv7l softfp.

Comment 1

[maillist-mozilla]

i was just informed that putting the objects into archive files or DSO's would not help.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

You need a system with more memory to link Firefox.

Comment 3

[Gregory Szorc [:gps]]

We recommend at least 8 GB of RAM to build Firefox.

Comment 4

[maillist-mozilla]

to this date, there is no single ARM system with 8GB RAM that i'm aware off, and even if there is one, it is aarch64, as 32bit systems cannot address more than 4GB.
so you're trying to sell me that building firefox is impossible on any existing ARM device ?

i'm changing this to WONTFIX since it appears that this is a real bug, and ppl here just dont want to fix it.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This is what cross compilers are for.

Comment 6

[Mike Hommey [:glandium]]

FWIW, Debian builds arm firefox on arm hardware, so this is definitely possible. It takes a lot of swap, a lot of time, and adding -Wl,--reduce-memory-overheads -Wl,--no-keep-memory to LDFLAGS, which you can do without changing anything to the build system.

Comment 7

[maillist-mozilla]

i already have -Wl,--no-keep-memory (see above) in my CFLAGS and CXXFLAGS (as LDFLAGS are ignored for the final link, which i found out after 2.5 days of building... but that should probably go into another bug report...). do you happen to know how much swap they added to make the link succeed on debian?

Comment 8

[maillist-mozilla]

i want to avoid crosscompiling, as the weird python build system made already enough trouble to get the build going on a native system, so i cant even imagine getting it to work for crosscompiling (see this patch https://github.com/sabotage-linux/sabotage/blob/master/KEEP/firefox-virtualenv.patch )

Comment 9

[Mike Hommey [:glandium]]

(In reply to maillist-mozilla from comment #7)
> i already have -Wl,--no-keep-memory (see above) in my CFLAGS and CXXFLAGS
> (as LDFLAGS are ignored for the final link, which i found out after 2.5 days
> of building... but that should probably go into another bug report...).

LDFLAGS is not ignored for the final link. If it is for you, it just means you didn't set it at the right time/place. It needs to be set when configure is invoked. Either in the environment or in the mozconfig file.

> do you happen to know how much swap they added to make the link succeed on
> debian?

No

Comment 10

[maillist-mozilla]

FYI: i built the gold linker from binutils 2.24, and using that, i could link the entire stuff together using less than 250 MB RAM. swap wasn't even touched.

now let's restart the build and wait for another 2.5 days to see if it finally runs through (this is my 7th attempt). i blame the "WTF" js engine C++ template bloat for these insane compile times. that library really deserves its name. yay code in headers.

Comment 11

[maillist-mozilla]

(In reply to Mike Hommey [:glandium] from comment #9)
> LDFLAGS is not ignored for the final link. If it is for you, it just means
> you didn't set it at the right time/place. It needs to be set when configure
> is invoked. Either in the environment or in the mozconfig file.

i passed it client.mk like this:

   LDFLAGS=-Wl,--no-keep-memory make -f client.mk

(i'm following the linux from scratch way of building firefox, which does not involve running configure)

Comment 12

[maillist-mozilla]

update regarding gold:
https://sourceware.org/bugzilla/show_bug.cgi?id=16728 (the generated libxul.so is broken)