936115 - Explore TOR cross-compiler approach for ESR (and other builds?)

Status: RESOLVED DUPLICATE of bug 921040

(Firefox Build System :: General, defect)

Description

[u429623]

Spinning this bug out from bug 921040 comment 109 where Ray Donnelly describes the current approach the Tor project is taking to compiling darwin-on-linux for existing ESR releases, with the hopes that the Firefox approach can be similar.
This bug will explore the opportunities for convergence in more detail.

bug 921040 comment 109:
I've been doing some work with Mike Perry [1] and Georg Koppen [2] & [3] from the Tor Project who have been using my old toolchain4-derived Darwin targeting cross compilers to build OS X TBB Firefox ESR17.

In order to build ESR24, they needed to upgrade their compilers to Clang and as luck would have it I was working with Yann Diorcet on a crosstool-ng fork [4] whose main purpose being to merge those toolchain4 compilers and add Clang support.

This work is starting to come to shape, and large projects, including Firefox 24 ESR and Python 2.7.5 can now be built with it.

Please see the attached archive with a "build.sh" script that I hacked up which should:
 1. Work on a fresh Ubuntu 12.04.3-desktop-amd64 install (desktop-i386 has an issue that I will investigate further; I suspect it runs out of memory when linking XUL).
 2. apt-get install all dependencies.
 3. Download the Flosoft Mac OS X 10.6 SDK from launchpad [5]
 4. Build Darwin targeting crosstool-ng (full ld64, LTO, Clang 3.3 and GCC 4.2).
 5. Download and patch Firefox ESR24.
 6. Build it for OS X (i686).

There are three patches included:
 1. use-ACTRYCOMPILE-to-check-for-how-to-copy-vaargs.patch by Nathan Froyd, backported to ESR24.
 2. disable-MOZ_ENABLE_PROFILER_SPS.patch as suggested by Georg Koppen.
 3. export-AR-and-RANLIB-to-fix-xdarwin-libffi-build.patch by myself so that libtool doesn't use the build machine's ar and ranlib.

And also two hacks perpetrated:
 1. *Important:* Creates a /System/Library symlink in your Ubuntu root because some of the ESR24 build system hard codes finding frameworks in /System/Library/PrivateFrameworks.
 2. A symlink is also made from "x86_64-apple-darwin10-otool" to "otool" as the build system tries to run "otool".

Yann and I intend this toolchain to be as similar as possible to the ones that Apple provides. Our crosstool-ng fork can be built to target iOS. Yann added support for building Windows-targeting MinGW-w64 toolchains and is also fixing Clang issues on Windows. I added the possibility to use Windows and OS X as the host and build machine (canadian builds are a long term goal)

The attached "build.sh" script *may* run on OS X, given the correct environment (homebrew).

You can see a screenshot of FirefoxDebug running [6].

If you want to test Firefox, you can get it at [7]. A Debug build can be gotten from [8].

I also uploaded prebuilt cross compilers - host/build=Ubuntu 12.04.3 x86-64, target=Darwin i686/x86-64 [9]. These *should* be fully relocatable. If you want to build Firefox using my "build.sh" script without first building the toolchain then extract this archive to your $HOME folder first.

I would like you to consider this option for your Mac cross build system and would be more than willing to provide any assistance needed to make this happen.

[1] https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details
[2] https://trac.torproject.org/projects/tor/ticket/9711
[3] https://trac.torproject.org/projects/tor/ticket/9829
[4] https://github.com/diorcety/crosstool-ng/tree/cctools-llvm
[5] https://launchpad.net/~flosoft/+archive/cross-apple/+files/apple-uni-sdk-10.6_20110407.orig.tar.gz
[6] https://www.dropbox.com/s/98ly6o190rh96up/FirefoxDebug-ScreenShot.png
[7] https://www.dropbox.com/s/v6tsfz4u965kk7h/Firefox-darwin-i686.app-20131107-built-on-linux-gnu-x86_64.tar.bz2
[8] https://www.dropbox.com/s/dmz7e1cs2htmyh2/FirefoxDebug-darwin-i686.app-20131107-built-on-linux-gnu-x86_64.tar.bz2
[9] https://www.dropbox.com/s/2f52qrwucs2dlzm/cross-target-x86_64-apple-darwin10-host-x86_64-linux.tar.xz

Comment 1

[Michael Shal [:mshal]]

> Please see the attached archive with a "build.sh" script that I hacked up
> which should:
>  1. Work on a fresh Ubuntu 12.04.3-desktop-amd64 install (desktop-i386 has
> an issue that I will investigate further; I suspect it runs out of memory
> when linking XUL).
>  2. apt-get install all dependencies.
>  3. Download the Flosoft Mac OS X 10.6 SDK from launchpad [5]
>  4. Build Darwin targeting crosstool-ng (full ld64, LTO, Clang 3.3 and GCC
> 4.2).

I have a few questions here:

1) Can you explain why GCC is still needed? We've been able to get a cross-build of trunk using clang 3.3 + ld64, without GCC at all.

2) How different is your version of ld64 from the one froydnj put together? See here: 
https://github.com/froydnj/ld64

>  5. Download and patch Firefox ESR24.
>  6. Build it for OS X (i686).

Any particular reason for i686 instead of x86_64? I've been trying to target x86_64, so I'm just curious if there are some issues I should be aware of.

> And also two hacks perpetrated:
>  1. *Important:* Creates a /System/Library symlink in your Ubuntu root
> because some of the ESR24 build system hard codes finding frameworks in
> /System/Library/PrivateFrameworks.

See bug 933071 for how the PrivateFrameworks issue was handled on the trunk. Maybe that could be back-ported as well?

>  2. A symlink is also made from "x86_64-apple-darwin10-otool" to "otool" as
> the build system tries to run "otool".

See the patch in bug 921040 - froydnj added support for using objdump from binutils instead of otool: https://bugzilla.mozilla.org/attachment.cgi?id=825330

> 
> I would like you to consider this option for your Mac cross build system and
> would be more than willing to provide any assistance needed to make this
> happen.

Personally I would love to get the toolchain unified so that it meets all of our needs. I don't know what would be involved to get things officially included in ESR24, but we are actively trying to get support for cross-building OSX into the m-c trunk. Hopefully that will make things easier down the road.

Comment 2

[Nathan Froyd [:froydnj]]

(In reply to Michael Shal [:mshal] from comment #1)
> >  2. A symlink is also made from "x86_64-apple-darwin10-otool" to "otool" as
> > the build system tries to run "otool".
> 
> See the patch in bug 921040 - froydnj added support for using objdump from
> binutils instead of otool:
> https://bugzilla.mozilla.org/attachment.cgi?id=825330

I think if we have some solution that uses otool, that is very slightly better.

I also think libtool supports setting OTOOL in configure; we should probably try to support that in our build system if possible.  The symbolic link shouldn't be necessary.

Comment 3

[Ray Donnelly]

Attachment: Script config files and patches for building Mac OS X cross compiler and Firefox ESR24 (copied from [Bug 921040])

.. as per Hal Wine's request on https://bugzilla.mozilla.org/show_bug.cgi?id=921040#c112

Also, I will be updating this so that it can be used on other distros and OSes, and will change it to use a specific git sha1 reference for crosstool-ng as insurance against breakage.

Comment 4

[Ray Donnelly]

(In reply to Michael Shal [:mshal] from comment #1)
> > Please see the attached archive with a "build.sh" script that I hacked up
> > which should:
> >  1. Work on a fresh Ubuntu 12.04.3-desktop-amd64 install (desktop-i386 has
> > an issue that I will investigate further; I suspect it runs out of memory
> > when linking XUL).
> >  2. apt-get install all dependencies.
> >  3. Download the Flosoft Mac OS X 10.6 SDK from launchpad [5]
> >  4. Build Darwin targeting crosstool-ng (full ld64, LTO, Clang 3.3 and GCC
> > 4.2).
> 
> I have a few questions here:
> 
> 1) Can you explain why GCC is still needed? We've been able to get a
> cross-build of trunk using clang 3.3 + ld64, without GCC at all.

GCC is still needed because building it generates the "cpp" program and I've found other "cpp" programs can have a hard time with Apple's headers ("#import" for example). Using "clang -E" *may* be workable for this though, it is something I need to investigate. We will be adding compiler_rt and libc++ support soon too so that the Clang toolchain is more stand-alone.

> 
> 2) How different is your version of ld64 from the one froydnj put together?
> See here: 
> https://github.com/froydnj/ld64

I merged Apple's ld64 and cctools projects, then autoconfiscated the composite project (some credit goes to the old Open Darwin project for initiating this). At that time I also add LLVM support for LTO* and OpenSSL for encryption (to support signing / App Store submission - I've not tested this side of things yet). I ensured that it could still be built on Darwin and added support for building on Windows. Some old release notes for cctools+ld64 can be found at my old toolchain4 release page: http://code.google.com/p/mingw-and-ndk/downloads/detail?name=multiarch-darwin11-cctools127.2-gcc42-5666.3-llvmgcc42-2336.1-Linux-120724.tar.xz

* Nathan said that LTO was "non-essential functionality from the point of Firefox." but I think LTO can be a big win as it can generate faster executables. Also, I have a goal that if any procedure can detect that an executable wasn't made using Apple's official tools then I have failed so supporting everything that Apple supports is a requirement for me. Another advantage of LTO is that it allows edit-and-continue functionality - as soon as I can get some time I want to enable remote LLDB debugging and IDE integration (via QtCreator). This is probably less useful for the Firefox build/CI system, but then again, being able to log into the Linux build machine and debug a broken executable remotely may one day be a life saver.

> 
> >  5. Download and patch Firefox ESR24.
> >  6. Build it for OS X (i686).
> 
> Any particular reason for i686 instead of x86_64? I've been trying to target
> x86_64, so I'm just curious if there are some issues I should be aware of.

Only because I've not got round to trying x64_64 and, at a minimum, an i686 build can still be used on x86_64. As I am not au-fait with Firefox, can someone tell me whether passing "-arch i386 -arch x86_64" to the compiler be enough to get working fat binaries or if there are likely to be other issues? The stuff you do with "ar" could be problematic I guess?

> 
> > And also two hacks perpetrated:
> >  1. *Important:* Creates a /System/Library symlink in your Ubuntu root
> > because some of the ESR24 build system hard codes finding frameworks in
> > /System/Library/PrivateFrameworks.
> 
> See bug 933071 for how the PrivateFrameworks issue was handled on the trunk.
> Maybe that could be back-ported as well?

I will investigate this tonight, thanks Michael!

> I think if we have some solution that uses otool, that is very slightly better.

I agree. I would worry that some Apple specific information (or fat binary support) would be problematic unless going through otool. Also it helps with keeping the build as built-on-OSX-alike as possible.

> I also think libtool supports setting OTOOL in configure; we should probably try to support that in our build system if possible.  The symbolic link shouldn't be necessary.

Ditto, thanks Nathan!

> 
> > 
> > I would like you to consider this option for your Mac cross build system and
> > would be more than willing to provide any assistance needed to make this
> > happen.
> 
> Personally I would love to get the toolchain unified so that it meets all of
> our needs. I don't know what would be involved to get things officially
> included in ESR24, but we are actively trying to get support for
> cross-building OSX into the m-c trunk. Hopefully that will make things
> easier down the road.

When I update the build script / patches I will add the new version(s).

From your perspective, would you guys rather I target trunk now than ESR24? If so, is this the command to get the right source code:

hg clone https://hg.mozilla.org/mozilla-central

I will also attempt to cross compile for Darwin on Windows (via MSYS2) because I'm odd like that.

Comment 5

[Nathan Froyd [:froydnj]]

(In reply to Ray Donnelly from comment #4)
> GCC is still needed because building it generates the "cpp" program and I've
> found other "cpp" programs can have a hard time with Apple's headers
> ("#import" for example). Using "clang -E" *may* be workable for this though,
> it is something I need to investigate. We will be adding compiler_rt and
> libc++ support soon too so that the Clang toolchain is more stand-alone.

clang -E seems to work fine for "our" build.  I'd recommend using that instead.

> * Nathan said that LTO was "non-essential functionality from the point of
> Firefox." but I think LTO can be a big win as it can generate faster
> executables.

I totally agree with you here.  The reason I described it as "non-essential functionality" is that we currently don't do LTO on Macs.  Ergo, ld64 supporting LTO makes no difference to our current setup.  If we start supporting LTO builds on Mac, then compiling ld64 with LTO support would become an essential requirement.

> Only because I've not got round to trying x64_64 and, at a minimum, an i686
> build can still be used on x86_64. As I am not au-fait with Firefox, can
> someone tell me whether passing "-arch i386 -arch x86_64" to the compiler be
> enough to get working fat binaries or if there are likely to be other
> issues? The stuff you do with "ar" could be problematic I guess?

-arch i386 -arch x86_64 doesn't work, for a variety of reasons, the biggest one being that the word size of the target machine is currently hardcoded as a result of configure (HAVE_64BIT_OS).  You can imagine that doesn't work so well with a multiarch build.  I've been working on killing whatever bits still depend on this--at least insofar as those bits don't matter to a Mac build--but there are various things still hanging around.  bug 925167 and bug 417044 have some backstory here.

> > I think if we have some solution that uses otool, that is very slightly better.
> 
> I agree. I would worry that some Apple specific information (or fat binary
> support) would be problematic unless going through otool. Also it helps with
> keeping the build as built-on-OSX-alike as possible.

Yeah.  I am not sure whether binutils supports fat archives/binaries/etc.

> > Personally I would love to get the toolchain unified so that it meets all of
> > our needs. I don't know what would be involved to get things officially
> > included in ESR24, but we are actively trying to get support for
> > cross-building OSX into the m-c trunk. Hopefully that will make things
> > easier down the road.
> 
> When I update the build script / patches I will add the new version(s).
> 
> From your perspective, would you guys rather I target trunk now than ESR24?

Getting patches into trunk is much easier than getting patches into ESR24.  It would be easier for us to get your patches in for the next ESR release if they were against trunk.

> If so, is this the command to get the right source code:
> 
> hg clone https://hg.mozilla.org/mozilla-central

Yes.

Comment 6

[Ray Donnelly]

Thanks, I'll try a mozilla-central build now.

A quick note for anyone trying my script.

Twice today I've had the cctools build pause on me (last output "Patching 'cctools-809'") - I've never seen this before. If you run into this then chances are if you do:
"ps -aux | grep libtoolize"
.. you'll see something like:
ray      10838 96.8  0.0   9456  1140 pts/0    R+   21:24  36:50 /bin/bash /home/ray/ctng-build/.build/tools/bin/libtoolize --help
if you run (your equivalent of!):
"kill 10838"
.. then the build will continue. Despite a brief investigation I'm not sure why libtoolize is pausing at this point.

Comment 7

[Ray Donnelly]

> I think if we have some solution that uses otool, that is very slightly better.

Mercurial is taking it's time. I got bored so continued with ESR24 work in the meantime.

Thank you for the PrivateFrameworks patch I back-ported it and the build succeeded, that's the really nasty hack removed.

I am now adding a patch to support ${prefixed}otool. The autotooling side shouldn't be too difficult, but I was hoping someone could cast their eyes over some of the other usages to let me know if they are relevant to this cross compilation or not (so an ack-grep from ESR24):

http://pastebin.kde.org/p8eejfcy2

Comment 8

[Nathan Froyd [:froydnj]]

(In reply to Ray Donnelly from comment #7)
> Thank you for the PrivateFrameworks patch I back-ported it and the build
> succeeded, that's the really nasty hack removed.

Hooray!

> I am now adding a patch to support ${prefixed}otool. The autotooling side
> shouldn't be too difficult, but I was hoping someone could cast their eyes
> over some of the other usages to let me know if they are relevant to this
> cross compilation or not (so an ack-grep from ESR24):
> 
> http://pastebin.kde.org/p8eejfcy2

The toolkit/library/dependentlibs.py bit needs to be fixed to pick up OTOOL from the build config.

The tools/rb/fix_macos_stack.py bit can be left alone, as that is only ever going to run on a mac machine.

The gyp/test bits can be left alone for the same reasons.

AFAICS, the other bits in media/webrtc are either in unused scripts or only executed in xcode builds (which aren't relevant to us).  I know WebRTC compiles without changing any of those instances of otool, and I don't think those bits are executed in packaging, so I think we're in the clear.

Comment 9

[Ray Donnelly]

Would changing from:
proc = subprocess.Popen(['otool', '-l', lib], stdout = subprocess.PIPE)
.. to:
proc = subprocess.Popen([TOOLCHAIN_PREFIX + 'otool', '-l', lib], stdout = subprocess.PIPE)
.. be enough to make this work?

Comment 10

[Ray Donnelly]

I did this and also added TOOLCHAIN_PREFIX=x86_64-apple-darwin10- to my .mozconfig and it worked. I no longer need an "otool" symlink. The next target for removal is GCC's cpp.

I am considering setting up a github project for this script an patches in-case anyone wants to keep a closer eye on progress (or even contribute?)

Comment 11

[u429623]

(In reply to Nathan Froyd (:froydnj) from comment #2)
> (In reply to Michael Shal [:mshal] from comment #1)
> > >  2. A symlink is also made from "x86_64-apple-darwin10-otool" to "otool" as
> > > the build system tries to run "otool".
> > 
> > See the patch in bug 921040 - froydnj added support for using objdump from
> > binutils instead of otool:
> > https://bugzilla.mozilla.org/attachment.cgi?id=825330
> 
> I think if we have some solution that uses otool, that is very slightly
> better.

I'd vote the other way. If we don't need any executables from a mac, it is very slightly better. If we need otool, can we use the one from OpenDarwin?

Comment 12

[Ray Donnelly]

I think this is crossed wires.

If I'm not mistaken, Nathan was saying that a crossed (i.e. native Linux executable) otool is better than using objdump, and you are saying that you think that using maloader to run an Apple otool is worse than using a crossed otool. If so then I agree with both of you.

Our crosstool-ng toolchain comes with a fully working crossed otool (with a heritage the does go back to OpenDarwin in the mists of time)

Comment 13

[u429623]

(In reply to Ray Donnelly from comment #12)
> I think this is crossed wires.
> 
> If I'm not mistaken, Nathan was saying that a crossed (i.e. native Linux
> executable) otool is better than using objdump, and you are saying that you
> think that using maloader to run an Apple otool is worse than using a
> crossed otool. If so then I agree with both of you.
> 
> Our crosstool-ng toolchain comes with a fully working crossed otool (with a
> heritage the does go back to OpenDarwin in the mists of time)

Ray, you're 100% correct -- thanks for straightening me out -- I'd missed that everyone-but-I was referring to a crossed otool.

That leads into your comment #10:
> I am considering setting up a github project for this script an patches
> in-case anyone wants to keep a closer eye on progress (or even contribute?)

Where is "our crosstool-ng toolchain" source/patches located now? (I plead guilty to not having unpacked and examined the scripts.) Wouldn't the script be part of that repo? Certainly, having a project will make it easier to discuss alternatives, so it sounds good to me.

Comment 14

[Ray Donnelly]

> Where is "our crosstool-ng toolchain" source/patches located now? 
Our crosstool-ng project (cctools-llvm is the one we work in):
> [4] https://github.com/diorcety/crosstool-ng/tree/cctools-llvm
The OSX SDK:
> [5] https://launchpad.net/~flosoft/+archive/cross-apple/+files/apple-uni-sdk-10.6_20110407.orig.tar.gz (launchpad is having connection issues for me ATM)

> Wouldn't the script be part of that repo?
The script is just an engineer's tool purely for this bug report and my Tor work which builds the cross compilers and then uses them to build Firefox ESR24 so it doesn't belong with the crosstool-ng project. The script function that builds the compiler (note it copies a config file also) is:

build_cross_clang()
{
  if [ ! -d "${HOME}"/x-tools/${CROSSCC} ]; then
    [ -d "${HOME}"/src ] || mkdir "${HOME}"/src
    [ -d crosstool-ng ] || ( git clone [e-mail]:diorcety/crosstool-ng.git && cd crosstool-ng && git checkout -b cctools-llvm remotes/origin/cctools-llvm && cd - ) || ( echo "Failed to clone crosstool-ng" && exit 1 )
    cd crosstool-ng
    [ -d samples/gitian-${BITS} ] || mkdir -p samples/gitian-${BITS}
    cp "${THISDIR}"/tbb-${BITS}-crosstool.config samples/gitian-${BITS}/crosstool.config
    ./bootstrap && ./configure ${CTNG_CFG_ARGS} && make && make install
    PATH="${PATH}":"${ROOT}"/bin
    cd -
    [ -d ctng-build ] || mkdir ctng-build
    cd ctng-build
    ct-ng gitian-${BITS}
    ct-ng build
    cd -
  fi
}

.. cross compilers I built with this script can be tested at:
[9] https://www.dropbox.com/s/2f52qrwucs2dlzm/cross-target-x86_64-apple-darwin10-host-x86_64-linux.tar.xz

Comment 15

[Nathan Froyd [:froydnj]]

(In reply to Ray Donnelly from comment #9)
> Would changing from:
> proc = subprocess.Popen(['otool', '-l', lib], stdout = subprocess.PIPE)
> .. to:
> proc = subprocess.Popen([TOOLCHAIN_PREFIX + 'otool', '-l', lib], stdout =
> subprocess.PIPE)
> .. be enough to make this work?

That would be enough, but I don't think it's the right way to go.  We want to detect an OTOOL in configure, similar to how AR, RANLIB, STRIP, etc. work.  Then OTOOL should be used in dependentlibs.py.

Comment 16

[Ray Donnelly]

I think this should probably be closed now?

Was Firefox 28 for OS X built on Linux?

Comment 17

[u429623]

(In reply to Ray Donnelly from comment #16)
> I think this should probably be closed now?

afaik, the tooling has not been finalized. I'm no longer directly involved though.

> 
> Was Firefox 28 for OS X built on Linux?

No. We do not have a production ready solution yet -- linking this into the tracker (bug 927061), so it doesn't get lost in some of the churn.

Comment 18

[Georg Koppen]

(In reply to Nathan Froyd (:froydnj) from comment #15)
> That would be enough, but I don't think it's the right way to go.  We want
> to detect an OTOOL in configure, similar to how AR, RANLIB, STRIP, etc.
> work.  Then OTOOL should be used in dependentlibs.py.

This is bug 1067893 now.

Comment 19

[(not currently active) Ted Mielczarek]

I think we've covered all of this in bug 921040, so I'm going to dupe it over there. We have cross-mac builds running on every push on mozilla-{inbound,central} now. We're still only doing debug builds because I haven't done the work to try to make universal builds work (bug 1183613). I'm hoping we'll be able to turn off universal builds in favor of single-arch builds instead of having to do that work.