Open Bug 1330608 Opened 7 years ago Updated 2 years ago

[meta] Add tor build chain to taskcluster

Categories

(Firefox Build System :: General, task)

Product:
Firefox Build System
Component:
General
Type:
task

Tracking

(Not tracked)

People

(Reporter: freddy, Assigned: tjr)

References

(Depends on 4 open bugs, Blocks 1 open bug)

Details

(Keywords: meta, Whiteboard: [tor])

Attachments

(1 file, 4 obsolete files)

Bug 1330608 Add the MinGW32 toolchain build to Taskcluster
59 bytes, text/x-review-board-request
glandium
: review+
Details 
This spun off of bug 1329593.
:gcp, :ted suggested we add the buildchain used by Tor Browser as a tier 1 (?) compile target to Taskcluster.

(Tor is compiling with mingw-cross to get reproducible builds, which seems to be an interesting topic for :arroway, CCing)
Adding :tjr as he is AFAICT currently concerned with setting up mingw-w64 build infrastructure to make it a tier 2 target.
We seem to break the mingw build fairly often, so if we're committing to support the Tor browser we should at a minimum ensure that their Windows builds work. I think getting these up and running in Taskcluster wouldn't be a huge effort, I stood up OS X cross-compiles a while ago (bug 935237) and most of the work was getting the toolchain and packaging sorted out, which I assume the Tor folks already have a handle on. The way we define Taskcluster tasks in the tree has changed a bit, but we have really good docs for how it works now:
http://gecko.readthedocs.io/en/latest/taskcluster/taskcluster/how-tos.html#hacking-task-graphs

The cross-mac build task definitions are here now:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/build/macosx.yml

Our Linux and Mac-cross builds build in a Docker container, defined here:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/desktop-build/Dockerfile

it's based on this base image:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build/Dockerfile
+ this one that just applies system updates (to keep the docker images smaller):
https://dxr.mozilla.org/mozilla-central/source/taskcluster/docker/centos6-build-upd/Dockerfile

For the mac-cross builds we didn't need too many additional dependencies, so I just added a handful of things to the desktop-build image and then put the rest in tooltool (cctools, the tools to build DMG files):
https://dxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/macosx64/cross-releng.manifest

If the tor mingw-cross builds need a lot of additional build tools we could create a new Docker image that derives from desktop-build and installs the additional build tools. If it's just a few things then we could use the desktop-build image and add things to it or put them in tooltool as appropriate. We tend to have toolchains in tooltool because we still have builds that don't run in Docker, but there's no real reason you'd have to use tooltool here, you could just bake the toolchain into the Docker image since these builds will only ever run in taskcluster.

I'd be happy to offer advice and hand-holding here if someone wants to take this, it should all be self-service, and you can test new build types on try.
I'll take this, but I will need some help along the way. =)

My intention, for Phase 1 at least, is to get a mingw build of FF going using a Docker container and into TaskCluster. It would be fantastic if we could promote this to Tier 1 (once we resolve the existing bugs that prevent compilation.) 

I'm following https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Cross_Compile_Mozilla_for_Mingw32 and encountering some errors, but have a few things to try yet.

(Phase N would add a FF that is compiled & preffed to behave similar to TBB and run it through our test suite, Phase M would add proxy bypass tests; and so on.)
Assignee: nobody → tom
It is unclear if the scope of this bug includes building in Tor's reproducible build environment. Either way, we should still track it against our reproducible build efforts.

FWIW, I'd like to move in the direction where the taskcluster taskgraph includes the build dependencies for Firefox. We could conceivably build all the build dependencies from source using a trusted base image then feed those into a trusted build image. Any time a dependency changes, it is recompiled from scratch and we get a new, deterministic build environment. This would essentially be recreating Tor's build environment but in a distributed execution environment. Bug 1313111 will likely be our first steps in this area.
Blocks: fx-reproducible-build
T(In reply to Gregory Szorc [:gps] from comment #4)
> It is unclear if the scope of this bug includes building in Tor's
> reproducible build environment. Either way, we should still track it against
> our reproducible build efforts.

The main goal here was to make sure we don't unknowingly break  building with mingw, thereby absolving the Tor people of needing to hunt and fix those regression. Making a reproducible build is a different, further step.
Yeah, I think the best bang for the buck here is just getting a mingw cross-compile stood up that is at least a close approximation of tor's build. We break the mingw build pretty frequently, so if they don't have to deal with that pain they'll be a lot better off. I haven't looked into the reproducible bits of their build, but those parts don't seem like things we're likely to break as often. If that turns out not to be true we could certainly stand up builds that use that machinery as a follow-up.
No longer depends on: 1344905
Depends on: 1344187
See Also: → 1355586
No longer blocks: 1332680
I think these patches have more to do with the build system, so I'm moving the bug.

(I'm not a huge fan of the Task Configuration component because its scope is so wide.)
Component: Task Configuration → Build Config
Product: Taskcluster → Core
I wouldn't mind moving and/or renaming that component :)
Would it be worth investigating cross-compiling with clang-cl instead of mingw? It seems like that would be less likely to break than the mingw builds and would also be useful for us more generally in being able to switch to cross compiling our release builds.
(In reply to Jeff Muizelaar [:jrmuizel] from comment #17)
> Would it be worth investigating cross-compiling with clang-cl instead of
> mingw? It seems like that would be less likely to break than the mingw
> builds and would also be useful for us more generally in being able to
> switch to cross compiling our release builds.

It would, and we did. :) We talked with the Chrome engineer who is driving that effort. That project (compiling chrome on Linux for Windows) is a hobby project really, and there are a lot of things left to be done (especially in the linking area). So we're keeping an eye on it, and asked him to keep us in the loop, but until it matures a bit (and then we invest in making it reproducible) Tor won't be able to switch over.
(In reply to Tom Ritter [:tjr] from comment #19)
> It would, and we did. :) We talked with the Chrome engineer who is driving
> that effort. That project (compiling chrome on Linux for Windows) is a hobby
> project really, and there are a lot of things left to be done (especially in
> the linking area). 

Is there a list somewhere of what these things all are. I'd perhaps be interested in contributing to that effort.
(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> (In reply to Tom Ritter [:tjr] from comment #19)
> > It would, and we did. :) We talked with the Chrome engineer who is driving
> > that effort. That project (compiling chrome on Linux for Windows) is a hobby
> > project really, and there are a lot of things left to be done (especially in
> > the linking area). 
> 
> Is there a list somewhere of what these things all are. I'd perhaps be
> interested in contributing to that effort.

Yup! I don't feel comfortable pasting emails into bugzilla, but I'll add you to a gdoc that has as much info as I have, and can add anyone else who is likewise interested.
Attachment #8873592 - Attachment is obsolete: true
Attachment #8881647 - Flags: review?(mh+mozilla)
Attachment #8873591 - Flags: review?(mh+mozilla)
Comment on attachment 8881647 [details]
Bug 1330608 Add the mingw32 build job (Work in Progress)

https://reviewboard.mozilla.org/r/152810/#review159228

::: taskcluster/scripts/misc/build-mingw-linux.sh:27
(Diff revision 8)
> +mkdir mingw-w64-toolchain
> +cd mingw-w64-toolchain
> +
> +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2
> +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2.sig
> +gpg --keyserver hkp://pool.sks-keyservers.net --recv-key EAF1C276A747E9ED86210CBAC3126D3B4AE55E93

It turns out that sometimes you can't get the keys from the keyserver in the pool. Fun.

My thinking is just pasting a giant public key blob at the top of this script (or in another file) and importing from there.
See build/unix/build-gcc/build-gcc.sh
You should actually reuse the script from there, as well as the keys. If you need keys that are not in the tree already, please use https://github.com/glandium/pgpstrip to make them smaller. Also see the commit message for bug 1335667, which succintly describes the steps to verify the keys themselves.
Comment on attachment 8881647 [details]
Bug 1330608 Add the mingw32 build job (Work in Progress)

https://reviewboard.mozilla.org/r/152810/#review159252

::: taskcluster/ci/toolchain/linux.yml:21
(Diff revision 10)
> +    run:
> +        using: toolchain-script
> +        script: build-mingw-linux.sh
> +        tooltool-downloads: public
> +        resources:
> +            - 'build/build-clang/**'

this field lists the (extra) scripts used by the build script, and you're not using build-clang files.

::: taskcluster/docker/desktop-build/Dockerfile:71
(Diff revision 10)
>  # Install the screen package here to use with xvfb.
>  # Install bison to build binutils.
>  RUN yum install -y bison screen
>  
> +# For the mingw toolchain
> +RUN yum install -y gmp-devel mpfr-devel libmpc-devel scons flex

See build/unix/build-gcc/build-gcc.sh, we shouldn't need gmp-devel, mpfr-devel or libmpc-devel.
I'm not sure why you need to add flex, but I suspect it should work without.
You don't need scons if you move building nsis to a separate job (which you should do ; please don't build multiple different tools in the same job)

(Also, touching this file has the annoying side effect of breaking unrelated jobs, like valgrind, thanks to the non-reproduceability of docker)

::: taskcluster/scripts/misc/build-mingw-linux.sh:25
(Diff revision 10)
> +mkdir -p $INSTALL_DIR
> +
> +mkdir mingw-w64-toolchain
> +cd mingw-w64-toolchain
> +
> +wget http://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2

It seems like you're mostly repeating what build/unix/build-gcc/build-gcc.sh does ; it feels like there should be some sharing of some sort between the scripts... or maybe modify the build-gcc script to allow to build the mingw-cross one.

::: taskcluster/scripts/misc/build-mingw-linux.sh:100
(Diff revision 10)
> +cd ..
> +wget https://downloads.sourceforge.net/project/nsis/NSIS%203/3.01/nsis-3.01-src.tar.bz2
> +echo "559703cc25f78697be1784a38d1d9a19c97d27a200dc9257d1483c028c6be9242cbcd10391ba618f88561c2ba57fdbd8b3607bea47ed8c3ad7509a6ae4075138  nsis-3.01-src.tar.bz2" | sha512sum -c -
> +bunzip2 nsis-3.01-src.tar.bz2
> +tar xf nsis-3.01-src.tar
> +cd nsis-3.01-src
> +# I don't know how to make the version work with the environment variables/config flags the way the author appears to
> +sed -i "s/'VERSION', 'Version of NSIS', cvs_version/'VERSION', 'Version of NSIS', '3.01'/" SConstruct
> +scons XGCC_W32_PREFIX=i686-w64-mingw32- ZLIB_W32=../zlib-1.2.11 SKIPUTILS="NSIS Menu" PREFIX=$INSTALL_DIR/ install
> +
> +cd ..
> +wget http://dl.winehq.org/wine/source/2.0/wine-2.0.1.tar.xz
> +wget http://dl.winehq.org/wine/source/2.0/wine-2.0.1.tar.xz.sign
> +gpg --keyserver hkp://pool.sks-keyservers.net --recv-key DA23579A74D4AD9AF9D3F945CEFAC8EAAF17519D
> +gpg --verify wine-2.0.1.tar.xz.sign
> +unxz wine-2.0.1.tar.xz
> +tar xf wine-2.0.1.tar
> +cd wine-2.0.1
> +./configure --prefix=$INSTALL_DIR/
> +make
> +make install
> +
> +cd ..
> +git clone https://github.com/tomrittervg/fxc2.git
> +cd fxc2
> +git checkout e00a32fbd3d5814b3f599c8528e37db9c1456803 # Asserts integrity of the clone (right?)
> +make
> +cp fxc2.exe $INSTALL_DIR/bin/

nsis, wine and fxc2 should go in separate jobs.

::: taskcluster/scripts/misc/build-mingw-linux.sh:128
(Diff revision 10)
> +git clone https://github.com/tomrittervg/fxc2.git
> +cd fxc2
> +git checkout e00a32fbd3d5814b3f599c8528e37db9c1456803 # Asserts integrity of the clone (right?)
> +make
> +cp fxc2.exe $INSTALL_DIR/bin/
> +cp d3dcompiler_47.dll $INSTALL_DIR/bin/

where does that file come from?
Attachment #8881647 - Flags: review?(mh+mozilla)
Comment on attachment 8873591 [details]
Bug 1330608 Add the mingw32 build job (Work in Progress)

https://reviewboard.mozilla.org/r/144966/#review159256

::: browser/config/mozconfigs/win32/mingw32:29
(Diff revision 20)
> +#    RUSTC
> +#    CARGO
> +. "$topsrcdir/browser/config/mozconfigs/common"
> +
> +# MinGW Stuff
> +export CROSS_COMPILE=1

Shouldn't be required.

::: browser/config/mozconfigs/win32/mingw32:30
(Diff revision 20)
> +#    CARGO
> +. "$topsrcdir/browser/config/mozconfigs/common"
> +
> +# MinGW Stuff
> +export CROSS_COMPILE=1
> +ac_add_options --enable-application=browser

That's the default.

::: browser/config/mozconfigs/win32/mingw32:33
(Diff revision 20)
> +# MinGW Stuff
> +export CROSS_COMPILE=1
> +ac_add_options --enable-application=browser
> +ac_add_options --target=i686-w64-mingw32
> +ac_add_options --with-toolchain-prefix=i686-w64-mingw32-
> +ac_add_options --enable-default-toolkit=cairo-windows

That /shouldn't/ be required.

::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:1
(Diff revision 20)
> +import os

would be nicer if this was marked as a copy of an existing mozharness config. If you are using mercurial, use hg copy ; if you are using git, ping me.
Attachment #8873591 - Flags: review?(mh+mozilla)
Attachment #8873591 - Attachment is obsolete: true
Uh oh. I accidentally committed one of the patches but not the other and screwed mozreview up something bad.  inter-diffs are going to be a mess, and the issues opened may not match the commit....




(In reply to Mike Hommey [:glandium] from comment #52)
> this field lists the (extra) scripts used by the build script, and you're
> not using build-clang files.

Fixed.


> See build/unix/build-gcc/build-gcc.sh, we shouldn't need gmp-devel,
> mpfr-devel or libmpc-devel.
> I'm not sure why you need to add flex, but I suspect it should work without.
> You don't need scons if you move building nsis to a separate job (which you
> should do ; please don't build multiple different tools in the same job)
> 
> (Also, touching this file has the annoying side effect of breaking unrelated
> jobs, like valgrind, thanks to the non-reproduceability of docker)

scons is needed for nsis; and flex for wine.

I can drop gmp-devel, mpfr-devel, and libmpc-devel - but should I leave scons and flex here? Or should I use some alternate method to supply them for those build tasks?


> It seems like you're mostly repeating what build/unix/build-gcc/build-gcc.sh
> does ; it feels like there should be some sharing of some sort between the
> scripts... or maybe modify the build-gcc script to allow to build the
> mingw-cross one.

I have edited and rearchitected this script. I'm not sure you'll like the result - I'm open to doing it how you would prefer it, just let me know.



> nsis, wine and fxc2 should go in separate jobs.

Done


> > +cp d3dcompiler_47.dll $INSTALL_DIR/bin/
> 
> where does that file come from?

My local machine. But it's signed by Microsoft?
All of these issues seem to have disappears from mozreview. =(


(In reply to Mike Hommey [:glandium] from comment #53)
> > +export CROSS_COMPILE=1
> 
> Shouldn't be required.

Removed.


> > +ac_add_options --enable-application=browser
> 
> That's the default.

Removed.


> > +ac_add_options --enable-default-toolkit=cairo-windows
> 
> That /shouldn't/ be required.

Removed.


> :::
> testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:1
> (Diff revision 20)
> > +import os
> 
> would be nicer if this was marked as a copy of an existing mozharness
> config. If you are using mercurial, use hg copy ; if you are using git, ping
> me.

I tried to do this... I'm not sure it took after I histedit+rolled though.
Flags: needinfo?(mh+mozilla)
Depends on: 1389965
Depends on: 1389967
Flags: needinfo?(mh+mozilla)
Attachment #8881647 - Flags: review?(mh+mozilla)
Attachment #8881647 - Attachment is obsolete: true
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review175532

::: taskcluster/ci/toolchain/linux.yml:264
(Diff revision 15)
> +        using: toolchain-script
> +        script: build-mingw32-nsis.sh
> +        tooltool-downloads: public
> +        resources:
> +            - 'taskcluster/scripts/misc/tooltool-download.sh'
> +            - 'browser/config/tooltool-manifests/linux64/mingw32-components.manifest'

See https://bugzilla.mozilla.org/show_bug.cgi?id=1391744
Attachment #8884418 - Flags: review?(mh+mozilla)
Okay, I think I have the toolchain half of this ready for review. There is some weirdness in the 'resources' section of the yml due to Bug 1391744, but other than that I think it's in good shape.
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review175970

::: taskcluster/ci/toolchain/linux.yml:265
(Diff revision 15)
> +        script: build-mingw32-nsis.sh
> +        tooltool-downloads: public
> +        resources:
> +            - 'taskcluster/scripts/misc/tooltool-download.sh'
> +            - 'browser/config/tooltool-manifests/linux64/mingw32-components.manifest'
> +        toolchain-artifact: public/build/nsis.tar.xz

Can I specify 'toolchains' here (like in taskcluster/build/ci/linux.yml) to indicate this toolchain relies on another toolchain?
Depends on: 1392604
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review177704

::: browser/config/tooltool-manifests/linux64/mingw32-components.manifest:3
(Diff revision 17)
> +[
> +  {
> +    "version": "mingw32: gcc 5.1.0, binutils 2.27, mingw-w64 9c7c712240",

why are you using a tooltool manifest for mingw gcc when you build one already?

::: build/unix/build-gcc/build-gcc.sh:100
(Diff revision 17)
>  }
>  
>  build_binutils() {
>    mkdir $root_dir/binutils-objdir
>    pushd $root_dir/binutils-objdir
> -  # gold is disabled because we don't use it on automation, and also we ran into
> +  ../binutils-$binutils_version/configure --prefix /tools/gcc/ $binutils_configure_flags

I'm not a big fan of duplicating the binutils and gcc configure options in the various scripts that source this one. How about leaving the flags if none are passed in, and replace them otherwise?

::: build/unix/build-gcc/build-gcc.sh:109
(Diff revision 17)
>  }
>  
>  build_gcc() {
>    mkdir $root_dir/gcc-objdir
>    pushd $root_dir/gcc-objdir
> -  ../gcc-$gcc_version/configure --prefix=/tools/gcc --enable-languages=c,c++  --disable-nls --disable-gnu-unique-object --enable-__cxa_atexit --with-arch-32=pentiumpro
> +  ../gcc-$gcc_version/configure --prefix=/tools/gcc $gcc_configure_flags

Note, you're not using build_gcc...

::: build/unix/build-gcc/build-gcc.sh:119
(Diff revision 17)
>    tar caf $root_dir/gcc.tar.xz gcc/
>    popd
>  }
> +
> +build_gcc_and_mingw() {
> +  export install_dir=$root_dir/tools/gcc/

Having /home/worker/workspace/build in the install dir is suspicious.

::: build/unix/build-gcc/build-gcc.sh:125
(Diff revision 17)
> +  mkdir -p $install_dir
> +
> +  export PATH=$PATH:$install_dir/bin/
> +
> +  cd $root_dir
> +  git clone git://git.code.sf.net/p/mingw-w64/mingw-w64

git clone -n

::: build/unix/build-gcc/build-gcc.sh:137
(Diff revision 17)
> +  ../gcc-$gcc_version/configure --prefix=$install_dir $gcc_configure_flags
> +  make $make_flags all-gcc
> +  make $make_flags install-gcc
> +  popd
> +
> +  cd $root_dir

you don't need to re-cd each time, popd brings you back there.

::: taskcluster/ci/toolchain/linux.yml:215
(Diff revision 17)
> +linux64-mingw32-gcc:
> +    description: "GCC toolchain build for MinGW32 Cross Compile"
> +    treeherder:
> +        kind: build
> +        platform: toolchains/opt
> +        symbol: TL(mingw32-gcc)

It feels like it would be better to have these jobs in a separate group than TL.

I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this.

::: taskcluster/ci/toolchain/linux.yml:216
(Diff revision 17)
> +    description: "GCC toolchain build for MinGW32 Cross Compile"
> +    treeherder:
> +        kind: build
> +        platform: toolchains/opt
> +        symbol: TL(mingw32-gcc)
> +        tier: 1

This and the other jobs surely are not tier 1.

::: taskcluster/ci/toolchain/linux.yml:220
(Diff revision 17)
> +        symbol: TL(mingw32-gcc)
> +        tier: 1
> +    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
> +    worker:
> +        docker-image: {in-tree: desktop-build}
> +        max-run-time: 36000

it would probably be a good idea to find more reasonable max run times for these jobs, instead of cargo culting the too long values we're using on other jobs :)

::: taskcluster/ci/toolchain/linux.yml:225
(Diff revision 17)
> +        max-run-time: 36000
> +    run:
> +        using: toolchain-script
> +        script: build-gcc-mingw32.sh
> +        resources:
> +            - 'build/unix/build-gcc/**'

Please don't use wildcards. See 9a82fd89dff95e2a87f7d3818179b9218e1feb1b.

::: taskcluster/ci/toolchain/linux.yml:233
(Diff revision 17)
> +linux64-mingw32-wine:
> +    description: "Wine build for MinGW32 Cross Compile"
> +    treeherder:
> +        kind: build
> +        platform: toolchains/opt
> +        symbol: TL(mingw32-wine)

This is just wine, right?

::: taskcluster/scripts/misc/build-gcc-4.9-linux.sh:26
(Diff revision 17)
> +$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key
> +$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key
> +$GPG --import $data_dir/80F98B2E0DAB6C8281BDF541A7C8C3B2F71EDF1C.key
> +$GPG --import $data_dir/90AA470469D3965A87A5DCB494D03953902C9419.key
> +$GPG --import $data_dir/B215C1633BCA0477615F1B35A5B3A004745C015A.key
> +$GPG --import $data_dir/B3C42148A44E6983B3E4CC0793FA9B1AB75C61B8.key

Why are these keys suddenly necessary?

::: taskcluster/scripts/misc/build-gcc-mingw32.sh:24
(Diff revision 17)
> +$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
> +$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key
> +$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key
> +$GPG --import $data_dir/80F98B2E0DAB6C8281BDF541A7C8C3B2F71EDF1C.key
> +$GPG --import $data_dir/90AA470469D3965A87A5DCB494D03953902C9419.key
> +$GPG --import $data_dir/B215C1633BCA0477615F1B35A5B3A004745C015A.key
> +$GPG --import $data_dir/B3C42148A44E6983B3E4CC0793FA9B1AB75C61B8.key

I doubt you need more than one here.

::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:20
(Diff revision 17)
> +
> +cd $WORKSPACE
> +
> +# --------------
> +
> +git clone https://github.com/tomrittervg/fxc2.git

git clone -n

::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:30
(Diff revision 17)
> +cp d3dcompiler_47.dll $INSTALL_DIR/bin/
> +
> +# --------------
> +
> +cd $WORKSPACE
> +tar cf fxc2.tar gcc

The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory.

You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it.

::: taskcluster/scripts/misc/build-mingw32-nsis.sh:37
(Diff revision 17)
> +export PATH="$TOOLTOOL_DIR/gcc-mingw/bin:$TOOLTOOL_DIR/gcc-mingw/i686-w64-mingw32/bin:$PATH"
> +mkdir -p $INSTALL_DIR
> +
> +cd $WORKSPACE
> +
> +echo "-----BEGIN PGP PUBLIC KEY BLOCK-----

gpg --import <<EOF
-----BEGIN PGP PUBLIC KEY BLOCK-----
...
-----END PGP PUBLIC KEY BLOCK-----
EOF

::: taskcluster/scripts/misc/build-mingw32-nsis.sh:78
(Diff revision 17)
> +wget http://zlib.net/zlib-1.2.11.tar.gz
> +wget http://zlib.net/zlib-1.2.11.tar.gz.asc
> +gpg --verify zlib-1.2.11.tar.gz.asc

why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file)

::: taskcluster/scripts/misc/build-mingw32-nsis.sh:81
(Diff revision 17)
> +gunzip zlib-1.2.11.tar.gz
> +tar xf zlib-1.2.11.tar

tar xaf

::: taskcluster/scripts/misc/build-mingw32-nsis.sh:89
(Diff revision 17)
> +bunzip2 nsis-3.01-src.tar.bz2
> +tar xf nsis-3.01-src.tar

tar xaf
Attachment #8884418 - Flags: review?(mh+mozilla)
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review177704

> why are you using a tooltool manifest for mingw gcc when you build one already?

This manifest is used to build fxc2 and nsis with MinGW, so they need the mingw toolchain to run their job. If I can make one toolchain have a dependency on another toolchain I can eliminate this.

> It feels like it would be better to have these jobs in a separate group than TL.
> 
> I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this.

Sure, I can put them wherever you like. What group should be in/symbol should I use?

> This is just wine, right?

Yup. I could yum install it into the image but I don't think we want wine in all our images....

> Why are these keys suddenly necessary?

I added them because it seemed prudent to be futureproof. If a new gcc release is signed by one of the other keys (which gcc says they may be) our build will suddenly break. It may not be that difficult to figure it out and fix it, but... it seemed more correct to accept any of the keys gcc says the release may be signed by, instead of just the one that seems most commonly used.
(In reply to Tom Ritter [:tjr] from comment #90)
> If I can make one toolchain have a dependency on another toolchain

You can. See how e.g. sccaches uses clang or clang uses gcc.
 
> > It feels like it would be better to have these jobs in a separate group than TL.
> > 
> > I'm also tempted to tell you to move them in a separate yml file, but I'm not completely sold on this.
> 
> Sure, I can put them wherever you like. What group should be in/symbol
> should I use?

Something that begins with T, and is not TL, TM or TW :)
TMW, maybe?

> > This is just wine, right?
> 
> Yup. I could yum install it into the image but I don't think we want wine in
> all our images....

I should have been more explicit: I meant "mingw32" in the name is not relevant.

> > Why are these keys suddenly necessary?
> 
> I added them because it seemed prudent to be futureproof. If a new gcc
> release is signed by one of the other keys (which gcc says they may be) our
> build will suddenly break. It may not be that difficult to figure it out and
> fix it, but... it seemed more correct to accept any of the keys gcc says the
> release may be signed by, instead of just the one that seems most commonly
> used.

Only two of those keys so far have been used to sign GCC tarballs we've used. I'd rather not add keys unconditionally if we don't need them. I doubt the other will be used to sign newer GCC releases anyways. And if they are, we can deal with that then ; we don't have to deal with it now. The only thing that can break is if *existing* releases are *resigned* with a different key, and I doubt that will happen. And if it does, it's better if we notice.
No longer depends on: 1393795
No longer depends on: 1393897
No longer depends on: 1393896
I know you already spent a lot of time on this, but before you spend some more time on it, a question came to me while under the shower: why exactly are we building mingw-gcc, wine, and nsis from source? Taking a step back, the main reason we build our own gcc and clang is that our base build docker image (mainly for binary compatibility reasons) is old. But we don't have to use the same docker image. So why not use, say, a docker image for debian stretch, freeze it on snapshot.debian.org and install the packages we need? nalexander did something like this for android, and could point you to the Dockerfile he's been using.
(I really should have thought of this before you went down this road in the first place :( ... hindsight is 20/20)
(In reply to Mike Hommey [:glandium] from comment #92)
> I know you already spent a lot of time on this, but before you spend some
> more time on it, a question came to me while under the shower: why exactly
> are we building mingw-gcc, wine, and nsis from source? Taking a step back,
> the main reason we build our own gcc and clang is that our base build docker
> image (mainly for binary compatibility reasons) is old. But we don't have to
> use the same docker image. So why not use, say, a docker image for debian
> stretch, freeze it on snapshot.debian.org and install the packages we need?
> nalexander did something like this for android, and could point you to the
> Dockerfile he's been using.

That could work for nsis and wine I suppose (and upx, which we need also but isn't available via the yum repos we are using)...

But we follow mingw-w64 master very closely. Jacek has upstreamed at least half a dozen patches for me over the course of this project. Relying on their release schedule wouldn't work.  Also, I like the idea of retaining the ability to apply our own patches on top of it, regardless of upstream status. (I think you might have mentioned that recently on a dev.platform thread...)
I'm splitting the warnings (that aren't compiler errors) out into Bug 1394433 to keep this one a little more manageable. (Should have done that in the beginning.)
No longer depends on: 1393196, 1393212, 1393487, 1393498, 1393535, 1394005, 1394007, 1394013, 1377605, 1377617, 1377948, 1377951, 1392594, 1392609, 1392617, 1392618, 1392626, 1392630, 1392641, 1392642, 1392677, 1392684, 1392688, 1393201, 1393216, 1393222, 1393528, 1393536, 1393538, 1393539, 1394002
See Also: → 1394433
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review177704

Okay I addressed all of the issues except for two (with comments below). You reviewed revision 17 if you want to diff to latest rev. (I may have a couple of typo-fixes post-18 because TC is busted right now and I can' get my jobs to complete.)

Also - will you review the TaskCluster build job, or should I find someone else to review that?

> The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory.
> 
> You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it.

I fixed the tar acf thing; but there is a problem with stem name issue.

When we build with winpthreads for MinGW, we need libwinpthread-1.dll to be in the same directory as the executables when we run them. This affects fxc2 for sure and I believe nsis also. The winpthread dll comes out of the gcc build. (Which occurs first.) I'd either need to do a post-tooltool step on the browser build machine to copy this dll out to those two directories (and I'm not sure where I'd shim that step in or how) - or I just build them, install them, and extract them out to the same gcc directory as the mingw compiler.

> why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file)

I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use build-gcc.sh directly or abstract the functions to a separate file, I'll be including across vastly different directory hierarchies. Is that alright?  (It's not that complicated of logic to duplicate...)
New tooltool manifest for rust 1.20.0 from bug 1396883.

{
  "algorithm": "sha512",
  "visibility": "public",
  "filename": "rustc-x86_64-unknown-linux-gnu-mingw32-cross-repack.tar.xz",
  "version": "rustc 1.20.0 (f3d6973f4 2017-08-27) repack",
  "unpack": true,
  "digest": "9f1af5cf4febda716f5d89e9de2482814d62c456637fbe235469225f2cfdb3401cfc1b955a7225d17600332c88b093bccbef6f702f106d117bbb23786ea3fcbd",
  "size": 211950648
}
(In reply to Tom Ritter [:tjr] from comment #98)
> Comment on attachment 8884418 [details]
> Bug 1330608 Add the MinGW32 toolchain build to Taskcluster
> 
> https://reviewboard.mozilla.org/r/155334/#review177704
> 
> Okay I addressed all of the issues except for two (with comments below). You
> reviewed revision 17 if you want to diff to latest rev. (I may have a couple
> of typo-fixes post-18 because TC is busted right now and I can' get my jobs
> to complete.)
> 
> Also - will you review the TaskCluster build job, or should I find someone
> else to review that?
> 
> > The implicit contract is that toolchain artifacts extract to a directory with the stem name of the archive. That is, unpacking foo.tar.xz should create a foo directory.
> > 
> > You should also be able to use tar -acf foo.tar.xz instead of first create a tar and then xz it.
> 
> I fixed the tar acf thing; but there is a problem with stem name issue.
> 
> When we build with winpthreads for MinGW, we need libwinpthread-1.dll to be
> in the same directory as the executables when we run them. This affects fxc2
> for sure and I believe nsis also. The winpthread dll comes out of the gcc
> build. (Which occurs first.) I'd either need to do a post-tooltool step on
> the browser build machine to copy this dll out to those two directories (and
> I'm not sure where I'd shim that step in or how) - or I just build them,
> install them, and extract them out to the same gcc directory as the mingw
> compiler.

If fxc2 needs the winpthreads dll, copy the winpthreads dll in the fxc2 package.

> > why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file)
> 
> I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and
> build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use
> build-gcc.sh directly or abstract the functions to a separate file, I'll be
> including across vastly different directory hierarchies. Is that alright? 
> (It's not that complicated of logic to duplicate...)

build-gcc.sh is already included from taskcluster/script/misc/ scripts.
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review183546

::: build/unix/build-gcc/build-gcc.sh:127
(Diff revision 19)
>    tar caf $root_dir/gcc.tar.xz gcc/
>    popd
>  }
> +
> +build_gcc_and_mingw() {
> +  export install_dir=$root_dir/tools/gcc/

Having /home/worker/workspace/build in the install dir is still suspicious.

::: taskcluster/ci/toolchain/linux.yml:224
(Diff revision 19)
> +        docker-image: {in-tree: desktop-build}
> +        max-run-time: 3600
> +    run:
> +        using: toolchain-script
> +        script: build-upx.sh
> +        tooltool-downloads: public

no need for this when you don't use a tooltool manifest.

::: taskcluster/scripts/misc/build-gcc-mingw32.sh:15
(Diff revision 19)
> +root_dir=$HOME_DIR
> +data_dir=$HOME_DIR/src/build/unix/build-gcc
> +
> +. $data_dir/build-gcc.sh
> +
> +gcc_version=5.4.0

Now that linux builds are using gcc 6, can that be updated?

::: taskcluster/scripts/misc/build-gcc-mingw32.sh:43
(Diff revision 19)
> +7e3c02ff52f8540f6a85534f54158968417fd676001651c8289c705bd0228f36  isl-0.14.tar.bz2
> +ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7  mpc-0.8.2.tar.gz
> +ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658  mpfr-3.1.5.tar.bz2
> +EOF
> +
> +#5f835b04b5f7dd4f4d2dc96190ec1621b8d89f2dc6f638f9f8bc1b1014ba8cad  gcc-5.2.0.tar.bz2

Please remove those commented-out lines.

::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:20
(Diff revision 19)
> +
> +cd $WORKSPACE
> +
> +# --------------
> +
> +git clone -n https://github.com/tomrittervg/fxc2.git

maybe move this under the mozilla/ group on github?

::: taskcluster/scripts/misc/build-mingw32-fxc2.sh:33
(Diff revision 19)
> +cp d3dcompiler_47.dll $INSTALL_DIR/i686-w64-mingw32/bin
> +
> +# --------------
> +
> +cd $WORKSPACE
> +tar caf fxc2.tar.xz gcc

the stem name problem still applies.
Attachment #8884418 - Flags: review?(mh+mozilla)
Comment on attachment 8899589 [details]
Bug 1330608 Add the MinGW32 browser build to Taskcluster

https://reviewboard.mozilla.org/r/170882/#review183550

You'll probably want some releng people to review the mozharness config.

::: browser/config/tooltool-manifests/mingw32/releng.manifest:6
(Diff revision 4)
> +[
> +  {
> +    "size": 11189216,
> +    "digest": "18bc52b0599b1308b667e282abb45f47597bfc98a5140cfcab8da71dacf89dd76d0dee22a04ce26fe7ad1f04e2d6596991f9e5b01fd2aaaab5542965f596b0e6",
> +    "algorithm": "sha512",
> +    "filename": "gtk3.tar.xz",

I doubt you need this.

::: taskcluster/ci/build/windows.yml:514
(Diff revision 4)
> +        platform: windows-mingw32-32/debug
> +        symbol: tc(B)
> +        tier: 2
> +    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
> +    worker:
> +        max-run-time: 72000

that's a lot.

::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:114
(Diff revision 4)
> +        'yasm', 'wget',
> +        'mpfr',  # required for system compiler
> +        'xorg-x11-font*',  # fonts required for PGO
> +        'imake',  # required for makedepend!?!
> +        ### <-- from releng repo
> +        'gcc45_0moz3', 'gcc454_0moz1', 'gcc472_0moz1', 'gcc473_0moz1',

you can remove this line.

::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:119
(Diff revision 4)
> +        'gcc45_0moz3', 'gcc454_0moz1', 'gcc472_0moz1', 'gcc473_0moz1',
> +        'yasm', 'ccache',
> +        ###
> +        'valgrind',
> +        ######## 32 bit specific ###########
> +        'glibc-static.i686', 'libstdc++-static.i686',

You probably can remove all those .i686 packages.

::: testing/mozharness/configs/builds/releng_base_windows_32_mingw_builds.py:154
(Diff revision 4)
> +        'freetype-devel-2.3.11-6.el6_1.8.i686',
> +        'freetype-2.3.11-6.el6_1.8.x86_64',
> +        ######## 32 bit specific ###########
> +    ],
> +    'src_mozconfig': 'browser/config/mozconfigs/win32/mingw32',
> +    'tooltool_manifest_src': "browser/config/tooltool-manifests/mingw32/releng.manifest",

No need to put the tooltool manifest here. The taskcluster configuration takes precedence.
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review183546

> Having /home/worker/workspace/build in the install dir is still suspicious.

I changed the build script to use a tmp directory for the root directory. I believe it still works, so this makes me think that the matching install and extraction directories is a conincidence and not something we're relying on.

> Now that linux builds are using gcc 6, can that be updated?

There are some bugs for gcc 6: 1389965 and 1393814 at least, with some more likely hidden behind those. I'm intending to get to those, but I would prefer not to block this on getting them resolved.
(In reply to Mike Hommey [:glandium] from comment #102)
> If fxc2 needs the winpthreads dll, copy the winpthreads dll in the fxc2
> package.

Yes, that makes sense. Done.

> > > why not use download_and_check from build-gcc.sh? (possibly moving it to a separate file)
> > 
> > I can, but to check first: build-gcc.sh lives in build/unix/build-gcc/ and
> > build-mingw32-nsis.sh lives in taskcluster/scripts/misc/. Whether I use
> > build-gcc.sh directly or abstract the functions to a separate file, I'll be
> > including across vastly different directory hierarchies. Is that alright? 
> > (It's not that complicated of logic to duplicate...)
> 
> build-gcc.sh is already included from taskcluster/script/misc/ scripts.

Okay. I abstracted download_and_check into a separate file.



Something else arose. This builds the browser successfully, but it does not run. I suspected it may have to do with my patch in Bug 1364560, but Georg from Tor pointed me at a msvcr problem.

The tor ticket is https://bugs.torproject.org/9084 and it talks about Firefox not starting on WinXP because it pulls in msvcrt.dll; and msvcrt.dll on WinXP does not have functions used by Firefox (such as the _s secure functions). While hunting this down they determined that gcc is responsible for pulling in msvcrt.dll (including on newer versions of Windows, such as Windows 10). They correct this to link against msvcr100.dll which they ship with the browser.

They fix this in four steps:
1) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-utils.yml#n92 (lines 92-99) they pause the gcc build in the middle and adjust gcc to link against msvcr100 before building the gcc libraries like libssp
2) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-firefox.yml#n49 (lines 49-72) they do the same thing - change the gcc spec file to use msvcr100, and add a build flag to the necessary environment variables to make it use that spec file
3) In https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/gitian/descriptors/windows/gitian-firefox.yml#n67 (line 67) they do an ugly hack on the nsprpub code to change a function name.
4) They ship msvcr100.dll and use it instead of any system dll that would be loaded

I definetly don't intend to do #3, I'll figure out how to get that fixed in a different bug. But for the build job I am going to see if 1, 2, and 4 will produce a running build.

Tor doesn't care for this hack. They would be very happy for this to be solved in a new way. But I am unsure what that would be. If anyone has suggestions we'd be happy to investigate them!
Comment on attachment 8884418 [details]
Bug 1330608 Add the MinGW32 toolchain build to Taskcluster

https://reviewboard.mozilla.org/r/155334/#review184828

::: build/unix/build-gcc/7F74F97C103468EE5D750B583AB00996FC26A641.key:1
(Diff revision 20)
> +-----BEGIN PGP PUBLIC KEY BLOCK-----

Can you run this through https://github.com/glandium/pgpstrip ?

::: build/unix/build-gcc/build-gcc.sh:105
(Diff revision 20)
>    tar caf $root_dir/gcc.tar.xz gcc/
>    popd
>  }
> +
> +build_gcc_and_mingw() {
> +  export install_dir=$root_dir/tools/gcc/

I should probably have been more explicit about what I meant. I didn't mean that specifically /home/worker/workspace/build was bad and that using a temporary directory would fix my concerns, but that $root_dir being in there at all is weird. OTOH, it works, and that's not used for tier-1 builds, so meh, let's land this already. Please file a followup, though.
Attachment #8884418 - Flags: review?(mh+mozilla) → review+
(In reply to Mike Hommey [:glandium] from comment #109)
> Can you run this through https://github.com/glandium/pgpstrip ?

Done.

> ::: build/unix/build-gcc/build-gcc.sh:105
> (Diff revision 20)
> >    tar caf $root_dir/gcc.tar.xz gcc/
> >    popd
> >  }
> > +
> > +build_gcc_and_mingw() {
> > +  export install_dir=$root_dir/tools/gcc/
> 
> I should probably have been more explicit about what I meant. I didn't mean
> that specifically /home/worker/workspace/build was bad and that using a
> temporary directory would fix my concerns, but that $root_dir being in there
> at all is weird. OTOH, it works, and that's not used for tier-1 builds, so
> meh, let's land this already. Please file a followup, though.

I filed Bug 1400484

Did you intend for me to land the toolchain jobs and then wait while the browser build is reviewed; or wait until both are ready?  

(I'm still working on landing a couple of some patches to our tree and mingw's to make the browser build green too.)
(In reply to Tom Ritter [:tjr] from comment #108)
> Something else arose. This builds the browser successfully, but it does not
> run. I suspected it may have to do with my patch in Bug 1364560, but Georg
> from Tor pointed me at a msvcr problem.

In my testing this is only a problem on Win7. On later Windows, msvcrt has all the necessary exports. The build stills fail to run but it is because of bug 1372958 instead.
Comment on attachment 8899589 [details]
Bug 1330608 Add the MinGW32 browser build to Taskcluster

https://reviewboard.mozilla.org/r/170882/#review186772

Please land the first half and move this to a followup.
Attachment #8899589 - Flags: review?(mh+mozilla)
Attachment #8899589 - Flags: review?(mh+mozilla)
Attachment #8899589 - Attachment is obsolete: true
I split off the second half into Bug 1403058 but did not move any dependencies (or edit this description).
Pushed by kwierso@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/fa216426a67b
Add the MinGW32 toolchain build to Taskcluster r=glandium
Product: Core → Firefox Build System
See Also: → 1463261
Depends on: 1470772
Depends on: 1511073

The leave-open keyword is there and there is no activity for 6 months.
:tjr, maybe it's time to close this bug?

Flags: needinfo?(tom)
Type: defect → task
Flags: needinfo?(tom)
Keywords: leave-open → meta
Summary: Add tor build chain to taskcluster → [meta] Add tor build chain to taskcluster
No longer depends on: 1500802
No longer depends on: 1490835
No longer depends on: 1576474
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: