Currently, it's not possible for a local developer to reproduce Mozilla's official build environment. This is mostly because the operating system config and packages used on official infrastructure are not publicly available. While there are good reasons for the tree to be developed and tested on as many OS/package/environment combinations as possible, there are also times where developers want to ensure their environment is as close to the official one as possible. e.g. when developing C++, you gain confidence that your local compiler warnings will be exactly what's encountered on the official build. I'm opening this bug to track the ability for local developers to reproduce RelEng's build environment so produced builds are deterministic and bit-identical to what they would get on RelEng hardware (at least to a reasonable degree, such as using the same toolchain/packages). While I'd like to see us do this on all 3 of the tier 1 platforms (Windows, OS X, and Linux), I'm limiting this bug to just Linux because I feel that's the only platform where this is realistically attainable at the moment. This effort is currently low priority. This bug is mostly for tracking purposes.
There are some useful packages at http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/. We *might* have enough content there to attempt automated reproduction of the official CentOS build environment...
Note that it's currently impossible to get idempotent builds even on the same machine. bug 742169 is one example.
Created attachment 766556 [details] [diff] [review] Add a Vagrantfile for creating the Linux64 build environment And here is my first stab at it. Once I publish the CentOS 6.2 image I created (will need to do it tomorrow once I have a faster internet connection), you should be able to: cd config/bootstrap-linux64 vagrant up # Wait a few minutes for the VM to initialize, download package # updates, populate mock_mozilla, etc. vagrant ssh mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/me/build --unpriv --shell /bin/bash ./mach build I currently mount the cloned source directory as /builds/slave/me/build, which is fine as the regular ssh user. However, it gets unmounted in the mock environment. I need to do some homework. For this to get checked into the tree, I'd like to have the Vagrant .box file published somewhere somewhat official. Unless people counts, I may start asking around for hosting. It should be a one-time thing, methinks.
Comment on attachment 766556 [details] [diff] [review] Add a Vagrantfile for creating the Linux64 build environment And wrong bug. Derp. See bug 886226.
Attachment #766556 - Attachment is obsolete: true
Summary: [meta] Deterministic, bit-identical Linux builds → [meta] Deterministic, bit-identical and/or verifiable Linux builds
Tor has published details of how they are doing deterministic builds. Good reading. https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details
Created attachment 8363336 [details] [diff] [review] Disable library timestamping libfaketime is unable to spoof these timestamps because they use sub-millisecond precision.
Created attachment 8363337 [details] [diff] [review] Misc reproducibility fixes (against FF24ESR) Miscellaneous reproducibility fixes to the Firefox build system for Tor Browser.
I've attached a couple fixes we had to do to the build system (beyond using libfaketime and Gitian to provide a controlled environment, which probably gives us quite a bit in terms of build paths, hostname, package versions, etc as described in the technical details post that Gregory Szorc linked in Comment 5). On Linux, we also had to remove the FIPS140 .chk library signatures for NSS, as these were created using a throwaway private key during the build process and are not reproducible. For Windows and Mac, we use MinGW-W64 and a fork of 'toolchain4', respectively, to cross-compile for those platforms in Gitian in a reproducible way.
Mike: Could you please open new bugs for each attachment so we can discuss each issue separately? This is intended to be a tracking bug with only high-level discussion and status updates. Feel free to ask for assistance in #build on irc.mozilla.org.
Also rebase on mozilla-central, because i'm sure some of the things in the second patch are already fixed.
I just discovered Gentoo Prefix (https://www.gentoo.org/proj/en/gentoo-alt/prefix/). It makes it possible to install Gentoo (a Linux distro distinguished by how much it relies on compiling from source) on pretty much any UNIX-like operating system, including OS X. It got me thinking that we could potentially use Gentoo Prefix to produce our build chroot/container archives/images. Not sure if it has any compelling advantages over Gitian (which I would assume to be the frontrunner at this stage). Just throwing the idea out there.
There is also an ongoing project for Debian, if it helps. https://wiki.debian.org/ReproducibleBuilds
Did you consider creating builder with Docker?
(In reply to Adam Stankiewicz from comment #14) > Did you consider creating builder with Docker? Efforts (not tracked by this bug) are underway to modernize Firefox's building infrastructure. This includes support for building in containers / Docker.
(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #15) > (In reply to Adam Stankiewicz from comment #14) > > Did you consider creating builder with Docker? > > Efforts (not tracked by this bug) are underway to modernize Firefox's > building infrastructure. This includes support for building in containers / > Docker. @Gregory, I realize there's been a lot of noise about deterministic builds (e.g. bug 1036178). However, I am curious, is there any status update on this? You said "Efforts are underway", are those efforts detailed anywhere? Can the community do anything to help move the efforts forward?
catlee: Could you please provide a status update?
From the infrastructure side, we've started work on building docker containers for FirefoxOS. This work should be a good basis for use for Firefox builds as well. I can't comment to any changes required by the build system.
Summary: [meta] Deterministic, bit-identical and/or verifiable Linux builds → [meta] Deterministic, reproducible, bit-identical and/or verifiable Linux builds
I was asked to look at the current state of this bug since it's been a while. For Linux, we currently have nearly bit-identical builds when specifying the buildid by exporting MOZ_BUILD_DATE, except for the following: 1) The NSS .chk files are always different (see #c8). 2) Timestamps of the files inside the .tar.bz2 package will differ, but untarring them and using a recursive diff will reveal no differences (except for the aforementioned .chk files) 3) PGO For 1), it's possible we can just remove them from the built package. We already don't ship them on OSX (see the thread starting here: https://bugzilla.mozilla.org/show_bug.cgi?id=1096494#c7). It's not clear if we still want to support FIPS-mode, or if it is even possible given that we ship with NSS from our tree rather than something that's be FIPS-certified. This is probably worth looking into so we can remove that packaging code and make diffing packages easier. We might just need someone senior enough to sign off on removing it. For 2) I'm not sure how much we care about generating two tarballs with the same hash. I think it's pretty straightforward to untar and then diff to determine equality, personally. However, it would be helpful if we had a clear idea of the end-goal for this bug to know whether this is something we should support or not. I believe this is part of what "Disable library timestamping" patch is trying to fix, along with use of libfaketime. The "Misc reproducibility fixes" patch has already been absorbed by other bugs (bug 982075, bug 1168316, and bug 943331). Part 3) is likely the bulk of the remaining effort, since our releases use PGO, and PGO is not currently reproducible. For this we need bug 935637, so that anyone who wants to verify a build can use the same PGO profile that we used to build a release. In addition to publishing PGO profiles, this may involve some build system work so that a PGO build can be done using an existing profile rather than first generating one. Note that all this only applies to Linux builds - I haven't investigated other platforms yet.
While we may have (mostly) deterministic builds, there is still the issue of reproducible. That requires a build environment that is reproducible. There is still a lot of work to do here. While TaskCluster is using Docker images on Linux now, Docker images are notoriously not very reproducible (because `yum update/install` installs the latest version of packages advertised on servers and that can change over time). We /could/ publish the Docker images Mozilla uses (they are probably already public for all I know). However, paranoid people will want to reproduce those independently. It's turtles all the way down of course. The question is how far do we want to go. That's why we need an end-goal for this bug :)
I started a discussion of what a good end-goal could be https://groups.google.com/forum/#!topic/mozilla.dev.platform/dFCNZC_pnq4
Clearing a 4 year old needinfo request.
An update on this project on Gregory's blog: https://gregoryszorc.com/blog/2018/06/20/deterministic-firefox-builds/
Attachment #8363336 - Attachment is obsolete: true
Attachment #8363337 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.