Last Comment Bug 885777 - (fx-reproducible-build) [meta] Deterministic, reproducible, bit-identical and/or verifiable Linux builds
(fx-reproducible-build)
: [meta] Deterministic, reproducible, bit-identical and/or verifiable Linux builds
Status: NEW
[tor]
: meta
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All All
-- normal with 8 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Gregory Szorc [:gps] (away until 2017-03-20)
Mentors:
https://brendaneich.com/2014/01/trust...
: 1036178 1083277 1166201 (view as bug list)
Depends on: 935637 1115874 1166547 1166550 1166554 1289812 1330608 768879 886226 896023 942091 943331 981558 982055 982075 1166243 1166538 1168231 1168316 1169158 1169174 1288610
Blocks: tb-reproducible-build
  Show dependency treegraph
 
Reported: 2013-06-21 09:46 PDT by Gregory Szorc [:gps] (away until 2017-03-20)
Modified: 2017-01-18 17:37 PST (History)
59 users (show)
gps: needinfo? (mikeperry)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Add a Vagrantfile for creating the Linux64 build environment (9.94 KB, patch)
2013-06-24 00:04 PDT, Gregory Szorc [:gps] (away until 2017-03-20)
no flags Details | Diff | Splinter Review
Disable library timestamping (1.37 KB, patch)
2014-01-21 16:55 PST, Mike Perry
no flags Details | Diff | Splinter Review
Misc reproducibility fixes (against FF24ESR) (6.56 KB, patch)
2014-01-21 16:57 PST, Mike Perry
no flags Details | Diff | Splinter Review

Description User image Gregory Szorc [:gps] (away until 2017-03-20) 2013-06-21 09:46:42 PDT
Currently, it's not possible for a local developer to reproduce Mozilla's official build environment. This is mostly because the operating system config and packages used on official infrastructure are not publicly available.

While there are good reasons for the tree to be developed and tested on as many OS/package/environment combinations as possible, there are also times where developers want to ensure their environment is as close to the official one as possible. e.g. when developing C++, you gain confidence that your local compiler warnings will be exactly what's encountered on the official build.

I'm opening this bug to track the ability for local developers to reproduce RelEng's build environment so produced builds are deterministic and bit-identical to what they would get on RelEng hardware (at least to a reasonable degree, such as using the same toolchain/packages). While I'd like to see us do this on all 3 of the tier 1 platforms (Windows, OS X, and Linux), I'm limiting this bug to just Linux because I feel that's the only platform where this is realistically attainable at the moment.

This effort is currently low priority. This bug is mostly for tracking purposes.
Comment 1 User image Gregory Szorc [:gps] (away until 2017-03-20) 2013-06-21 10:17:20 PDT
There are some useful packages at http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/.

We *might* have enough content there to attempt automated reproduction of the official CentOS build environment...
Comment 2 User image Ted Mielczarek [:ted.mielczarek] 2013-06-21 11:13:11 PDT
Note that it's currently impossible to get idempotent builds even on the same machine. bug 742169 is one example.
Comment 3 User image Gregory Szorc [:gps] (away until 2017-03-20) 2013-06-24 00:04:11 PDT
Created attachment 766556 [details] [diff] [review]
Add a Vagrantfile for creating the Linux64 build environment

And here is my first stab at it. Once I publish the CentOS 6.2 image I
created (will need to do it tomorrow once I have a faster internet
connection), you should be able to:

  cd config/bootstrap-linux64
  vagrant up
  # Wait a few minutes for the VM to initialize, download package
  # updates, populate mock_mozilla, etc.
  vagrant ssh
  mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/me/build --unpriv --shell /bin/bash
  ./mach build

I currently mount the cloned source directory as /builds/slave/me/build,
which is fine as the regular ssh user. However, it gets unmounted in the
mock environment. I need to do some homework.

For this to get checked into the tree, I'd like to have the Vagrant .box
file published somewhere somewhat official. Unless people counts, I may
start asking around for hosting. It should be a one-time thing,
methinks.
Comment 4 User image Gregory Szorc [:gps] (away until 2017-03-20) 2013-06-24 00:05:48 PDT
Comment on attachment 766556 [details] [diff] [review]
Add a Vagrantfile for creating the Linux64 build environment

And wrong bug. Derp. See bug 886226.
Comment 5 User image Gregory Szorc [:gps] (away until 2017-03-20) 2013-11-06 11:13:59 PST
Tor has published details of how they are doing deterministic builds. Good reading.

https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise
https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details
Comment 6 User image Mike Perry 2014-01-21 16:55:57 PST
Created attachment 8363336 [details] [diff] [review]
Disable library timestamping

libfaketime is unable to spoof these timestamps because they use sub-millisecond precision.
Comment 7 User image Mike Perry 2014-01-21 16:57:07 PST
Created attachment 8363337 [details] [diff] [review]
Misc reproducibility fixes (against FF24ESR)

Miscellaneous reproducibility fixes to the Firefox build system for Tor Browser.
Comment 8 User image Mike Perry 2014-01-21 17:04:31 PST
I've attached a couple fixes we had to do to the build system (beyond using libfaketime and Gitian to provide a controlled environment, which probably gives us quite a bit in terms of build paths, hostname, package versions, etc as described in the technical details post that Gregory Szorc linked in Comment 5).

On Linux, we also had to remove the FIPS140 .chk library signatures for NSS, as these were created using a throwaway private key during the build process and are not reproducible.

For Windows and Mac, we use MinGW-W64 and a fork of 'toolchain4', respectively, to cross-compile for those platforms in Gitian in a reproducible way.
Comment 9 User image Gregory Szorc [:gps] (away until 2017-03-20) 2014-01-21 19:11:15 PST
Mike: Could you please open new bugs for each attachment so we can discuss each issue separately? This is intended to be a tracking bug with only high-level discussion and status updates. Feel free to ask for assistance in #build on irc.mozilla.org.
Comment 10 User image Mike Hommey [:glandium] 2014-01-21 19:45:39 PST
Also rebase on mozilla-central, because i'm sure some of the things in the second patch are already fixed.
Comment 11 User image Gregory Szorc [:gps] (away until 2017-03-20) 2014-03-24 21:24:09 PDT
I just discovered Gentoo Prefix (https://www.gentoo.org/proj/en/gentoo-alt/prefix/). It makes it possible to install Gentoo (a Linux distro distinguished by how much it relies on compiling from source) on pretty much any UNIX-like operating system, including OS X.

It got me thinking that we could potentially use Gentoo Prefix to produce our build chroot/container archives/images. Not sure if it has any compelling advantages over Gitian (which I would assume to be the frontrunner at this stage). Just throwing the idea out there.
Comment 12 User image Justin Dolske [:Dolske] 2014-07-08 17:07:26 PDT
*** Bug 1036178 has been marked as a duplicate of this bug. ***
Comment 13 User image dp 2014-07-10 10:56:53 PDT
There is also an ongoing project for Debian, if it helps. https://wiki.debian.org/ReproducibleBuilds
Comment 14 User image Adam Stankiewicz 2014-07-12 03:17:08 PDT
Did you consider creating builder with Docker?
Comment 15 User image Gregory Szorc [:gps] (away until 2017-03-20) 2014-07-12 09:39:26 PDT
(In reply to Adam Stankiewicz from comment #14)
> Did you consider creating builder with Docker?

Efforts (not tracked by this bug) are underway to modernize Firefox's building infrastructure. This includes support for building in containers / Docker.
Comment 16 User image blong 2014-09-04 20:07:35 PDT
(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #15)
> (In reply to Adam Stankiewicz from comment #14)
> > Did you consider creating builder with Docker?
> 
> Efforts (not tracked by this bug) are underway to modernize Firefox's
> building infrastructure. This includes support for building in containers /
> Docker.

@Gregory, I realize there's been a lot of noise about deterministic builds (e.g. bug 1036178).  However, I am curious, is there any status update on this?  You said "Efforts are underway", are those efforts detailed anywhere?  Can the community do anything to help move the efforts forward?
Comment 17 User image Gregory Szorc [:gps] (away until 2017-03-20) 2014-09-05 08:18:02 PDT
catlee: Could you please provide a status update?
Comment 18 User image Chris AtLee [:catlee] 2014-09-16 07:39:34 PDT
From the infrastructure side, we've started work on building docker containers for FirefoxOS. This work should be a good basis for use for Firefox builds as well. I can't comment to any changes required by the build system.
Comment 19 User image Mike Hommey [:glandium] 2015-05-19 02:19:22 PDT
*** Bug 1166201 has been marked as a duplicate of this bug. ***
Comment 20 User image Sylvestre Ledru [:sylvestre] 2015-05-19 03:24:31 PDT
*** Bug 1083277 has been marked as a duplicate of this bug. ***
Comment 21 User image Michael Shal [:mshal] 2016-07-15 13:08:46 PDT
I was asked to look at the current state of this bug since it's been a while.

For Linux, we currently have nearly bit-identical builds when specifying the buildid by exporting MOZ_BUILD_DATE, except for the following:

 1) The NSS .chk files are always different (see #c8).
 2) Timestamps of the files inside the .tar.bz2 package will differ, but untarring them and using a recursive diff will reveal no differences (except for the aforementioned .chk files)
 3) PGO

For 1), it's possible we can just remove them from the built package. We already don't ship them on OSX (see the thread starting here: https://bugzilla.mozilla.org/show_bug.cgi?id=1096494#c7). It's not clear if we still want to support FIPS-mode, or if it is even possible given that we ship with NSS from our tree rather than something that's be FIPS-certified. This is probably worth looking into so we can remove that packaging code and make diffing packages easier. We might just need someone senior enough to sign off on removing it.

For 2) I'm not sure how much we care about generating two tarballs with the same hash. I think it's pretty straightforward to untar and then diff to determine equality, personally. However, it would be helpful if we had a clear idea of the end-goal for this bug to know whether this is something we should support or not.

I believe this is part of what "Disable library timestamping" patch is trying to fix, along with use of libfaketime. The "Misc reproducibility fixes" patch has already been absorbed by other bugs (bug 982075, bug 1168316, and bug 943331).

Part 3) is likely the bulk of the remaining effort, since our releases use PGO, and PGO is not currently reproducible. For this we need bug 935637, so that anyone who wants to verify a build can use the same PGO profile that we used to build a release. In addition to publishing PGO profiles, this may involve some build system work so that a PGO build can be done using an existing profile rather than first generating one.

Note that all this only applies to Linux builds - I haven't investigated other platforms yet.
Comment 22 User image Gregory Szorc [:gps] (away until 2017-03-20) 2016-07-15 13:51:25 PDT
While we may have (mostly) deterministic builds, there is still the issue of reproducible. That requires a build environment that is reproducible. There is still a lot of work to do here.

While TaskCluster is using Docker images on Linux now, Docker images are notoriously not very reproducible (because `yum update/install` installs the latest version of packages advertised on servers and that can change over time). We /could/ publish the Docker images Mozilla uses (they are probably already public for all I know). However, paranoid people will want to reproduce those independently. It's turtles all the way down of course. The question is how far do we want to go.

That's why we need an end-goal for this bug :)
Comment 23 User image David Bruant 2016-07-17 09:41:11 PDT
I started a discussion of what a good end-goal could be https://groups.google.com/forum/#!topic/mozilla.dev.platform/dFCNZC_pnq4

Note You need to log in before you can comment on or make changes to this bug.