1115107 - Add a Nix expression for building Firefox.

Status: NEW

(Firefox Build System :: General, defect)

Description

[Nicolas B. Pierron [:nbp] {away}]

Attachment: Add a Nix expression for building Firefox.

Using a Nix expression for building Firefox has some advantages. With such Nix expression, building Firefox is just a matter of:
 - Installing Nix (Linux / Mac)
 - Running one command.

 $ nix-build ./build/nix/release.nix -A build.x86_64-linux

This command pull all the dependencies listed in this file from a trusted binary cache.  Then it computes a sha1 of the sources, and a sha1 corresponding to the expression used for building firefox.  If a trusted remote has the build result, then it is pulled.  If no one has the build result, then it is built locally, and others might pull from you.

A developer, can spawn a shell with all the dependencies needed for the build, by running the following command:

 $ nix-shell ./build/nix/release.nix -A build.x86_64-linux

Comment 1

[Gregory Szorc [:gps]]

Comment on attachment 8540887 [details] [diff] [review]
Add a Nix expression for building Firefox.

Review of attachment 8540887 [details] [diff] [review]:
-----------------------------------------------------------------

Cool!

I see you have this building Firefox. That's fine, but I have other plans.

I'd like to explore using Nix to manage the build environment for Firefox itself. e.g. |mach build| would set up a Nix env for building and would subsequently perform building from within an "activated" environment. In this world, the build and run-time environment is abstracted away from the user and existing mach functionality "just works." I think this world is more compatible with a Dockerized/containerized future, which is inevitably where we're likely to end up.

I imagine this playing out something like:

1) User types |mach build|, |mach run|, etc.
2) We consult the in-tree Nix file(s) and (re-)build our Nix environment as necessary. We could even create the environment under the objdir!
3) Execute the underlying functionality inside the Nix env.

Another ideal property is there is determinism over time in the contents of the environment. e.g. I want people on different machines to be able to reproduce the same environment. e.g. if a year from now I check out a revision of Firefox 37, I should be able to build the same environment that I built today. This ensures that we can rebuild any changeset over time. I'm not sure how we would "pin" packages, etc to facilitate this.

You don't have to implement this extra functionality (I or someone else familiar with build foo could do that).

Comment 2

[cajbir (:cajbir)]

Comment on attachment 8540887 [details] [diff] [review]
Add a Nix expression for building Firefox.

Review of attachment 8540887 [details] [diff] [review]:
-----------------------------------------------------------------

Nice!

Comment 3

[Mike Hommey [:glandium]]

Comment on attachment 8540887 [details] [diff] [review]
Add a Nix expression for building Firefox.

Review of attachment 8540887 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/nix/release.nix
@@ +39,5 @@
> +          xlibs.damageproto xlibs.libXdamage xlibs.libXrender xlibs.kbproto
> +
> +          gnome.libart_lgpl gnome.libbonobo gnome.libbonoboui
> +          gnome.libgnome gnome.libgnomecanvas gnome.libgnomeui
> +          gnome.libIDL

The main problem I see with this is that it is yet another list of dependencies that is not next to the other ones. So now, when something new is needed, not only do you have to remember to update the mach bootstrap code, but now, there's that as well.

Comment 4

[Nicolas B. Pierron [:nbp] {away}]

(In reply to Gregory Szorc [:gps] from comment #1)
> I imagine this playing out something like:
> 
> 1) User types |mach build|, |mach run|, etc.
> 2) We consult the in-tree Nix file(s) and (re-)build our Nix environment as
> necessary. We could even create the environment under the objdir!

I did that before the creation of the nix-shell tool, and I use it daily to have different tool-chains per objdir (gcc/clang * i686/x86_64 * opt/dbg/…).

> 3) Execute the underlying functionality inside the Nix env.

This is what the nix-shell is meant for:

  $ nix-shell ./build/nix/release.nix -A build.x86_64-linux --pure --command "make -C $objdir"

Note: I have to update the patch to remove the dependency on the source directory when we are using the nix-shell command.

> Another ideal property is there is determinism over time in the contents of
> the environment. e.g. I want people on different machines to be able to
> reproduce the same environment. e.g. if a year from now I check out a
> revision of Firefox 37, I should be able to build the same environment that
> I built today. This ensures that we can rebuild any changeset over time. I'm
> not sure how we would "pin" packages, etc to facilitate this.

If you want to do so, then you want to pin a nixpkgs revision [1], and use it for building.  This is something which can be done by looking at the nixpkgs channel [2] which is used.  Each channel contains a copy of the nixpkgs repository as well as the commit sha1 of it.

You can easily extract which commit of nixpkgs channel you are building against with a simple shell command:

 $ cat ~/.nix-defexpr/*/{*/,}nixpkgs/svn-revision </dev/null

Then building with environment which are a year old should be fine, but ideally, you want to ensure that you do not depend websites to rebuilding the environment, either by keeping the sources for rebuilding everything[3], or by keeping the binaries [4].

[1] https://github.com/NixOS/nixpkgs
[2] http://nixos.org/channels/
[3] https://nixos.org/wiki/Download_all_sources , with the following command:
 $ IN_NIX_SHELL=1 nix-instantiate ./build/nix/release.nix -A build.x86_64-linux

[4] http://sandervanderburg.blogspot.com.au/2014/07/backing-up-nix-and-hydra-builds.html

Comment 5

[Nicolas B. Pierron [:nbp] {away}]

Attachment: Add a Nix expression for building Firefox.

Delta:
 - Removing debugging statements.
 - nix-shell is now spawned in a second instead of a minute. (discard the
   dependency on the source when using the nix-shell)

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Mike Hommey [:glandium] from comment #3)
> The main problem I see with this is that it is yet another list of
> dependencies that is not next to the other ones. So now, when something new
> is needed, not only do you have to remember to update the mach bootstrap
> code, but now, there's that as well.

We currently have this problem with dependencies being specified in arbitrary places in configure as well as in the bootstrap scripts. If we could figure out some way to make one place the canonical source of truth for this information and use it everywhere else that'd be fantastic.

Comment 7

[Morgan Phillips [:mrrrgn]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> (In reply to Mike Hommey [:glandium] from comment #3)
> > The main problem I see with this is that it is yet another list of
> > dependencies that is not next to the other ones. So now, when something new
> > is needed, not only do you have to remember to update the mach bootstrap
> > code, but now, there's that as well.
> 
> We currently have this problem with dependencies being specified in
> arbitrary places in configure as well as in the bootstrap scripts. If we
> could figure out some way to make one place the canonical source of truth
> for this information and use it everywhere else that'd be fantastic.

So, I'm using the mozharness configs as the canonical source for our build environment docker images: https://github.com/mozilla/build-environments We could use the same approach here I think.