1237688 - Investigate using mach artifact install and/or mach build faster builds on infra to speed up non-clobber builds for front-end changes

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[:Gijs (he/him)]

<nalexander>	Gijs: there are three issues I can think of. The first is that the current scheme expects a live internet connection to determine what to build.
<nalexander>	Gijs: the second is related; we have no clear way to make "this job" depend on the binaries from "that job". Growing such a thing is possible but non-trivial, I think.
<nalexander>	Gijs: third, the build is neutered -- DCE doesn't work with tests (yet!) and |mach package-tests| and some other things. We will grow testing back (I hope!) and could probably grow those other things back, but a better compile cache (that cached libs) might be wiser long term.

I think none of these apply to ./mach build faster - if we know the last revision we built on a tree, and we use per-tree objdirs, we can determine relatively easily whether the result can be re-used using just ./mach build faster instead of ./mach build, by checking the files that are part of  `hg st -r <last-rev>`. In fact, for builds on revisions that only change tests and test manifests, we could avoid running the build at all...

./mach artifact builds would be useful for try, because a lot of the builds will be based on revisions that have already been built, but it's true that they're trickier to do.

Comment 1

[:Gijs (he/him)]

Don't know if this is the best place, feel free to move it around more.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to :Gijs Kruitbosch from comment #0)
> <nalexander>	Gijs: there are three issues I can think of. The first is that
> the current scheme expects a live internet connection to determine what to
> build.
> <nalexander>	Gijs: the second is related; we have no clear way to make "this
> job" depend on the binaries from "that job". Growing such a thing is
> possible but non-trivial, I think.
> <nalexander>	Gijs: third, the build is neutered -- DCE doesn't work with
> tests (yet!) and |mach package-tests| and some other things. We will grow
> testing back (I hope!) and could probably grow those other things back, but
> a better compile cache (that cached libs) might be wiser long term.

It's not clear, but everything above is specific to artifact builds.

> I think none of these apply to ./mach build faster - if we know the last
> revision we built on a tree, and we use per-tree objdirs, we can determine
> relatively easily whether the result can be re-used using just ./mach build
> faster instead of ./mach build, by checking the files that are part of  `hg
> st -r <last-rev>`. In fact, for builds on revisions that only change tests
> and test manifests, we could avoid running the build at all...

I agree.  There's work to determine stuff like this for choosing jobs (Android-only, for example); but I don't know where that configuration is specified.  I'd love to be able to annotate (with Files in moz.build, or more automatically) what builds are impacted by what files, and use that to help building fewer things.  Great idea!

Comment 3

[Armen [:armenzg]]

I've spoke with maja_zf and chmanchester and this bug contains very good information, however, all the actionable items to get artifact builds running on automation are already filed (see bug 1284882).

I'm removing the dependency.

(In reply to :Gijs Kruitbosch from comment #0)
> <nalexander>	Gijs: the second is related; we have no clear way to make "this
> job" depend on the binaries from "that job". Growing such a thing is
> possible but non-trivial, I think.
>
There's only one piece missing and that is to control Buildbot's scheduling from in-tree (via Buildbot bridge).

mozci can schedule test jobs based on builds from any push. It's a matter of pointing them to the right URLs.


> I think none of these apply to ./mach build faster - if we know the last
> revision we built on a tree, and we use per-tree objdirs, we can determine
> relatively easily whether the result can be re-used using just ./mach build
> faster instead of ./mach build, by checking the files that are part of  `hg
> st -r <last-rev>`.

What is the difference between ./mach build and ./mach build faster?

> In fact, for builds on revisions that only change tests
> and test manifests, we could avoid running the build at all...
> 
That is bug 1268481.

> ./mach artifact builds would be useful for try, because a lot of the builds
> will be based on revisions that have already been built, but it's true that
> they're trickier to do.

Bug 1284882 will be tracking this.

Comment 4

[:Gijs (he/him)]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #3)
> > I think none of these apply to ./mach build faster - if we know the last
> > revision we built on a tree, and we use per-tree objdirs, we can determine
> > relatively easily whether the result can be re-used using just ./mach build
> > faster instead of ./mach build, by checking the files that are part of  `hg
> > st -r <last-rev>`.
> 
> What is the difference between ./mach build and ./mach build faster?

'faster' doesn't compile anything. It just sticks things in jars / in the objdir on disk, and just about manages to run preprocessing etc. on combined files. It's enough when all you do is change some JS/CSS/XUL, and it generally runs in less than 15 seconds, which isn't something you can say for ./mach build... For a more technical definition, you'd have to ask gps.

Comment 5

[Armen [:armenzg]]

I'm interesting on understanding what actionable items we can take from this bug since two parts of it are already filed (bug 1268481 and bug 1284882).

(In reply to :Gijs Kruitbosch from comment #4)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #3)
> > > I think none of these apply to ./mach build faster - if we know the last
> > > revision we built on a tree, and we use per-tree objdirs, we can determine
> > > relatively easily whether the result can be re-used using just ./mach build
> > > faster instead of ./mach build, by checking the files that are part of  `hg
> > > st -r <last-rev>`.
> > 
> > What is the difference between ./mach build and ./mach build faster?
> 
> 'faster' doesn't compile anything. It just sticks things in jars / in the
> objdir on disk, and just about manages to run preprocessing etc. on combined
> files. It's enough when all you do is change some JS/CSS/XUL, and it
> generally runs in less than 15 seconds, which isn't something you can say
> for ./mach build... For a more technical definition, you'd have to ask gps.

Is this different from artifact builds?

Comment 6

[Chris Manchester (limited bugmail, email directly)]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #5)
> I'm interesting on understanding what actionable items we can take from this
> bug since two parts of it are already filed (bug 1268481 and bug 1284882).
> 
> (In reply to :Gijs Kruitbosch from comment #4)
> > (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> > comment #3)
> > > > I think none of these apply to ./mach build faster - if we know the last
> > > > revision we built on a tree, and we use per-tree objdirs, we can determine
> > > > relatively easily whether the result can be re-used using just ./mach build
> > > > faster instead of ./mach build, by checking the files that are part of  `hg
> > > > st -r <last-rev>`.
> > > 
> > > What is the difference between ./mach build and ./mach build faster?
> > 
> > 'faster' doesn't compile anything. It just sticks things in jars / in the
> > objdir on disk, and just about manages to run preprocessing etc. on combined
> > files. It's enough when all you do is change some JS/CSS/XUL, and it
> > generally runs in less than 15 seconds, which isn't something you can say
> > for ./mach build... For a more technical definition, you'd have to ask gps.
> 
> Is this different from artifact builds?

Since bug 1239217, the differences should be minimal. That bug means we're automatically using parts of the "faster" backend when artifact builds are in use.

Comment 7

[Armen [:armenzg]]

Good to know.
Is there anything left in here?

Comment 8

[:Gijs (he/him)]

(In reply to Chris Manchester (:chmanchester) from comment #6)
> (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> comment #5)
> > I'm interesting on understanding what actionable items we can take from this
> > bug since two parts of it are already filed (bug 1268481 and bug 1284882).
> > 
> > (In reply to :Gijs Kruitbosch from comment #4)
> > > (In reply to Armen Zambrano [:armenzg] - Engineering productivity from
> > > comment #3)
> > > > > I think none of these apply to ./mach build faster - if we know the last
> > > > > revision we built on a tree, and we use per-tree objdirs, we can determine
> > > > > relatively easily whether the result can be re-used using just ./mach build
> > > > > faster instead of ./mach build, by checking the files that are part of  `hg
> > > > > st -r <last-rev>`.
> > > > 
> > > > What is the difference between ./mach build and ./mach build faster?
> > > 
> > > 'faster' doesn't compile anything. It just sticks things in jars / in the
> > > objdir on disk, and just about manages to run preprocessing etc. on combined
> > > files. It's enough when all you do is change some JS/CSS/XUL, and it
> > > generally runs in less than 15 seconds, which isn't something you can say
> > > for ./mach build... For a more technical definition, you'd have to ask gps.
> > 
> > Is this different from artifact builds?
> 
> Since bug 1239217, the differences should be minimal. That bug means we're
> automatically using parts of the "faster" backend when artifact builds are
> in use.

Well... I don't know about that, but I can use ./mach build faster after a full ./mach build on non-artifact builds. IOW, ./mach build faster doesn't require that you use artifact builds, though artifact builds, if I'm reading your comment right, do use parts of the 'faster' backend. So, for instance, you could likely use ./mach build faster if you were an infra build and your objdir was built off rev X, and hg status --rev X --rev . only contained .js/.jsm/.xul/.css/.png/.gif/.jpg/.svg/.mn/moz.build files. Which would speed up build times without relying on artifact builds.

Comment 9

[Aki Sasaki (not active)]

A lot has changed in the past 6 years - taskcluster, in-tree configs, artifact builds.
Resolving incomplete, but if still wanted, most likely we want to be in the Task Configuration or a test-related component for moving tests from non-artifact builds to artifact builds.