1132574 - gecko: Use branch specific caches for builds

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[James Lal [:lightsofapollo]]

In general this is to be used so we can isolate the amount of build system related issues that crop up during creation of builds across branches (this is a bandaid) our builder nodes are generally larger (160 gig or more) so we can store 5+ objdir without any issues.

Comment 2

[Greg Arndt [:garndt]]

It appears in the last couple of days, most worker types seem to be ok on disk space.  For the builders, emulator-l seems the one with the least average free disk space at 60gib or so.  We could change our task configs to add a branch specific suffix and monitor the metrics.  We also have alerts setup in papertrail when we get workers hitting diskspace thresholds too frequently.  With the monitoring we should be ok and can revert back if we notice the workers are clearing diskspace too frequently, resulting in having to repopulate caches

Comment 3

[Greg Arndt [:garndt]]

Ran an emulator-L debug build, and the cached workspace is 35gb so caching per branch would probably not work well in this case unless we change what we cache or the instance type.

There are three higher level directories in the workspace:
gecko 3.3gb
ccache 3.1 gb
B2G 29gb (12gb just in .repo)

mozilla-taskcluster currently monitors 9 repos and emulators can build on any one of them.  A worker would need at least 315g of disk space to contain a cache for each branch but we would also want more space in case a build task is claimed prior to the volume for a previous task is freed up to be used.  

some possible instance types depending on cost/availability:
c3.8xlarge - 2x320gb SSD (.4/hr, some spikes so maybe limited availability)
d2.2xlarge - 6 x 2000 gb HDD (.16/hr, probably a lot slower builds, not in available in us-west-1)
r3.8xlarge - 2 x 320 SSD (.4 to 3.50/hr, probably not many instances considering pricing spikes over the course of the week)

These prices are considerably more compared to the m3.2xlarge used for emulators that are around .1/hr.

Per-branch caching is definitely more doable for b2g desktop (13gb) and a little more doable for device builds that are around 25gb.

Comment 4

[Greg Arndt [:garndt]]

objdir-gecko is only 4.8gb so it's quite possible to not have to change instance types and just move the objdir caching out of workspace caching

Comment 5

[Greg Arndt [:garndt]]

Attachment: MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

Bug 1132574 - Use per branch cache for b2g and mulet builders r=wcosta

Comment 6

[Greg Arndt [:garndt]]

Comment on attachment 8646305 [details]
MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

Comment 7

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8646305 [details]
MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

https://reviewboard.mozilla.org/r/15657/#review14005

Ship It\!

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/e1141b1d3538

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e1141b1d3538