Closed Bug 1132574 Opened 7 years ago Closed 6 years ago

gecko: Use branch specific caches for builds

Categories

(Taskcluster :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jlal, Assigned: garndt)

References

Details

Attachments

(1 file)

In general this is to be used so we can isolate the amount of build system related issues that crop up during creation of builds across branches (this is a bandaid) our builder nodes are generally larger (160 gig or more) so we can store 5+ objdir without any issues.
Duplicate of this bug: 1132565
Blocks: 1154669
It appears in the last couple of days, most worker types seem to be ok on disk space.  For the builders, emulator-l seems the one with the least average free disk space at 60gib or so.  We could change our task configs to add a branch specific suffix and monitor the metrics.  We also have alerts setup in papertrail when we get workers hitting diskspace thresholds too frequently.  With the monitoring we should be ok and can revert back if we notice the workers are clearing diskspace too frequently, resulting in having to repopulate caches
Ran an emulator-L debug build, and the cached workspace is 35gb so caching per branch would probably not work well in this case unless we change what we cache or the instance type.

There are three higher level directories in the workspace:
gecko 3.3gb
ccache 3.1 gb
B2G 29gb (12gb just in .repo)

mozilla-taskcluster currently monitors 9 repos and emulators can build on any one of them.  A worker would need at least 315g of disk space to contain a cache for each branch but we would also want more space in case a build task is claimed prior to the volume for a previous task is freed up to be used.  

some possible instance types depending on cost/availability:
c3.8xlarge - 2x320gb SSD (.4/hr, some spikes so maybe limited availability)
d2.2xlarge - 6 x 2000 gb HDD (.16/hr, probably a lot slower builds, not in available in us-west-1)
r3.8xlarge - 2 x 320 SSD (.4 to 3.50/hr, probably not many instances considering pricing spikes over the course of the week)

These prices are considerably more compared to the m3.2xlarge used for emulators that are around .1/hr.

Per-branch caching is definitely more doable for b2g desktop (13gb) and a little more doable for device builds that are around 25gb.
objdir-gecko is only 4.8gb so it's quite possible to not have to change instance types and just move the objdir caching out of workspace caching
Assignee: nobody → garndt
Component: TaskCluster → General
Product: Testing → Taskcluster
No longer blocks: 1080265
Bug 1132574 - Use per branch cache for b2g and mulet builders r=wcosta
Attachment #8646305 - Flags: review?(wcosta)
Attachment #8646305 - Attachment description: MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=wcosta → MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj
Attachment #8646305 - Flags: review?(wcosta) → review?(jopsen)
Comment on attachment 8646305 [details]
MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj
Comment on attachment 8646305 [details]
MozReview Request: Bug 1132574 - Use per branch cache for b2g and mulet builders r=jonasfj

https://reviewboard.mozilla.org/r/15657/#review14005

Ship It\!
Attachment #8646305 - Flags: review?(jopsen) → review+
See Also: → 1201179
You need to log in before you can comment on or make changes to this bug.