Decision and action tasks should share workspace cache across {{project}}s

RESOLVED FIXED

Status

Firefox Build System
Task Configuration
RESOLVED FIXED
2 years ago
5 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Assignee)

Description

2 years ago
Explanation forthcoming in a commit message.
(Assignee)

Comment 1

2 years ago
Created attachment 8776053 [details]
Bug 1290516 - Remove {{project}} from cache path for decision and action tasks;

Currently, the gecko-decision and action tasks maintain a separate cache
for each "project" (Firefox repo) source checkout. In my opinion, this
is not necessary.

Separate caches/source checkouts mean a lower chance for a cache hit:
a lower chance that any given worker will have a source checkout readily
available for the "project" being built. They also mean workers store
more data and files in caches.

By using a shared cache for source checkouts, workers will ideally
only have a single source checkout cache and will be pretty much
guaranteed to have a checkout readily available (the main exception
being the first task on a freshly-provisioned worker).

A downside of a shared cache is that workers may spend more time doing
`hg update` operations. e.g. if the last task was from mozilla-release
and the current task is from mozilla-central, thousands of files will
be updated. However, I believe we will still be better off because
separate caches *increase* the probability for a full checkout, which
involves writing ~150,000 files! So I believe this change will lower
the overall amount of VCS work done by workers while possibly
sacrificing the lowest possible execution time (which is achieved by
minimizing the `hg update` work).

It's worth noting that `hg robustcheckout` (which is used by these
tasks) does the equivalent of an `hg purge --all`. This ensures that all
untracked and ignored files from previous tasks are deleted. So there
should be no contamination from previous tasks.

Review commit: https://reviewboard.mozilla.org/r/67984/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/67984/
Attachment #8776053 - Flags: review?(dustin)
Comment on attachment 8776053 [details]
Bug 1290516 - Remove {{project}} from cache path for decision and action tasks;

https://reviewboard.mozilla.org/r/67984/#review65110
Attachment #8776053 - Flags: review?(dustin) → review+

Comment 3

2 years ago
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f351e9f3e564
Remove {{project}} from cache path for decision and action tasks; r=dustin

Comment 4

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f351e9f3e564
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Updated

5 months ago
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.