[gecko] Emulator and device builds should not re-use cached object directories between branches

RESOLVED FIXED

Status

defect
RESOLVED FIXED
4 years ago
5 months ago

People

(Reporter: garndt, Assigned: wcosta)

Tracking

Details

Attachments

(1 attachment, 4 obsolete attachments)

These builds reuse the entire workspace cache between builds which could cause builds on other branches to fail due to cache poisoning.

The workspace is very large (I think around 25gb or so last time I checked) so creating a separate one for each branch is not feasible unless we found an instance that could handle doing builds as well as a massive amount of disk space.

Perhaps the object directory itself could be branch specific (see bug 1132574 for making b2g desktop workspaces branch specific) and live outside of the typical workspace that's cached.
See Also: → 1132574
Status: NEW → ASSIGNED
Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.
Attachment #8657885 - Attachment is obsolete: true
Attachment #8657885 - Flags: review?(jlund)
Attachment #8657885 - Flags: review?(garndt)
Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.
Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

https://tools.taskcluster.net/task-graph-inspector/#MzsKoxVdTI24_CaxxHGOGQ/USPhRYP8RImJ7YCwywSXSg/
Attachment #8658172 - Flags: review?(jlund)
Attachment #8658172 - Flags: review?(garndt)
Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Review of attachment 8658172 [details] [diff] [review]:
-----------------------------------------------------------------

I think we need to address the idea that branch specific caches of the objdir are handled at the task level rather than something within our build scripts.

::: testing/taskcluster/scripts/builder/build-emulator-x86.sh
@@ +31,5 @@
>  fi
>  
>  rm -rf $WORKSPACE/B2G/out/target/product/generic_x86/tests/
>  
> +gecko_objdir=$WORKSPACE/B2G/objdir-gecko-$(basename $GECKO_HEAD_REPOSITORY)

We should do this in the task definition similar to how we break out the workspace in b2g desktop.  If this happens, it means that a single workspace could contain an objdir for each branch where we want to cache that independently I think.

This way nothing has to change within the task, the gecko-objdir would always be something like /home/worker/objdir and the task has no concept of making things branch specific.
Attachment #8658172 - Flags: review?(garndt)
Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Review of attachment 8658172 [details] [diff] [review]:
-----------------------------------------------------------------

r+ to the mh bit, (b2g_build.py)

I'll leave the tc bits to garndt, specifically around the design change request he made in https://bugzilla.mozilla.org/show_bug.cgi?id=1201179#c5
Attachment #8658172 - Flags: review?(jlund) → review+
Attachment #8658172 - Attachment is obsolete: true
Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.
Attachment #8658674 - Flags: review?(garndt)
Comment on attachment 8658674 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Review of attachment 8658674 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great \o/
Attachment #8658674 - Flags: review?(garndt) → review+
backed out for bustage like https://tools.taskcluster.net/task-inspector/#O3jkdMRwR2GiiyXPL6gk2w/0
Flags: needinfo?(wcosta)
Attachment #8658674 - Attachment is obsolete: true
Flags: needinfo?(wcosta)
Backed out changeset aa3cbda42b11 (bug 1201179)
Attachment #8659270 - Attachment is obsolete: true
Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.
Comment on attachment 8659271 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Problem with previous patch is that buildot runs Python 2.6, and this version of Python does not accept string format of the form "{}", you must specify "{0}".

So the line

cfg.write('GECKO_OBJDIR={}'.format(self.objdir))

was changed to

cfg.write('GECKO_OBJDIR={0}'.format(self.objdir))

https://treeherder.allizom.org/#/jobs?repo=try&revision=a96111fec9b0
Attachment #8659271 - Flags: review?(garndt)
Comment on attachment 8659271 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Review of attachment 8659271 [details] [diff] [review]:
-----------------------------------------------------------------

looks good
Attachment #8659271 - Flags: review?(garndt) → review+
See Also: → 1203596
https://hg.mozilla.org/mozilla-central/rev/d38a3f773d07
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Component: Integration → Services
You need to log in before you can comment on or make changes to this bug.