1201179 - [gecko] Emulator and device builds should not re-use cached object directories between branches

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Greg Arndt [:garndt]]

These builds reuse the entire workspace cache between builds which could cause builds on other branches to fail due to cache poisoning.

The workspace is very large (I think around 25gb or so last time I checked) so creating a separate one for each branch is not feasible unless we found an instance that could handle doing builds as well as a massive amount of disk space.

Perhaps the object directory itself could be branch specific (see bug 1132574 for making b2g desktop workspaces branch specific) and live outside of the typical workspace that's cached.

Comment 1

[Wander Lairson Costa]

Attachment: Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.

Comment 2

[Wander Lairson Costa]

Comment on attachment 8657885 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

https://tools.taskcluster.net/task-inspector/#lN9_irU7RLOUTjyD4YrvhQ/1
https://tools.taskcluster.net/task-inspector/#dM6Ulg8FToK8kbHrLvmQqQ/0
https://tools.taskcluster.net/task-inspector/#B7pDFj1wRv2rS9W1so2mLw/0
https://tools.taskcluster.net/task-inspector/#sAGs2XClS_a5CtVHSE1CGA/

Comment 3

[Wander Lairson Costa]

Attachment: Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.

Comment 4

[Wander Lairson Costa]

Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

https://tools.taskcluster.net/task-graph-inspector/#MzsKoxVdTI24_CaxxHGOGQ/USPhRYP8RImJ7YCwywSXSg/

Comment 5

[Greg Arndt [:garndt]]

Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Review of attachment 8658172 [details] [diff] [review]:
-----------------------------------------------------------------

I think we need to address the idea that branch specific caches of the objdir are handled at the task level rather than something within our build scripts.

::: testing/taskcluster/scripts/builder/build-emulator-x86.sh
@@ +31,5 @@
>  fi
>  
>  rm -rf $WORKSPACE/B2G/out/target/product/generic_x86/tests/
>  
> +gecko_objdir=$WORKSPACE/B2G/objdir-gecko-$(basename $GECKO_HEAD_REPOSITORY)

We should do this in the task definition similar to how we break out the workspace in b2g desktop.  If this happens, it means that a single workspace could contain an objdir for each branch where we want to cache that independently I think.

This way nothing has to change within the task, the gecko-objdir would always be something like /home/worker/objdir and the task has no concept of making things branch specific.

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8658172 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt,jlund

Review of attachment 8658172 [details] [diff] [review]:
-----------------------------------------------------------------

r+ to the mh bit, (b2g_build.py)

I'll leave the tc bits to garndt, specifically around the design change request he made in https://bugzilla.mozilla.org/show_bug.cgi?id=1201179#c5

Comment 7

[Wander Lairson Costa]

Attachment: Use custom branch gecko objdir for emulator and phone builds. r=garndt

Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.

Comment 8

[Wander Lairson Costa]

https://tools.taskcluster.net/task-graph-inspector/#pTMl7ro4To2oynEhsvzpBA/JgN97W6sRl67Oo0hUWGA5w/

Comment 9

[Greg Arndt [:garndt]]

Comment on attachment 8658674 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Review of attachment 8658674 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great \o/

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/aa3cbda42b11

Comment 11

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/b2g-inbound/rev/c12e4663ce2b

Comment 12

[Carsten Book [:Tomcat]]

backed out for bustage like https://tools.taskcluster.net/task-inspector/#O3jkdMRwR2GiiyXPL6gk2w/0

Comment 13

[Wander Lairson Costa]

Attachment: Backed out 1 changesets (bug 1201179) for bustage on a CLOSED TREE

Backed out changeset aa3cbda42b11 (bug 1201179)

Comment 14

[Wander Lairson Costa]

Attachment: Use custom branch gecko objdir for emulator and phone builds. r=garndt

Messing up with shared cached object directories for different branches
often triggers build bustage. Make the object directory unique for each
branch.

Comment 15

[Wander Lairson Costa]

Comment on attachment 8659271 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Problem with previous patch is that buildot runs Python 2.6, and this version of Python does not accept string format of the form "{}", you must specify "{0}".

So the line

cfg.write('GECKO_OBJDIR={}'.format(self.objdir))

was changed to

cfg.write('GECKO_OBJDIR={0}'.format(self.objdir))

https://treeherder.allizom.org/#/jobs?repo=try&revision=a96111fec9b0

Comment 16

[Greg Arndt [:garndt]]

Comment on attachment 8659271 [details] [diff] [review]
Use custom branch gecko objdir for emulator and phone builds. r=garndt

Review of attachment 8659271 [details] [diff] [review]:
-----------------------------------------------------------------

looks good

Comment 17

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/d38a3f773d07

Comment 18

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/d38a3f773d07