1201372 - (some?) try builds are not clobbered

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Mike Hommey [:glandium]]

Try builds are supposed to always be clobbers. I have evidence that it's not the case for at least "B2G KK Emulator debug" builds:
https://queue.taskcluster.net/v1/task/udCRfi8rQ1WwgyHN58WNaw/runs/0/artifacts/public/logs/live_backing.log
https://queue.taskcluster.net/v1/task/GOvl0tusRZWCXD6-TirOZw/runs/0/artifacts/public/logs/live_backing.log

These two logs, incidentally on the same machine, show an error that can't happen on a clobber build.

Comment 1

[Mike Hommey [:glandium]]

Flame KK builds are also affected.

Comment 2

[Mike Hommey [:glandium]]

In fact, they are all affected.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Greg & Wander,

Is this something that is simple to add to mozilla-taskcluster? Or should this be part of bug 1171809? Or?

Comment 4

[Wander Lairson Costa]

(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> Greg & Wander,
> 
> Is this something that is simple to add to mozilla-taskcluster? Or should
> this be part of bug 1171809? Or?

bug 1171809 is surely a condition to fix this. Once it lands, we need to always send objdir purge cache messages after try builds. I don't know, however, which component is the most appropriate to send these messages.

The real problem is if we need to clobber android "out/" directory. It is shared among all branches and it is not easily fixable, as it is hardcoded in the build system.

Comment 5

[Greg Arndt [:garndt]]

It would not be difficult to add if we wanted to add it to mozilla-taskcluster.

However I'm wondering if purge cache is the right choice.  First, it's per worker type, so it will send out a message for all workers to purge a cache that are part of the same worker type.  Not sure if that's bad or not depending on how it gets implemented into the worker.

Also, the purge cache is a best effort service because of using pulse and the timing of receiving messages and processing them.  There could be a race (or a situation where a message is not received at all) and an existing cache is reused for a try job.

If we do this, we should also split out the objdir from the b2g desktop builds to be branch specific so that we are not blowing away the sources that were used.  B2G desktop reuses the same workspace, which not only includes the objdir but also all the checked out bits.


Perhaps this could be some logic built into the builder images to call `make clobber` if it's being run on try?  I think that's the make target for triggering a clobber.

Comment 6

[Wander Lairson Costa]

Attachment: Remove workspace caches for try builds. r=garndt

Try build are supposed to always be clobbered, so we remove workspace
caches for them.

Comment 7

[Wander Lairson Costa]

Comment on attachment 8660095 [details] [diff] [review]
Remove workspace caches for try builds. r=garndt

https://tools.taskcluster.net/task-graph-inspector/#PcwGjimESwGP_PYFIWG6wA/RFfjE9ZcRmuo5KYhm_vk2w/

Comment 8

[Greg Arndt [:garndt]]

Comment on attachment 8660095 [details] [diff] [review]
Remove workspace caches for try builds. r=garndt

Review of attachment 8660095 [details] [diff] [review]:
-----------------------------------------------------------------

looks good.

Comment 9

[Greg Arndt [:garndt]]

Comment on attachment 8660095 [details] [diff] [review]
Remove workspace caches for try builds. r=garndt

Review of attachment 8660095 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/taskcluster/mach_commands.py
@@ +188,5 @@
>      if "features" not in payload:
>          payload["features"] = {}
>      payload["features"]["interactive"] = True
>  
> +def remove_caches_from_task(task):

We should probably just be safe about it and also remove any scopes that contain the cache key in it.

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/b2g-inbound/rev/078d2338db65

Comment 11

[Wander Lairson Costa]

(In reply to Greg Arndt [:garndt] from comment #9)
> Comment on attachment 8660095 [details] [diff] [review]
> Remove workspace caches for try builds. r=garndt
> 
> Review of attachment 8660095 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: testing/taskcluster/mach_commands.py
> @@ +188,5 @@
> >      if "features" not in payload:
> >          payload["features"] = {}
> >      payload["features"]["interactive"] = True
> >  
> > +def remove_caches_from_task(task):
> 
> We should probably just be safe about it and also remove any scopes that
> contain the cache key in it.

Sh*t, bugzilla sent the email later and I missed this comment.

Comment 12

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/078d2338db65