1191991 - Make tc-vcs cache warming happen on a schedule

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Selena Deckelmann :selenamarie :selena]

We have a shell script that can warm caches. Cleaning this up and putting it on a schedule.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Got a test run in Heroku, but only 14 of the 33 tasks were launched. Have some bad theories about why, but nothing definitive.

Comment 2

[Selena Deckelmann :selenamarie :selena]

Attachment: https://github.com/taskcluster/taskcluster-vcs/pull/21

Comment 3

[Pete Moore [:pmoore][:pete]]

Until we have a taskcluster scheduler piece, it might be worth looking at RelEng's "Bad Penny" RelEngAPI periodic scheduling service:

https://api.pub.build.mozilla.org/docs/deployment/badpenny/
https://github.com/mozilla/build-relengapi/blob/master/relengapi/lib/badpenny.py

Might be worth chatting with Dustin about - I'm not sure what else is running in Bad Penny, but if this is mozilla-specific (i.e. an integration piece that should not live inside taskcluster) this might be a good candidate. If it is something generic for taskcluster, we probably shouldn't depend on RelEngAPI, as atm Taskcluster is a relatively standalone-tool.

NI'ing Jonas for his input too.

Pete

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

@pmoore,
I'm not sure it's clear in bug 1149789 (taskcluster-hooks) but I think adding an
optional "schedule" property would be fine. So that is where I would put periodic tasks.

Well, actually I would put a trigger-task in hooks, and then have that trigger-tasks
clone hg and use in-tree configuration to decide when to run what tasks. Then run the
trigger-tasks maybe 4 times a day.

Comment 5

[Selena Deckelmann :selenamarie :selena]

Attachment: https://github.com/taskcluster/taskcluster-vcs/pull/44

Comment 6

[Selena Deckelmann :selenamarie :selena]

Reocurring task scheduled for non-emulator repos: https://tools.taskcluster.net/hooks/#taskcluster/tc-vcs-refresh

Comment 7

[Selena Deckelmann :selenamarie :selena]

Automated runs are working! https://tools.taskcluster.net/task-inspector/#d4qYz0RsQPSmCRjwcO1hpw/0 and task-graph: https://tools.taskcluster.net/task-graph-inspector/#XNU0Ye73RVWcBeHXWhoBkg/

Comment 8

[Greg Arndt [:garndt]]

Comment on attachment 8695518 [details] [review]
https://github.com/taskcluster/taskcluster-vcs/pull/44

Left some comments in the PR, but all of them really are just suggestions and even if it landed as is wouldn't be a problem.

Comment 9

[Selena Deckelmann :selenamarie :selena]

This happened!  This is scheduled and visible in tools.taskcluster.net/hooks