Closed Bug 1252948 Opened 9 years ago Closed 8 years ago

cron.yml for periodic in-tree tasks

Categories

(Firefox Build System :: Task Configuration, task)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jonasfj, Assigned: dustin)

References

Details

Attachments

(2 files, 2 obsolete files)

Bug 1252948: remove unused --revision-hash;
59 bytes, text/x-review-board-request
garndt
: review+
Details
Bug 1252948: support for periodic taskgraphs;
59 bytes, text/x-review-board-request
jonasfj
: review+
Details
We should have a nightly.yml or cron.yml file in-tree that specifies tasks to run periodically, so developers can add tests that only needs to run periodically. This at a minimum requires treeherder to auto-create result-sets. It could still use mozilla-taskcluster rather block on bug 1183996. This likely amounts to making a hook with: task.scopes = ['assume:moz-tree:level:1', 'scheduler:create-task-graph'] And a command that clones hg, looks in cron.yml, builds one or more task-graphs and creates them by calling the scheduler through the proxy.
Depends on: 1199291
I would like to have a hand in implementing a way to have periodic in-tree tasks. I'm working with bug 1282180 involved with scheduling fennec nightlies. I think this would be beneficial for our use case since we can store information about scheduling nightly releases in-tree and have the information be more transparent than directly via a hook. I've helped with the creation of the hooks service last summer, and this seems like it would be a good way to contribute back. I don't have any context behind the treeherder and the result-sets, and I'm wondering if there's anyone that would also benefit from having in-tree periodic scheduling.
You're welcome to hack on it! I'm sure a general solution would be useful outside of gecko, and I'm certain that Firefox devs would find interesting uses for a .cron.yml -- maybe once-weekly valgrind runs, for example..
No longer depends on: 1285022
What kinds of guarantees should we put on this cron file? I've thought about a few problems including unscheduled tasks and a lack of re-triggering. There is a possibility of having tasks that are not scheduled because there are no hard guarantees on when tasks are run. For example, a `parse_cron.py` runs on a hook of period T=15 minutes. There can be some delay D >= 0 in the execution of the registered hook. `parse_cron.py` will schedule any task that falls within time t and t+T. When there is no delay, every task defined within the cron file will be added to the task queue. For any delay of D>0, there is a small gap in which any task defined between [t+T, t+T+D] will not get scheduled. This problem is not an issue within the hook service itself because it maintains a queue in azure of tasks to be run. At every period T, the hook service will create all tasks that are past their scheduled time of execution. The effects of a delay can be mitigated by changing when and how often we parse the cron file. For example, we could say that tasks scheduled on the top of the hour with a minimum period of one hour are unlikely to encounter any problems. The hook that runs the `parse_cron.py` script would then run once an hour at the 5th minute. This hook could see a delay of up to 55 minutes (which includes the overhead of setting up the docker container and hg) and still run the tasks within the cron file. We truncate any information about the minute and second that it runs. Alternatively, we rewrite hooks as a separate service to handle this particular use case or integrate polling mercurial into hooks. The other issue is that we can't easy re-trigger a job if it fails. Someone could manually paste the task definition into the task creator, but that process would take several steps as opposed to triggering a webhook. I think this is something that is best handled by registering the task outside of the tree.
One option is for parse_cron.py to keep some kind of state. Perhaps the easiest way to do that is to use an Azure table, credentials for which you can get from the auth service. On the other hand, I don't know of any decent Azure libraries in Python, so you might be better off writing parse_cron.js instead. As for re-triggering, we have the same issue with per-push decision tasks, right? I expect we wouldn't want to re-trigger the cron task if it fails; we might want to re-trigger the decision tasks that the cron task creates, though.
Actually, I remembered that you can poll a hook for its status (/hooks/<hookGroupId>/<hookId>/status) to get its last and next time it fires. That information along with the period, might be enough to solve this without resolving to new state for this problem.
Assignee: nobody → dustin
Dustin, is there any progress here? Manual scheduling has landed (bug 1245566), the green-ness of the runs is slowly improving (bug 1248365, bug 1320894), it found its first previously-unknown defect (bug 1320094) and is being used "for real" to test gcc 6 portability fixes (bug 1316556, bug 1232696). It would be great to have this last step done.
I probably won't get a chance to look until after the all-hands.
:dustin pinged me with regard to requirements for Nightly Graph. I'd like some way to basically do a "decision-graph-like-thing"... Where we get revision-of-tree upfront (so viewable in treeherder) We need to be able to tie scopes that are different than other cron entries, and different than the overall tree's decision task to it (so we can do things like allow a nightly-only signing scope, or allow uploading to archive.m.o, etc) in a way that won't allow arbitrary other tasks to do so. Having some way to trigger these cron'd etries outside of their cron'd time is also an ideal for me. (in-case a nightly busts because of code issues, we can trigger a new one, out of band). I have a few potential designed solutions in mind, but some complications with each one that I haven't mentally solved. We can discuss on them if the solution-side matters here.
Notes from our conversation: Design ------ Every 15 minutes, the hooks service schedules a "cron task" which runs `./mach taskgraph cron --repo=..` for each tree. That hook runs with scopes sufficient to run any taskgraph -- nightly release, valgrind, whatever. The decision tasks it creates have the minimal scopes required to do their own thing, and their parameters dictate what that thing is. Those parameters would include a fixed revision calculated within the cron task. The cron task will look up its $TASK_ID's `created` timestamp and round down to the nearest 15 minutes. That way, even if task execution is delayed, each interval's tasks will eventually get created. We'll set a short deadline to avoid piling up too many pending cron tasks in an outage. Scopes ------ This is pretty deeply into the "it's all arbitrary code" world, since the cron task runs mach. It can basically do whatever it wants with the scopes it runs as. A malicious actor with level-3 access gets to define "what it wants". If they would like to create a malicious nightly-signed build, they can simply commit their malfeasance and wait for the next nightly to be built. That's already the case with Buildbot. That sets a pretty low floor for access control. Providing that malicious actor with the extra ability to reschedule the nightly, or to schedule some other "cron task" that will make a nightly release, doesn't materially help them. There is probably room for some sanity checks here, within the chain-of-trust validation, but it's hard to see how they could be more than sanity checks. Aki, thoughts? Retriggering ------------ In the case Callek mentioned, you will want to run a *different* decision task, with a later revision. I think the easiest way to do this is to create a "force" cron task, something like `./mach taskgraph cron --force-run=nightly-android --repo=..` which would force the nightly-android cron entry to trigger regardless of the current time. Most users won't have the scopes to create such a task, since it has a wide scope-set. In the short term, relengers will have this scope and can use something like tctalker to create these force cron tasks. Longer-term, someone can develop a web service (maybe in relengapi or ship-it) that can do a careful privilege escalation to create force tasks. That will be security-sensitive, and is passing arguments to a shell script, so it will need some careful coding!
So, this is my work-in-progress. It works! I have a hook: https://tools.taskcluster.net/hooks/#project-releng/dustin-cron which generated a "cron task" https://tools.taskcluster.net/task-inspector/#F3xpwpsKQRGtOl95G9PL6Q/ which, in turn, generated a decision task https://tools.taskcluster.net/task-inspector/#XxSCsGSSSn2dpuTB8e7p-g/0 which failed.. The idea is that the hook will run once every 15m, for each repository, with a scope named after that repository. I'll write a little utility to set those hooks up. Each cron job can create a decision task that will run with `assume:repo:<repo>:cron:<job_name>`, allowing us to give extra scopes to specific cron tasks (noting that it's not difficult to "steal" those scopes if you have commit access to the repo!). There's still a bunch of mess to solve, so no r? yet. * treeherder routes / metadata * all those irrelevant decision arguments * finding the right mix of defaults in `base.py` to allow easy implementation of crontasks * in-tree docs * update to taskcluster docs for the `assume:repo:...:cron:...` scope format
https://github.com/taskcluster/taskcluster-docs/pull/140 for the docs. In thinking about how to land this, maybe it would be best to land it on date first (with an adjustment to nightlies.py) since that's where we're running nightlies now. Then I can merge the result to an integration branch with Callek's help :)
Comment on attachment 8820913 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/100296/#review101036 The code indicates that nightlies only run on m-c but currently they are running on date via a hook. Should the date branch also be included so we can continue to test on that branch and use this new functionality?
Attachment #8820913 - Flags: review?(kmoir) → review+
I want to explicitely ask. Could this be used to run PGO builds? IIUC we use SETA to control when to run Linux PGO builds (I could be wrong; I have not been following as closely on the current state).
(In reply to Kim Moir [:kmoir] from comment #17) > The code indicates that nightlies only run on m-c but currently they are > running on date via a hook. Should the date branch also be included so we > can continue to test on that branch and use this new functionality? Good eye -- yes. I didn't think about landing on date until I'd submitted the review request, but I'll make that change before landing there. (In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #18) > I want to explicitely ask. Could this be used to run PGO builds? IIUC we use > SETA to control when to run Linux PGO builds (I could be wrong; I have not > been following as closely on the current state). That's the idea, yes. Also, weekly valgrind runs. Exciting new possibilities!!
Keywords: leave-open
Comment on attachment 8820913 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/100296/#review101710 ::: taskcluster/taskgraph/cron/jobs/nightlies.py:22 (Diff revision 1) > + self.symbol = symbol > + self.run_at = run_at > + self.target_tasks_method = target_tasks_method > + > + def should_run(self, params): > + if params['project'] != 'mozilla-central': as kim said, this should include `date`
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review101708 Pretty close, and very nice! We should be careful even when deploying to date, that we disable the hooks (e.g. with a `/bin/false && ...` rather than deletion) when we enable the cron for date. ::: taskcluster/docs/cron.rst:55 (Diff revision 2) > +with the ``taskcluster.taskgraph.cron:define_cron_job`` function, which takes a > +name and a job. > + > +The ``should_run`` method should decide whether to run the task. It gets > +passed a set of parameters defined in > +``taskcluster/taskgraph/cron/jobs/nightlies.py``. file not present in this changeset. I'm presuming of course we don't want the generic cron parameters to be defined in nightlies.py ? ::: taskcluster/taskgraph/cron/__init__.py:24 (Diff revision 2) > +from . import jobs > +from ..create import create_task > + > +logger = logging.getLogger(__name__) > + > +session = requests.Session() can we avoid `requests.Session()` being at module scope? ::: taskcluster/taskgraph/cron/__init__.py:93 (Diff revision 2) > + time = datetime.datetime.strptime(created, '%Y-%m-%dT%H:%M:%S.%fZ') > + > + # round down to the nearest 15m > + minute = time.minute - (time.minute % 15) > + time = time.replace(minute=minute, second=0, microsecond=0) > + logger.info("cron time is {}".format(time)) nit: Calculated cron schedule time is {} ::: taskcluster/taskgraph/cron/jobs/base.py:22 (Diff revision 2) > + """A CronJob instance determines whether or not to run, and creates zero > + or more tasks when it runs.""" > + > + # Subclasses should implement both should_run and run: > + > + def should_run(self, params): overall, I'd love some validationy way of getting the time "should run" info. E.g. a "run_at_time(...)" because, right now, we do cron-class specific information on arbitrary time, which just happens to be normalized to 15 min. With this format on time there is no good way to validate against invalid inputs, and thus unexpected "hey, why didn't this run". The counterpoint to this, though, is its less flexible (there may be cron stuff whose decision to run is based on things other than time, or for things to run weekly, daily, monthly, 4 hours a day, etc.) which makes logic a bit harder to express. I leave it to you which side of this tradeoff we fall on for now though. ::: taskcluster/taskgraph/cron/jobs/base.py:83 (Diff revision 2) > + th = task['extra']['treeherder'] > + th['groupSymbol'] = 'cron' > + th['symbol'] = symbol > + > + # add a scope based on the repository, with a cron:<job_name> suffix > + # TODO: update namespace doc too I don't want to land with a TODO if possible.
Attachment #8816604 - Flags: review?(bugspam.Callek) → review-
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review101708 > can we avoid `requests.Session()` being at module scope? I could, but then it needs to be passed along the call chain to `create_decision_task`, and that requires that jobs pass it along. Is there a reason not to put it here? It *is* a global shared resource, after all! Another option is to write a `get_session` function that returns a singleton session, but that's effectively the same thing. > overall, I'd love some validationy way of getting the time "should run" info. > > E.g. a "run_at_time(...)" because, right now, we do cron-class specific information on arbitrary time, which just happens to be normalized to 15 min. > > With this format on time there is no good way to validate against invalid inputs, and thus unexpected "hey, why didn't this run". > > The counterpoint to this, though, is its less flexible (there may be cron stuff whose decision to run is based on things other than time, or for things to run weekly, daily, monthly, 4 hours a day, etc.) which makes logic a bit harder to express. > > I leave it to you which side of this tradeoff we fall on for now though. If I understand correctly, you're saying that it would be easy to write a `should_run` that tries to run at 10 minutes past the hour, but in fact would never run. Probably the easiest way to prevent that is to provide some utility functions for time matching which will raise exceptions if the inputs would never run. > I don't want to land with a TODO if possible. It's already done, too :)
Comment on attachment 8820913 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/100296/#review101930 changes to this patch look good (for completeness)
Attachment #8820913 - Flags: review+
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review101934 Looks good to me now, the Session issue can be resolved without needing me to look again.
Attachment #8816604 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8820912 [details] Bug 1252948: remove unused --revision-hash; https://reviewboard.mozilla.org/r/100294/#review102026
Attachment #8820912 - Flags: review?(garndt) → review+
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review102306 ::: taskcluster/docs/cron.rst:38 (Diff revision 4) > +.. important:: > + > + The individual cron scopes are a useful check to ensure that a job is not > + accidentally doing something it should not, but cannot actually *prevent* a > + job from using any of the scopes afforded to the cron task itself (the > + ``..cron:*`` scope). This is simply because the cron task runs aribtrary aribtrary is a typo ::: taskcluster/taskgraph/cron/jobs/base.py:39 (Diff revision 4) > + """ > + raise NotImplementedError > + > + # Utilities for subclasses > + > + def match_utc(self, params, hour=None, minute=None): Why does this belong here? If it doesn't use protected members it has no business being a member of abstract class. Given that the CronJob is intended to be subclassed and: A) it doesn't have any _protected members_ (as it doesn't have a _constructor_), and, B) it doesn't have any _private virtual_ methods I don't see any reason it shouldn't a _pure virtual class_ or _interface_ if you will. ::: taskcluster/taskgraph/cron/jobs/base.py:55 (Diff revision 4) > + return False > + return True > + > + def make_decision_task(self, params, symbol, arguments=[], head_rev=None): > + """Generate a basic decision task, based on the root > + .taskcluster.yml""" What is the thinking in encouraging cron job to create decision tasks based on `.taskcluster.yml`? Do we primarily want cron jobs to run a set of _try flags_? If so wouldn't it be simpler to have a `cron.yml` file on the form: ```yaml --- schedule: '* * * * * 0' # A string specifying the interval projects: # Empty to run on all projects - gecko - <list of projects it only runs for> onChange: # List of file patterns that must have changed # Leave empty to run regardless of files changed - toolkit/**.cpp - gfx/**.* - chrome/*.js tryFlags: '-b linux -t crashtest' # try flags to run --- schedule: '...' ... ``` --- I guess I'm not really suggesting that we replace the CronJob python class, but if decision task based things with different try flags is going to be the common use-case perhaps we can do something simpler. Perhaps: ```yaml --- implementation: taskcluster.taskgraph.cron.jobs.default schedule: '* * * * * 0' # A string specifying the interval projects: # Empty to run on all projects - gecko - <list of projects it only runs for> onChange: # List of file patterns that must have changed # Leave empty to run regardless of files changed - toolkit/**.cpp - gfx/**.* - chrome/*.js tryFlags: '-b linux -t crashtest' # try flags to run --- ... ``` Where `taskcluster.taskgraph.cron.jobs.default` has two functions `should_run` and `run`. Both are given all the keys from the entry in `cron.yml`. I argue for functions I don't see why we should allow them to be stateful. The `taskcluster.taskgraph.cron.jobs.default:should_run` function would know how to read and interpret the `schedule`, `projects` and `onChange` keys. The `taskcluster.taskgraph.cron.jobs.default:run` function would know how to interpret the `tryFlags` key. Then if you just want to run a specific try flag on an interval, then that's pretty easy. And if you want something more advanced you can make your own implementation of `should_run` and `run`, which can then be reused by anyone who like the logic you came up with. --- Also I would consider renaming the `should_run` to `filter`, but that might be crazy...
Attachment #8816604 - Flags: review?(jopsen) → review-
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review102306 > Why does this belong here? > If it doesn't use protected members it has no business being a member of abstract class. > > Given that the CronJob is intended to be subclassed and: > A) it doesn't have any _protected members_ (as it doesn't have a _constructor_), and, > B) it doesn't have any _private virtual_ methods > I don't see any reason it shouldn't a _pure virtual class_ or _interface_ if you will. It's a utility method. This isn't Java - there's no such thing as protected members. It is just a parent class (a convenient way to document the methods it's expected to have), with some utility methods. I could move those elsewhere as regular functions, but then implementers just have to import them. I guess I'm not sure what you're suggesting. While there is zope.interfaces to support interfaces in Python, we haven't vendored it and I'm not sure we would want to. There's the `abc` module in the standard library, but using that seems like overkill here. This "duck typing" class inheritance is pretty common in Python. > What is the thinking in encouraging cron job to create decision tasks based on `.taskcluster.yml`? > > Do we primarily want cron jobs to run a set of _try flags_? > If so wouldn't it be simpler to have a `cron.yml` file on the form: > ```yaml > --- > schedule: '* * * * * 0' # A string specifying the interval > projects: # Empty to run on all projects > - gecko > - <list of projects it only runs for> > onChange: > # List of file patterns that must have changed > # Leave empty to run regardless of files changed > - toolkit/**.cpp > - gfx/**.* > - chrome/*.js > tryFlags: '-b linux -t crashtest' # try flags to run > > --- > schedule: '...' > ... > ``` > > --- > > I guess I'm not really suggesting that we replace the CronJob python class, but if decision task based things with different try flags is going to be the common use-case perhaps we can do something simpler. > > Perhaps: > ```yaml > --- > implementation: taskcluster.taskgraph.cron.jobs.default > schedule: '* * * * * 0' # A string specifying the interval > projects: # Empty to run on all projects > - gecko > - <list of projects it only runs for> > onChange: > # List of file patterns that must have changed > # Leave empty to run regardless of files changed > - toolkit/**.cpp > - gfx/**.* > - chrome/*.js > tryFlags: '-b linux -t crashtest' # try flags to run > > --- > ... > ``` > > Where `taskcluster.taskgraph.cron.jobs.default` has two functions `should_run` and `run`. Both are given all the keys from the entry in `cron.yml`. > I argue for functions I don't see why we should allow them to be stateful. > > The `taskcluster.taskgraph.cron.jobs.default:should_run` function would know how to read and interpret the `schedule`, `projects` and `onChange` keys. The `taskcluster.taskgraph.cron.jobs.default:run` function would know how to interpret the `tryFlags` key. > > Then if you just want to run a specific try flag on an interval, then that's pretty easy. And if you want something more advanced you can make your own implementation of `should_run` and `run`, which can then be reused by anyone who like the logic you came up with. > > > --- > Also I would consider renaming the `should_run` to `filter`, but that might be crazy... No, cron jobs are not running try flags. Try flags are an abomination! Most cron jobs will run decision tasks with adjusted options, and in most cases that will be a different `target_task_method`, but I want to leave open the opportunity to do fancy things. Regarding the proposed YAML, `onChange` is pretty hard to implement -- certainly not something I want to support in the short term. So, I guess it boils down to this: do we try to specify most/all cron jobs in a `.cron.yml` file, or in Python files under `taskcluster/taskgraph/cron/jobs`? I argue for the latter, based on a few observations: * we are likely to have a lot of similar cronjobs, for example nightlies for different projects at different times. It's a lot easier to parameterize that in Python than to implement some kind of unique macro system for YAML * schedules will probably be all manner of "interesting": "every 6 hours unless a change has landed on branch X in the last 6 hours". Not a great fit for YAML. * probably half of the cron jobs will be "fancy" in one way or another -- not running decision tasks. Let's not put half of the cron jobs in `.cron.yml` and hide the others in Python * we already have a few YAML "dialects" defined, and adding one more would increase the confusion.
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review102306 > No, cron jobs are not running try flags. Try flags are an abomination! Most cron jobs will run decision tasks with adjusted options, and in most cases that will be a different `target_task_method`, but I want to leave open the opportunity to do fancy things. > > Regarding the proposed YAML, `onChange` is pretty hard to implement -- certainly not something I want to support in the short term. > > So, I guess it boils down to this: do we try to specify most/all cron jobs in a `.cron.yml` file, or in Python files under `taskcluster/taskgraph/cron/jobs`? I argue for the latter, based on a few observations: > > * we are likely to have a lot of similar cronjobs, for example nightlies for different projects at different times. It's a lot easier to parameterize that in Python than to implement some kind of unique macro system for YAML > * schedules will probably be all manner of "interesting": "every 6 hours unless a change has landed on branch X in the last 6 hours". Not a great fit for YAML. > * probably half of the cron jobs will be "fancy" in one way or another -- not running decision tasks. Let's not put half of the cron jobs in `.cron.yml` and hide the others in Python > * we already have a few YAML "dialects" defined, and adding one more would increase the confusion. Bug 1328722 provides a good example of why we want to base decision tasks on `.taskcluster.yml`: then we can do chain of trust configuration in one place.
Blocks: cot-v2
@dustin, let's talk Monday, my concern is mostly that creating cron jobs involved writing code, instead of just being a declaration of what you want to run. I agree try flags aren't great, but then maybe it's a declaration of the decision task you want to run. (note: OOP terms are still applicable in python, even if concepts aren't enforced by the language).
Comment on attachment 8823711 [details] Bug 1252948: make CronJob an ABC, move utilities to util module; https://reviewboard.mozilla.org/r/102230/#review103534 This is definitely better... Let's chat monday about the other part.
Attachment #8823711 - Flags: review?(jopsen) → review+
So I think "the other part" is about describing cron tasks as code instead of as YAML. One of the strengths of the in-tree task-graph generation is that it does not make a strong distinction between the two: it's all in the same repo, and some tasks are easily accomplished with a YAML change, while others require code changes. We had the advantage in designing that system of lots of experience with changes to what gets built and how -- so a good sense for how often different kinds of changes will be required. We don't have that experience with periodic tasks. Another advantage is, since it's all in the same repo, it's very easy to refactor without compatibility concerns. As I said above, I think we will always need the possibility of designing a cron task in code. So *only* supporting that arrangement is strictly simpler. All of which is to say, I'd like to land something like this and see where folks go with it. If we do end up with a lot of formulaic tasks that would map well to configuration files, we can add that support later. I'm sorry to make my argument to a review that comes down to "no, I'd like to land it as-is".. but that's my feelings on the matter :)
Comment on attachment 8816604 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/97272/#review104974 dustin I discussed using a single cron.yml file with declarative schedule, so make it easier to get an overview.
Attachment #8816604 - Flags: review?(jopsen) → review-
Attachment #8816604 - Attachment is obsolete: true
Attachment #8823711 - Attachment is obsolete: true
Comment on attachment 8820913 [details] Bug 1252948: support for periodic taskgraphs; https://reviewboard.mozilla.org/r/100296/#review106490 ::: .taskcluster.yml:16 (Diff revision 5) > # side to whatever scopes are set by mozilla-taskcluster. > - queue:* > - docker-worker:* > - scheduler:* > > -# Available mustache parameters (see the mozilla-taskcluster source): > +# This file undergoes substitution to create tasks. For on-push tasks, that This file is a contract between `gecko` and `mozilla-taskcluster`, I'm not very happy about using it for other purposes... But I'll let that slide for now, as that's already the case. Random Note: I really want to use json-e for this file too, hence, why we need a python implementation too. ::: taskcluster/docs/cron.rst:28 (Diff revision 5) > +Decision Tasks > +.............. > + > +For ``job.using`` "decision-task", tasks are created based on > +``.taskcluster.yml`` just like the decision tasks that result from a push to a > +repository. They run with a distinct taskGroupId, and are free to create NIT: `taskGroupId` is an identifier...
Attachment #8820913 - Flags: review?(jopsen) → review+
Depends on: 1332421
Pushed by dmitchell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/ce16140b93fa remove unused --revision-hash; r=garndt https://hg.mozilla.org/integration/autoland/rev/459fd779b235 support for periodic taskgraphs; r=Callek,jonasfj,kmoir
Callek, I added crontasks for all of the level-2 and higher branches for which there are scopes. They are in the `project-releng` hook group and start with `cron-task-`. None of them are scheduled yet, so no harm is possible. What are the next steps to deploying this? I am using [tcadmin](https://github.com/taskcluster/taskcluster-admin) to set up the hooks so I can easily add the schedule to all of them when you give me the word.
Flags: needinfo?(bugspam.Callek)
Here are the commands I ran, by the way; ./tcadmin make-gecko-cron-hook projects/alder alder 2 ./tcadmin make-gecko-cron-hook projects/ash ash 2 ./tcadmin make-gecko-cron-hook projects/cedar cedar 2 ./tcadmin make-gecko-cron-hook projects/cypress cypress 2 ./tcadmin make-gecko-cron-hook projects/date date 3 ./tcadmin make-gecko-cron-hook projects/elm elm 2 ./tcadmin make-gecko-cron-hook projects/graphics graphics 2 ./tcadmin make-gecko-cron-hook projects/jamun jamun 2 ./tcadmin make-gecko-cron-hook projects/larch larch 2 ./tcadmin make-gecko-cron-hook projects/oak oak 2 ./tcadmin make-gecko-cron-hook projects/pine pine 2 ./tcadmin make-gecko-cron-hook mozilla-central mozilla-central 3 ./tcadmin make-gecko-cron-hook incubator/stylo stylo 2 ./tcadmin make-gecko-cron-hook integration/autoland autoland 3 ./tcadmin make-gecko-cron-hook integration/mozilla-inbound inbound 3 ./tcadmin make-gecko-cron-hook integration/fx-team fx-team 3 ./tcadmin make-gecko-cron-hook releases/mozilla-aurora mozilla-aurora 3 ./tcadmin make-gecko-cron-hook releases/mozilla-beta mozilla-beta 3 ./tcadmin make-gecko-cron-hook releases/mozilla-esr45 mozilla-esr45 3 ./tcadmin make-gecko-cron-hook releases/mozilla-release mozilla-release 3 in src/make-gecko-cron-hook.js, I have the schedule commented out, so the only thing required to set that up is to remove the comment and re-run the commands above (and commit and push the change to tcadmin).
:dustin, how can we use this cron.yml feature to run mochitest-valgrind on mozilla-central once/day (the nightly schedule)?
Flags: needinfo?(dustin)
Just add a new entry to cron.yml. But note comment 53 -- the hooks aren't set up yet.
Flags: needinfo?(dustin)
(In reply to Dustin J. Mitchell [:dustin] from comment #55) > Just add a new entry to cron.yml. But note comment 53 -- the hooks aren't > set up yet. Let's coordinate in #tcmigration in just a few minutes but heres my current thinking as to potential blockers: We need different schedules for nightly on aurora/central (and ideally also different for date) Before we enable we should also setup the scopes properly for the cron-format for nightlies, I don't envision this part is hard Also before enabling lets manually run the cron.yml on date, and on a tree that *has* the cron.yml code but no nightlies, and lastly on a tree that does not have the cron.yml code at all, to verify all 3 cases work fine. We can first enable this on any trees that don't have nightlies scheduled + date. Once we have the per-project schedule support we can enable on m-c. Lastly we should make `projects` work like `run_on_projects` for the normal taskgraph, so we can specify projects the same way in both places. I envision the project-specific schedule to work in cron.yml via `by-project:`
Flags: needinfo?(bugspam.Callek)
Depends on: 1334167
I moved those issues and a few others to bug 1334167 so I can post a new review.
OK, bug 1334167 is autolanded. Callek, once that's merged to date, I can update the cron hooks with actual schedules, confident that they won't trigger unexpected task graphs. Then I can help transition from the hand-built crontasks to in-tree crontasks. Sorry for all the delay on this!
Flags: needinfo?(bugspam.Callek)
As discussed on IRC today, once the followups merge into central we'll have the hook enabled on central and I'll do a date merge and enable on date, we'll also turn on the nightlies via cron on date, and see how that fares. if all goes well, we'll enable the nightlies via cron on central and aurora and enable the hook everywhere.
Flags: needinfo?(bugspam.Callek)
OK, I just enabled all of the hooks to run every 15 minutes. Nothing should happen (in fact, tasks for repos where this hasn't merged yet will fail, but that's OK). When ready, Callek and the nightly team can land changes to set `when` to something other than `[]` and disable the old nightly hooks.
[task 2017-02-03T21:24:32.856249Z] The taskgraph command does not accept the arguments: cron --base-repository=https://hg.mozilla.org/mozilla-unified --head-repository=https://hg.mozilla.org/projects/cypress --head-ref=default --project=cypress --level=2 and nobody gets hurt, or even emailed.
https://tools.taskcluster.net/task-inspector/#QGvOdf9bRBW5ni06NBHFJw/0 https://tools.taskcluster.net/task-inspector/#TktBYOUPTTWY499-ABhV1w/0 both decided not to run any jobs (as expected).
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
See Also: → 1337300
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: