1402457 - Run buildbot's periodic_file_update job scheduled via taskcluster.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Justin Wood (:Callek)]

This bug will be 2 parts:
* Create a kind/task that can schedule in buildbot periodic file update jobs (to update hsts hpkp and blocklist)
* Create the cron.yml entry to actually schedule these, and the relevant target_tasks method.

This is so that we can do the hsts/hpkp scheduling via taskcluster, we need the job defined to do so.

Comment 1

[Justin Wood (:Callek)]

Attachment: Bug 1402457 - Add periodic file update task, but don't run it yet.

Review commit: https://reviewboard.mozilla.org/r/182808/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/182808/

Comment 2

[Justin Wood (:Callek)]

Attachment: Bug 1402457 - Actually schedule periodic file updates.

Review commit: https://reviewboard.mozilla.org/r/183314/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/183314/

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8911911 [details]
Bug 1402457 - Actually schedule periodic file updates.

https://reviewboard.mozilla.org/r/183314/#review188886

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8911347 [details]
Bug 1402457 - Add periodic file update task, but don't run it yet.

https://reviewboard.mozilla.org/r/182808/#review188884

::: taskcluster/taskgraph/transforms/job/buildbot.py:37
(Diff revision 1)
> +    run = job['run']
> +    worker = taskdesc['worker']
> +    branch = config.params['project']
> +    product = run['product']
> +
> +    worker.pop('env', None)

Is something setting `env` here?  That's sort of unfortunate..

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 8911347 [details]
Bug 1402457 - Add periodic file update task, but don't run it yet.

https://reviewboard.mozilla.org/r/182808/#review188884

> Is something setting `env` here?  That's sort of unfortunate..

It was actually done in the mozharness BBB section which I copied, I just tested that its not needed here, so I removed it entirely.

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8911911 [details]
Bug 1402457 - Actually schedule periodic file updates.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/183314/diff/1-2/

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b7658f89b02
Add periodic file update task, but don't run it yet. r=dustin
https://hg.mozilla.org/integration/autoland/rev/1009ff962050
Actually schedule periodic file updates. r=dustin

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/7b7658f89b02
https://hg.mozilla.org/mozilla-central/rev/1009ff962050

Comment 9

[Ryan VanderMeulen [:RyanVM]]

I was asked by Callek to disable this cron job for now due to some unforeseen issues with the amount of work it's doing.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/935eca685536
Disable the TC periodic-update cron job due to more work being scheduled than expected. r=Callek, a=me

Comment 11

[Justin Wood (:Callek)]

Attachment: Properly select the right job

This reverts RyanVM's `when` clearing that I requested, and fixes the underlying issue.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f6f706a84c45
fix the file update target task to not schedule the world. r=aki

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/f6f706a84c45

Comment 14

[Justin Wood (:Callek)]

Attachment: [bb-configs] Disable the daily-* scheduler, which only schedules periodic file updates.

Comment 15

[Kim Moir [:kmoir] ET]

Comment on attachment 8917524 [details] [diff] [review]
[bb-configs] Disable the daily-* scheduler, which only schedules periodic file updates.

universal_master_sqlite.cfg needs to be modified too

Comment 16

[Justin Wood (:Callek)]

https://hg.mozilla.org/build/buildbot-configs/rev/846dffdeb159765afae73b4e6fbbad4c4f749234
https://hg.mozilla.org/build/buildbot-configs/rev/40289a691194e9bf5d43ff095512b328355dd48b