1499590 - Restore periodic file updates on Taskcluster

Status: RESOLVED FIXED

(Thunderbird :: Build Config, enhancement)

Description

[Rob Lemley [:rjl]]

Port M-C's periodic file updates taskcluster task to work with Thunderbird. 

The only update we need to do for Thunderbird is the blocklist.xml update. The remaining ones (HSTS, HPKP, and remote-settings) are only in the mozilla-* trees.

The update script itself was written with Thunderbird in mind, likely won't need to be updated.

Taskcluster pieces will include adding to .cron.yml and creating a new "repo-updates" kind.

Comment 1

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Taskcluster configuration for blocklist.xml updates

- Create new cron job to run the update
- Create new kind to generate take for Thunderbird

Comment 2

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Thunderbird fixes in periodic_file_updates.sh.

Updates to periodic_file_updates.sh to complete Thunderbird support

- added comm- branch support so that repository URL's are correctly determined
- set COMMIT_AUTHOR appropriately for Thunderbird

Comment 3

[Rob Lemley [:rjl]]

Attachment 9017756 [details] is for comm-central. 
Attachment 9018622 [details] is for mozilla-central.

I've tested the changes to periodic_file_updates.sh on my dev machine and the build ran successfully. This should be it for code changes.

Waiting on Thunderbird Releng access to Phabricator (from a script), and then the token needs to be added to Taskcluster secrets.

Comment 4

[Rob Lemley [:rjl]]

Attachment: periodic-updates.txt

I ran the script (runme.sh) in a Docker container locally.
Two take-aways:
1) The modifications I made to periodic_file_updates.sh work
2) It will take about 5 minutes to run this*



The container itself was built with:
mach taskcluster-build-image periodic-updates

then a container was launched with a bash prompt:
docker run -ti --rm taskcluster/periodic-updates bash

A few environment variables normally set by Taskcluster were set manually:
# These are set by Taskcluster
export DO_BLOCKLIST=1
export PRODUCT="thunderbird"
export BRANCH="comm-central"
export REVIEWERS="rjl"

And ARC_TOKEN was set in order to test Phabricator submission:
export ARC_TOKEN="api-XXXXXXXXXXXX"

In a real world setting, $TASK_ID would be set by Taskcluster, and the token would be downloaded from Taskcluster secrets. This part of the process needs testing.

Comment 5

[Rob Lemley [:rjl]]

Comment on attachment 9017756 [details]
Bug 1499590 - Taskcluster configuration for blocklist.xml updates

this should have only been set to r? in phabricator.

Comment 6

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Add periodic-file-updates task to .cron.yml. r?dustin

Added configuration for periodic-file-updates task to comm's .cron.yml.

This cron task mirrors one found in mozilla- repositories, and was part of
comm-'s buildbot configuration that needed to be moved to Taskcluster.

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Enable taskcluster-cron on additional comm- repos. r?dustin

The periodic-file-updates cron task is meant to run on comm-central, comm-beta,
and comm-esr60 like it's mozilla- counterpart task.

Comment 8

[Rob Lemley [:rjl]]

These pieces will need to be landed in a specific order as they go against various repositories.

1) Attachment 9020195 [details] (Enable taskcluster-cron on additional comm- repos) is for ci-configuration repository. DONE
2) Attachment 9018622 [details] (Thunderbird fixes in periodic_file_updates.sh) is for Mozilla-Central. When this gets landed, I will let the responsible parties know on the Firefox end so they are aware of the change.
3) Attachment 9017756 [details] (Taskcluster configuration for blocklist.xml updates) and Attachment 9020194 [details] (Add periodic-file-updates task to .cron.yml) are both for Comm-Central and can be landed together once the above have landed.

Comment 9

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/integration/autoland/rev/479efb379f5439b2b0f0e541fa71d9911237a904
Bug 1499590 - Thunderbird fixes in periodic_file_updates.sh. r=sfraser
(no idea why automation/pulsebot didn't post this comment)

Comment 10

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/mozilla-central/rev/479efb379f54

Comment 11

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/479efb379f54

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4595e058edbd
Taskcluster configuration for blocklist.xml updates. r=darktrojan
https://hg.mozilla.org/comm-central/rev/089769dbf256
Add periodic-file-updates task to .cron.yml. r=dustin

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/e6c4c2cd5363
Follow-up: Reindent ci/repo-update/kind.yml to multiples of 4 spaces. rs=bustage-fix DONTBUILD

Comment 14

[Rob Lemley [:rjl]]

For the record, this worked as expected this morning.
https://phabricator.services.mozilla.com/D10533

Comment 15

[Rob Lemley [:rjl]]

Attachment: Backout periodic-update cron configuration while bug 1503894 is fixed

Removing the periodic-updates cron while we're working out bug 1503894. This way we can have comm-central nightly builds back.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d3c5b1d08580
Backed out changeset 089769dbf256 to remove periodic-file-updates cron task. a=backout

Comment 17

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Add periodic-file-updates task to .cron.yml. r?darktrojan

Added configuration for periodic-file-updates task to comm's .cron.yml.
This cron task mirrors one found in mozilla- repositories, and was part
of comm-'s buildbot configuration that needed to be moved to Taskcluster.

This is a reapply of an old patch. I've removed the comm-beta and comm-esr60
repositories from the configuration.

Comment 18

[Rob Lemley [:rjl]]

Attachment: Bug 1499590 - Add periodic-file-updates task to .cron.yml. r?darktrojan

Added configuration for periodic-file-updates task to comm's .cron.yml.
This cron task mirrors one found in mozilla- repositories, and was part
of comm-'s buildbot configuration that needed to be moved to Taskcluster.

This is a reapply of an old patch. I've removed the comm-beta and comm-esr60
repositories from the configuration.

Comment 19

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/e23cfc1643f8

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/b5746afc4831
Add periodic-file-updates task to .cron.yml. r=darktrojan,dustin

Comment 21

[Rob Lemley [:rjl]]

Successful update this morning, this can finally be closed!

Comment 22

[Jorg K (CEST = GMT+2)]

TB 60.5 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/31a546cf81a7dd5a39fa877f2dec4ca505b8cdf3