1891052 - Periodic File Updates is slow to run

Status: NEW

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Mark Banner (:standard8)]

I was recently testing out changes to periodic_file_updates.sh locally, and noticed it was slow to run the remote settings updates. On Treeherder, it looks like it takes almost three hours currently.

This task is only run twice a week per tree (via crontab), but it still seems it may be worth speeding it up.

For the remote settings part it is generally doing this for every collection:

• Download the current version of the dump for the collection from hg.mozilla.org
• Download the latest version of the collection from remote settings
• Compare the differences

At the end if there are differences it clone the mercurial repository and apply them.

I think this could be speeded up by first cloning the mercurial repository, and then obtaining the dumps from the repository. This would save multiple round-trips to hg.mozilla.org.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

AFAIK, the main source of slowness is the HSTS/HPKP updating. The job runs in 10-15min on mozilla-release where we skip that part. We have bug 1810856 on file for splitting that part into a separate task, but this script is borderline unowned these days.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:ahal, could you have a look please?

For more information, please visit BugBot documentation.