1706056 - getHSTSPreloadList.js takes too long

Status: VERIFIED FIXED

(Core :: Security: PSM, task)

Description

[Julien Cristau [:jcristau]]

Updating the HSTS preload list is now taking roughly 5 hours, with about 100k domains on the list. Can we somehow improve that?

Comment 1

[Dana Keeler (she/her) [:keeler]]

Attachment: Bug 1706056 - increase the parallelism of getHSTSPreloadList.js r?#releng

Before this patch, getHSTSPreloadList.js would have no more than 250 requests
in flight at any given time. In the past, this limit was 500. After a refactor
that inadvertantly completely removed the limit, it was reduced to 250.
Presumably increasing it to 500 again will not have negative effects and will
allow the script to run faster.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/042c5a186d30
increase the parallelism of getHSTSPreloadList.js r=releng-reviewers,jmaher

Comment 3

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/042c5a186d30

Comment 4

[Julien Cristau [:jcristau]]

Monday's update (https://firefox-ci-tc.services.mozilla.com/tasks/flPOTfqjQoigj1eFY1kQbg) ran for 297 minutes.
Today's update (https://firefox-ci-tc.services.mozilla.com/tasks/IjAP2s7fQ3yggIpmwgot3A) finished in 157 minutes.

Thanks Dana!

Comment 5

[Julien Cristau [:jcristau]]

Attachment: Bug 1706056 - reduce max-run-time for repo-update task. r?#releng,ryanvm

Now that getHSTSPreloadList.js runs with increased parallelism, it goes
much faster and we don't need to let the task run for 6 hours.

The timeout was 3 hours for a while, then bumped to 5 hours in bug
1570646, then to 6 hours in bug 1705639 (with 250 in-flight requests).
The new run time with 500 in-flight requests seems to be around 2.5
hours, so pick 4 hours as a compromise.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1650540c413b
reduce max-run-time for repo-update task. r=RyanVM,releng-reviewers,bhearsum DONTBUILD

Comment 7

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/1650540c413b

Comment 8

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7e05197cbf4e
https://hg.mozilla.org/releases/mozilla-beta/rev/e5d9cf590ce2

Comment 9

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/405eaeaadb98
https://hg.mozilla.org/releases/mozilla-esr78/rev/1d47c7b0d2d0