Closed Bug 1706056 Opened 3 years ago Closed 3 years ago

getHSTSPreloadList.js takes too long

Categories

(Core :: Security: PSM, task)

task

Tracking

()

VERIFIED FIXED
90 Branch
Tracking Status
firefox-esr78 --- fixed
firefox89 --- fixed
firefox90 --- verified

People

(Reporter: jcristau, Assigned: keeler)

References

Details

(Whiteboard: [psm-assigned])

Attachments

(2 files)

Updating the HSTS preload list is now taking roughly 5 hours, with about 100k domains on the list. Can we somehow improve that?

Before this patch, getHSTSPreloadList.js would have no more than 250 requests
in flight at any given time. In the past, this limit was 500. After a refactor
that inadvertantly completely removed the limit, it was reduced to 250.
Presumably increasing it to 500 again will not have negative effects and will
allow the script to run faster.

Assignee: nobody → dkeeler
Status: NEW → ASSIGNED
Whiteboard: [psm-assigned]
Pushed by dkeeler@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/042c5a186d30
increase the parallelism of getHSTSPreloadList.js r=releng-reviewers,jmaher
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 90 Branch

Monday's update (https://firefox-ci-tc.services.mozilla.com/tasks/flPOTfqjQoigj1eFY1kQbg) ran for 297 minutes.
Today's update (https://firefox-ci-tc.services.mozilla.com/tasks/IjAP2s7fQ3yggIpmwgot3A) finished in 157 minutes.

Thanks Dana!

Status: RESOLVED → VERIFIED

Now that getHSTSPreloadList.js runs with increased parallelism, it goes
much faster and we don't need to let the task run for 6 hours.

The timeout was 3 hours for a while, then bumped to 5 hours in bug
1570646, then to 6 hours in bug 1705639 (with 250 in-flight requests).
The new run time with 500 in-flight requests seems to be around 2.5
hours, so pick 4 hours as a compromise.

Pushed by jcristau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1650540c413b
reduce max-run-time for repo-update task. r=RyanVM,releng-reviewers,bhearsum DONTBUILD
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: