As https://hg.mozilla.org/mozilla-central/annotate/e03e0c60462c775c7558a1dc9d5cf2076c3cd1f9/security/manager/ssl/tests/unit/test_sts_preloadlist_perwindowpb.js#l1 says, we have a set of tests that depend on having bugzilla.mozilla.org, login.persona.org, and www.torproject.org included in the preload list. Surprisingly, persona was not the one of those to break first (the "service has shut down" page apparently still pointlessly includes the hsts header), but because bugzilla.mozilla.org is currently mitigating an attack by sending strangers to hardhat.cdn.whatevertherestofthatis, the automatic updates of the preload list have removed bugzilla.mozilla.org, resulting in permaorange in test_sts_preloadlist_perwindowpb.js, test_sts_preloadlist_selfdestruct.js, test_sss_readstate.js, and test_sss_readstate_empty.js. That's a great big party foul, having your test depend on hitting an external site over the network, even if the external site access is done in a separate task from the actual test run, so I'm disabling those tests on esr45/esr52/aurora/mozilla-central (even though they haven't yet broken on mozilla-central since updates are separately broken there). Not sure what the clean solution is, since the only things guaranteed to remain on the list are "pins": "google", which leaves you at the mercy of Google not deciding to remove glass.google.com or something less imaginable like losing a trademark battle and losing control of www.gmail.com.
Pushed by email@example.com: https://hg.mozilla.org/mozilla-central/rev/2ecf610d3185 disable hsts tests which depend on bugzilla.mozilla.org always being in the preload list, a=bustage
Indeed, making these assumptions was not the best idea. Anyways, a solution that will work is to have the preload script always insert some test entries that we can then use in tests. This is what the HPKP script already does. https://treeherder.mozilla.org/#/jobs?repo=try&revision=f9ef7edd12bae839ac0675646d90209c076c7caf
Assignee: nobody → cykesiopka.bmo
Status: NEW → ASSIGNED
Priority: -- → P1
Comment on attachment 8851623 [details] Bug 1350599 - Use guaranteed preloaded test domains instead of real domains in HSTS tests. https://reviewboard.mozilla.org/r/123884/#review126494 Great - thanks.
Attachment #8851623 - Flags: review?(dkeeler) → review+
Thanks! https://treeherder.mozilla.org/#/jobs?repo=try&revision=340c6e05b684cdbf967c3cf2110a931d7c1b9ea0 (As a gentle reminder, the changes in Bug 1350868 need to land first.)
Pushed by firstname.lastname@example.org: https://hg.mozilla.org/integration/autoland/rev/9cd5529dcc38 Use guaranteed preloaded test domains instead of real domains in HSTS tests. r=keeler
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Created attachment 8854954 [details] [diff] [review] bug1350599_fix-hsts-tests_esr52v1.patch This is the ESR 52 version of the patch.
Attachment #8854954 - Flags: review+
https://bugzilla.mozilla.org/attachment.cgi?id=8851623 is test only and should be uplifted to Aurora. https://bugzilla.mozilla.org/attachment.cgi?id=8854954 is test only and should be uplifted to ESR 52. Thanks!
Whiteboard: [psm-assigned] → [psm-assigned][checkin-needed-aurora][checkin-needed-esr52]
status-firefox54: --- → fixed
Whiteboard: [psm-assigned][checkin-needed-aurora][checkin-needed-esr52] → [psm-assigned][checkin-needed-esr52]
status-firefox-esr52: --- → fixed
Whiteboard: [psm-assigned][checkin-needed-esr52] → [psm-assigned]
You need to log in before you can comment on or make changes to this bug.