periodic HSTS updates failing with ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so)

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
2 years ago
2 months ago

People

(Reporter: keeler, Assigned: catlee)

Tracking

({regression})

unspecified
regression

Firefox Tracking Flags

(firefox49+ fixed, firefox50+ fixed, firefox51+ fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

58 bytes, text/x-review-board-request
nthomas
: review+
catlee
: checked-in+
Details | Review
(Reporter)

Description

2 years ago
https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm94-build1-build0.txt.gz says:

INFO: Downloading all the necessary pieces to update HSTS...
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js
2016-08-13 03:05:06 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js [16469/16469] -> "getHSTSPreloadList.js" [1]
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors
2016-08-13 03:05:07 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors [124404/124404] -> "nsSTSPreloadList.errors" [1]
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc
2016-08-13 03:05:08 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc [1237675/1237675] -> "nsSTSPreloadList.inc" [1]
INFO: Generating new HSTS preload list...
INFO: Running "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/lib:. ./xpcshell /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc"
./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so)
cat: nsSTSPreloadList.errors: No such file or directory
ERROR: xpcshell exited with a non-zero exit code: 1
program finished with exit code 70
[Tracking Requested - why for this release]:

Once this is running again we should manually update the list copy on newer branches. The last update I see was on 2016-06-04 which was 2 days before Fx49 was merged to aurora. We might consider Fx49 OK, but 50 definitely needs an update.

Note: obviously it's not great if we aren't adding this security feature to newly added sites, but the real problem is that the stale list might include sites that should be _dropped_ from the list. This may "black hole" sites (or parts of sites) that had to stop using HSTS, forcing users to switch to another browser to access those sites or pages.
status-firefox49: --- → affected
status-firefox50: --- → affected
status-firefox51: --- → affected
tracking-firefox49: --- → ?
tracking-firefox50: --- → +
tracking-firefox51: --- → +
Keywords: regression
Duplicate of this bug: 1285847
(from bug 1285847 comment #0)
> Broke sometime in the window 2016-07-11 and 2016-07-18:
> 
> http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-
> linux64/mozilla-central-linux64-periodicupdate-bm72-build1-build4.txt.gz
> 
> INFO: Running
> "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/
> lib:. ./xpcshell
> /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js
> /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc"
> ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found
> (required by ./libxul.so)
> cat: nsSTSPreloadList.errors: No such file or directory
> ERROR: xpcshell exited with a non-zero exit code: 1
> 
> Likely to be bug 1278456, and I think we'll need 
> * to update the manifest 
>  https://dxr.mozilla.org/build-central/source/tools/scripts/
> periodic_file_updates/periodic_file_updates.manifest
> to unpack gcc or clang (following the linux64 build)
> * and add this sort of logic
>  http://hg.mozilla.org/mozilla-central/file/default/build/unix/mozconfig.
> stdcxx
> in
>  http://hg.mozilla.org/build/tools/file/default/scripts/
> periodic_file_updates/periodic_file_updates.sh
Tracking for 49. catlee can you help find someone to take this on?
tracking-firefox49: ? → +
Flags: needinfo?(catlee)
(Assignee)

Comment 5

2 years ago
Will do.

I really wish these tasks weren't so fragile. Do we really need to run xpcshell for this?
Flags: needinfo?(catlee)

Comment 6

2 years ago
I don't think it would be that hard to make a little Python thing that dumped it in whatever format that Firefox needs it in.  TBH, it would probably best if it could just consume the Chromium pseudo-JSON directly.  If not, the Mozilla HTTP Observatory already uses that data and this code could probably get you most of the way towards what you need; wouldn't be too difficult to make a little Docker container that consumed things and output the right file.

https://github.com/mozilla/http-observatory/blob/master/httpobs/scanner/analyzer/utils.py#L11

It produces this kind of output:

https://gist.github.com/marumari/76dc2cf7a0ee4402a856567ccc925582
(Reporter)

Comment 7

2 years ago
(In reply to Chris AtLee [:catlee] from comment #5)
> I really wish these tasks weren't so fragile. Do we really need to run
> xpcshell for this?

The rationale for using xpcshell is that it will process the HSTS headers the same way Firefox will. That is, sites will only be accepted as HSTS by this script if Firefox would also accept them as HSTS when visiting them during normal browsing.

Comment 8

2 years ago
I can look at this on or after August 29th when I return from vacation. 

If it's more urgent, someone else will need to pick it up. philor already has a good starting place in comment #3.
I'm not sure how urgent it is or how much time for testing we would need. The RC build will be Sept. 5, and we should release on Sept. 13th.

Comment 10

2 years ago
I agree with dveditz that this is pretty urgent, and pretty bad from a security perspective.  Many sites rely on HSTS preloading to protect their users from MitM attacks on the first use of a website, before they've seen any HSTS headers.
(Reporter)

Comment 11

2 years ago
One option we have is for someone to run the job manually on their machine and check the changes in to the various trees. I don't have time to take care of it today, but I can probably at least run the script on Monday.
(Reporter)

Comment 12

2 years ago
Update on comment 11: when I run this locally it hangs, so either it only works on our infrastructure machines (well, other than when this bug is an issue), or it broke in other ways since June.
(Assignee)

Comment 13

2 years ago
Can you try with a different / older version of Firefox?

How can we make these jobs more resilient? These have historically been very fragile.
Comment hidden (mozreview-request)

Comment 15

2 years ago
mozreview-review
Comment on attachment 8784061 [details]
Bug 1295268 - Fix HSTS updates

https://reviewboard.mozilla.org/r/73634/#review71554

lgtm
Attachment #8784061 - Flags: review?(nthomas) → review+
(Assignee)

Updated

2 years ago
Attachment #8784061 - Flags: checked-in+
(Assignee)

Updated

2 years ago
Assignee: nobody → catlee
(Assignee)

Updated

2 years ago
Blocks: 1298056
(Assignee)

Comment 19

2 years ago
I filed 1298056 to track the hang. This isn't something RelEng can fix.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
David, can you help here and in the related bug?  Is this something we can fix before 49 release?
Flags: needinfo?(dkeeler)
Sounds like there may be 2 issues:  update this before 49 release and also more long term address its fragility.
(Reporter)

Comment 22

2 years ago
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #20)
> David, can you help here and in the related bug?

Yes - I'll investigate.

> Is this something we can fix before 49 release?

I don't know.
Flags: needinfo?(dkeeler)
(Assignee)

Comment 23

2 years ago
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> Sounds like there may be 2 issues:  update this before 49 release and also
> more long term address its fragility.

The automated updates are working for beta (49). e.g. http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed August 20th.
(In reply to Chris AtLee [:catlee] from comment #23)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> > Sounds like there may be 2 issues:  update this before 49 release and also
> > more long term address its fragility.
> 
> The automated updates are working for beta (49). e.g.
> http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed
> August 20th.

I don't claim to understand this bug, should status-firefox49 be "fixed"?
Flags: needinfo?(catlee)
(Assignee)

Comment 25

2 years ago
I believe so. AFAICT, 49 is being updated regularly. It was updated again on the weekend:
http://hg.mozilla.org/releases/mozilla-beta/rev/571cff42892d
Flags: needinfo?(catlee)
(Reporter)

Comment 26

2 years ago
The blocklist is separate from the hsts and hpkp periodic updates, so that may be succeeding while the others are not. Thanks to https://hg.mozilla.org/build/tools/rev/3600436770f2000f77ec76be02e1f06d98c3b485 and bug 1298056, the hsts updates are now working again on mozilla-central: https://hg.mozilla.org/mozilla-central/rev/2cdc70c1d8e1

What we'll need to do now is uplift the changes in bug 1298056 to all channels that need it (this is being taken care of in that bug). We may also need to cherry-pick a recent version of the preload list to land on beta (since if I recall correctly, we don't run the hsts periodic updater on beta).
The current configuration is:
* blocklist is updated on central, aurora, beta, release, and esr45.
* HSTS and HPKP are updated on central, aurora, and esr45
David, do you still need to cherrypick from bug 1298056? Or can we resolve this?
Flags: needinfo?(dkeeler)
(Reporter)

Comment 29

2 years ago
From what I can tell, the issue described in this bug has been addressed. Also, it looks like the preload list in mozilla-beta was successfully updated on July 30th, so I think as long as we iron out the remaining issue on esr45 (bug 1285849), we're good here (no cherry-picking necessary, I think).
Flags: needinfo?(dkeeler)
status-firefox49: affected → fixed
status-firefox50: affected → fixed
status-firefox51: affected → fixed
Something went wrong in the mozilla-central update this weekend in HPKP:
ERROR: xpcshell exited with a non-zero exit code: 3

See http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm91-build1-build0.txt.gz. New bug ?
New bug that already exists, bug 1300305
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.