1295268 - periodic HSTS updates failing with ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm94-build1-build0.txt.gz says:

INFO: Downloading all the necessary pieces to update HSTS...
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js
2016-08-13 03:05:06 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js [16469/16469] -> "getHSTSPreloadList.js" [1]
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors
2016-08-13 03:05:07 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors [124404/124404] -> "nsSTSPreloadList.errors" [1]
INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc
2016-08-13 03:05:08 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc [1237675/1237675] -> "nsSTSPreloadList.inc" [1]
INFO: Generating new HSTS preload list...
INFO: Running "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/lib:. ./xpcshell /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc"
./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so)
cat: nsSTSPreloadList.errors: No such file or directory
ERROR: xpcshell exited with a non-zero exit code: 1
program finished with exit code 70

Comment 1

[Daniel Veditz [:dveditz]]

[Tracking Requested - why for this release]:

Once this is running again we should manually update the list copy on newer branches. The last update I see was on 2016-06-04 which was 2 days before Fx49 was merged to aurora. We might consider Fx49 OK, but 50 definitely needs an update.

Note: obviously it's not great if we aren't adding this security feature to newly added sites, but the real problem is that the stale list might include sites that should be _dropped_ from the list. This may "black hole" sites (or parts of sites) that had to stop using HSTS, forcing users to switch to another browser to access those sites or pages.

Comment 3

[Phil Ringnalda (:philor)]

(from bug 1285847 comment #0)
> Broke sometime in the window 2016-07-11 and 2016-07-18:
> 
> http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-
> linux64/mozilla-central-linux64-periodicupdate-bm72-build1-build4.txt.gz
> 
> INFO: Running
> "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/
> lib:. ./xpcshell
> /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js
> /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc"
> ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found
> (required by ./libxul.so)
> cat: nsSTSPreloadList.errors: No such file or directory
> ERROR: xpcshell exited with a non-zero exit code: 1
> 
> Likely to be bug 1278456, and I think we'll need 
> * to update the manifest 
>  https://dxr.mozilla.org/build-central/source/tools/scripts/
> periodic_file_updates/periodic_file_updates.manifest
> to unpack gcc or clang (following the linux64 build)
> * and add this sort of logic
>  http://hg.mozilla.org/mozilla-central/file/default/build/unix/mozconfig.
> stdcxx
> in
>  http://hg.mozilla.org/build/tools/file/default/scripts/
> periodic_file_updates/periodic_file_updates.sh

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 49. catlee can you help find someone to take this on?

Comment 5

[Chris AtLee [:catlee]]

Will do.

I really wish these tasks weren't so fragile. Do we really need to run xpcshell for this?

Comment 6

[April King [:April]]

I don't think it would be that hard to make a little Python thing that dumped it in whatever format that Firefox needs it in.  TBH, it would probably best if it could just consume the Chromium pseudo-JSON directly.  If not, the Mozilla HTTP Observatory already uses that data and this code could probably get you most of the way towards what you need; wouldn't be too difficult to make a little Docker container that consumed things and output the right file.

https://github.com/mozilla/http-observatory/blob/master/httpobs/scanner/analyzer/utils.py#L11

It produces this kind of output:

https://gist.github.com/marumari/76dc2cf7a0ee4402a856567ccc925582

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Chris AtLee [:catlee] from comment #5)
> I really wish these tasks weren't so fragile. Do we really need to run
> xpcshell for this?

The rationale for using xpcshell is that it will process the HSTS headers the same way Firefox will. That is, sites will only be accepted as HSTS by this script if Firefox would also accept them as HSTS when visiting them during normal browsing.

Comment 8

[Chris Cooper [:coop] (he/him)]

I can look at this on or after August 29th when I return from vacation. 

If it's more urgent, someone else will need to pick it up. philor already has a good starting place in comment #3.

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

I'm not sure how urgent it is or how much time for testing we would need. The RC build will be Sept. 5, and we should release on Sept. 13th.

Comment 10

[April King [:April]]

I agree with dveditz that this is pretty urgent, and pretty bad from a security perspective.  Many sites rely on HSTS preloading to protect their users from MitM attacks on the first use of a website, before they've seen any HSTS headers.

Comment 11

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

One option we have is for someone to run the job manually on their machine and check the changes in to the various trees. I don't have time to take care of it today, but I can probably at least run the script on Monday.

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Update on comment 11: when I run this locally it hangs, so either it only works on our infrastructure machines (well, other than when this bug is an issue), or it broke in other ways since June.

Comment 13

[Chris AtLee [:catlee]]

Can you try with a different / older version of Firefox?

How can we make these jobs more resilient? These have historically been very fragile.

Comment 14

[Chris AtLee [:catlee]]

Attachment: Bug 1295268 - Fix HSTS updates

Review commit: https://reviewboard.mozilla.org/r/73634/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/73634/

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8784061 [details]
Bug 1295268 - Fix HSTS updates

https://reviewboard.mozilla.org/r/73634/#review71554

lgtm

Comment 16

[Chris AtLee [:catlee]]

https://hg.mozilla.org/build/tools/rev/3600436770f2000f77ec76be02e1f06d98c3b485
Bug 1295268 - Fix HSTS updates r=nthomas

Comment 17

[Chris AtLee [:catlee]]

FWIW, this has been working on beta and release:

https://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843
https://hg.mozilla.org/releases/mozilla-release/rev/440936afbdbc

Comment 18

[Chris AtLee [:catlee]]

With my patch landed, it's hanging on central too:
https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm77-build1-build1.txt.gz

Comment 19

[Chris AtLee [:catlee]]

I filed 1298056 to track the hang. This isn't something RelEng can fix.

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

David, can you help here and in the related bug?  Is this something we can fix before 49 release?

Comment 21

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds like there may be 2 issues:  update this before 49 release and also more long term address its fragility.

Comment 22

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #20)
> David, can you help here and in the related bug?

Yes - I'll investigate.

> Is this something we can fix before 49 release?

I don't know.

Comment 23

[Chris AtLee [:catlee]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> Sounds like there may be 2 issues:  update this before 49 release and also
> more long term address its fragility.

The automated updates are working for beta (49). e.g. http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed August 20th.

Comment 24

[David Bolter [:davidb] (NeedInfo me for attention)]

(In reply to Chris AtLee [:catlee] from comment #23)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21)
> > Sounds like there may be 2 issues:  update this before 49 release and also
> > more long term address its fragility.
> 
> The automated updates are working for beta (49). e.g.
> http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed
> August 20th.

I don't claim to understand this bug, should status-firefox49 be "fixed"?

Comment 25

[Chris AtLee [:catlee]]

I believe so. AFAICT, 49 is being updated regularly. It was updated again on the weekend:
http://hg.mozilla.org/releases/mozilla-beta/rev/571cff42892d

Comment 26

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

The blocklist is separate from the hsts and hpkp periodic updates, so that may be succeeding while the others are not. Thanks to https://hg.mozilla.org/build/tools/rev/3600436770f2000f77ec76be02e1f06d98c3b485 and bug 1298056, the hsts updates are now working again on mozilla-central: https://hg.mozilla.org/mozilla-central/rev/2cdc70c1d8e1

What we'll need to do now is uplift the changes in bug 1298056 to all channels that need it (this is being taken care of in that bug). We may also need to cherry-pick a recent version of the preload list to land on beta (since if I recall correctly, we don't run the hsts periodic updater on beta).

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

The current configuration is:
* blocklist is updated on central, aurora, beta, release, and esr45.
* HSTS and HPKP are updated on central, aurora, and esr45

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

David, do you still need to cherrypick from bug 1298056? Or can we resolve this?

Comment 29

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

From what I can tell, the issue described in this bug has been addressed. Also, it looks like the preload list in mozilla-beta was successfully updated on July 30th, so I think as long as we iron out the remaining issue on esr45 (bug 1285849), we're good here (no cherry-picking necessary, I think).

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

Something went wrong in the mozilla-central update this weekend in HPKP:
ERROR: xpcshell exited with a non-zero exit code: 3

See http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm91-build1-build0.txt.gz. New bug ?

Comment 31

[Phil Ringnalda (:philor)]

New bug that already exists, bug 1300305