1187966 - Automated HSTS/HPKP/blocklist updates broken on trunk

Status: RESOLVED FIXED

(Core :: Security, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Looks like GTK3 fallout?

http://people.mozilla.org/~coop/hsts_failures/central-failure-july25.log

INFO: Generating new HSTS preload list...
INFO: Running "LD_LIBRARY_PATH=. ./xpcshell /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js /builds/slave/m-cen-l64-periodicupdate-00000/"
./xpcshell: error while loading shared libraries: libcairo-gobject.so.2: cannot open shared object file: No such file or directory

Comment 1

[Mike Hommey [:glandium]]

The environment running this job obviously doesn't have gtk3 installed.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Thanks for the helpful reply. Any ideas who might be able to assist in getting this security process unbroken?

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #2)
> Thanks for the helpful reply. Any ideas who might be able to assist in
> getting this security process unbroken?

Well, it's probably going to be me.

Comment 4

[Chris Cooper [:coop] (he/him)]

I mucked around with this on a dev slave and managed to get it working. Since we only need xpcshell and not a fully functional browser context, I was able to download the gtk3 tarball via tooltool and then get the periodic_update script working by setting the LD_LIBRARY_PATH.

Should have a patch tomorrow, and I can always run it by hand if we need the in-tree updates sooner.

Comment 5

[Chris Cooper [:coop] (he/him)]

Attachment: [tools] Download gtk3 for periodic update script

xpcshell requires gtk3 now, so we download it via tooltool and then set the LD_LIBRARY_PATH.

Comment 6

[Chris Cooper [:coop] (he/him)]

Attachment: [buildbotcustom] Set the env for periodic file update jobs.

We inherit the env so we can make use of TOOLTOOL_CACHE.

Comment 7

[Chris Cooper [:coop] (he/him)]

Here's the job output from these changes running in dry-run mode (-n) in staging. The script returns 2 in dry-run mode if there are updates which is why it's marked as failed.

http://dev-master2.bb.releng.use1.mozilla.com:8044/builders/Linux%20x86-64%20mozilla-central%20periodic%20file%20update/builds/0

Comment 8

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8640783 [details] [diff] [review]
[buildbotcustom] Set the env for periodic file update jobs.

Review of attachment 8640783 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbotcustom/rev/dc0535892866

Comment 9

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8640779 [details] [diff] [review]
[tools] Download gtk3 for periodic update script

Review of attachment 8640779 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/tools/rev/7a9a79f809aa

Comment 10

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/dc0535892866

Comment 11

[Chris Cooper [:coop] (he/him)]

I kicked off a build and it succeeded. Here's the treeherder job triggered by the updates:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=23e525e2ba35