periodic updates failing in xpcshell
Categories
(Release Engineering :: Release Automation, defect)
Tracking
(firefox-esr60 fixed, firefox66 fixed, firefox67 fixed)
People
(Reporter: mtabara, Assigned: sfraser)
Details
(Whiteboard: [releaseduty])
Attachments
(1 file)
The periodic updates graphs started failing today for some reason. Treeherder reports can be seen here.
As far as I fug in the taskcluster side, nothing changed from our end hence there must be something external.
The errors we're getting are:
+ LD_LIBRARY_PATH=:.
+ ./xpcshell /home/worker/scripts/genHPKPStaticPins.js /home/worker/data/PreloadedHPKPins.json /home/worker/data/StaticHPKPins.h.out
./xpcshell: error while loading shared libraries: libX11-xcb.so.1: cannot open shared object file: No such file or directory
+ echo 'INFO: Checking whether new HPKP preload list is valid...'
INFO: Checking whether new HPKP preload list is valid...
+ '[' '!' -s /home/worker/data/StaticHPKPins.h.out ']'
+ echo '/home/worker/data/StaticHPKPins.h.out is empty. That'\''s less good.'
/home/worker/data/StaticHPKPins.h.out is empty. That's less good.
+ exit 52
Full log can be found here
Simon suggested that xpcshell may have changed and he was wondering whgether running the javascript that way is still valid.
|
Reporter | |
Comment 1•6 years ago
|
||
I dug a bit to see what could've caused this. The periodic updates run twice a week, 10:00 UTC on Mondays and Thursdays given their cron schedule.
Last one on central on Thursday worked like a charm. Task is here with logs here
+ compare_hpkp_files
+ cd /home/worker
++ basename /home/worker/data/PreloadedHPKPins.json
+ HPKP_PRELOAD_JSON_HG=https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/PreloadedHPKPins.json
+ HPKP_PRELOAD_OUTPUT_HG=https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/StaticHPKPins.h
+ rm -f /home/worker/data/StaticHPKPins.h.out
+ wget -nv -O /home/worker/data/StaticHPKPins.h https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/StaticHPKPins.h
2019-02-28 12:20:45 URL:https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/StaticHPKPins.h [61495/61495] -> "/home/worker/data/StaticHPKPins.h" [1]
+ wget -nv -O /home/worker/data/PreloadedHPKPins.json https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/PreloadedHPKPins.json
2019-02-28 12:20:47 URL:https://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/PreloadedHPKPins.json [14497/14497] -> "/home/worker/data/PreloadedHPKPins.json" [1]
+ echo 'INFO: Generating new HPKP preload list...'
INFO: Generating new HPKP preload list...
+ cd /home/worker/firefox
+ LD_LIBRARY_PATH=:.
+ ./xpcshell /home/worker/scripts/genHPKPStaticPins.js /home/worker/data/PreloadedHPKPins.json /home/worker/data/StaticHPKPins.h.out
+ echo 'INFO: Checking whether new HPKP preload list is valid...'
INFO: Checking whether new HPKP preload list is valid...
+ '[' '!' -s /home/worker/data/StaticHPKPins.h.out ']'
+ cd /home/worker
+ echo 'INFO: diffing old/new HPKP preload lists...'
On mozilla-beta, last Thursday's one worked like a charm while today's one is still running. Task can be found here. Typically IIUC thesee tasks take 2-3h and today's one runs for at least one hour so I suspect it passed that breaking point that we have in xpcshell on central.
I suspect there's been a library update or something on central that's breaking this. Next thing I'd do is to ping an XPCShell owner to query for potential changes in the last four days.
|
Assignee | |
Updated•6 years ago
|
|
|
Assignee | |
Comment 2•6 years ago
|
||
|
|
Assignee | |
Comment 3•6 years ago
|
||
I wonder if there's a way we can 'follow' changes to this so that we get alerted if the dependencies change again.
|
|
Reporter | |
Updated•6 years ago
|
|
|
Reporter | |
Comment 4•6 years ago
|
||
@jmaher: I'm trying to understand what caused this change in central in the past fours days. Could it have been a dependency change under the hood that added libX11-xcb under the hood?
If we find that, the least we can do is to add Herald on it for a bunch of RelEng folks to keep an eye on it in the future.
Feel free to 302 if wiki owner module is outdated.
Thank you!
|
||
Comment 5•6 years ago
|
||
while I am a module owner for xpcshell, that doesn't necessarily translate to this failure being xpcshell related, although it could.
Do we know that libx11-xcb solved the problem?
I see that this uses a different docker image than what we use in automation for xpcshell. We recently updated our linux docker images to have newer fonts which I landed on Friday:
https://phabricator.services.mozilla.com/D21360
I looked at the build dependencies and didn't see anything that intentionally included libx11-xcb in the last month or so.
maybe :ted would know?
|
|
Assignee | |
Comment 6•6 years ago
|
||
This seems to be an ubuntu change.
libgtk-3-0=3.22.30-1ubuntu1 has the libx11-xcb1 dependency, through installing libwayland-egl1-mesa
3.22.30-1ubuntu2 does not result in libx11-xcb1 being installed.
Running the container manually and checking with ldd and running xpcshell, adding libx11-xcb1 seems to fix things.
|
||
Comment 8•6 years ago
|
||
| bugherder | ||
|
||
Comment 9•6 years ago
|
||
| bugherder uplift | ||
|
||
Comment 10•6 years ago
|
||
| bugherder uplift | ||
| Comment hidden (Intermittent Failures Robot) |
Description
•