ccache sometimes hits fatal permission denied errors when trying to create temp files

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Found in https://tbpl.mozilla.org/php/getParsedLog.php?id=31326368&tree=Mozilla-Central:
10:34:44     INFO -  ccache: FATAL: Could not create /builds/ccache/3/7/d76bc25f070bca8f461fca34c9556f-1914220.o.tmp.stdout.bld-linux64-ec2-313.build.releng.usw2.mozilla.com.26108 (permission denied?)
https://tbpl.mozilla.org/php/getParsedLog.php?id=31633894&tree=B2g-Inbound
Duplicate of this bug: 945848
https://tbpl.mozilla.org/php/getParsedLog.php?id=31653649&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=31765984&tree=Fx-Team
Summary: ccache sometimes hits fatal errors when trying to create temp files → ccache sometimes hits fatal permission denied errors when trying to create temp files
https://tbpl.mozilla.org/php/getParsedLog.php?id=31854453&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=31887539&tree=Holly
https://tbpl.mozilla.org/php/getParsedLog.php?id=31899394&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=31954843&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=31963489&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=31990243&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=31994356&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=31995826&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=32037349&tree=Try
https://tbpl.mozilla.org/php/getParsedLog.php?id=32056527&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32382233&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32359076&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32395521&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32450494&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=32660653&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32716321&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32853638&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32856146&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32865855&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=32976255&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32998735&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33072808&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=33081988&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33137482&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=33149062&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33260726&tree=B2g-Inbound
What kind of speedup do these builds get from ccache?
https://tbpl.mozilla.org/php/getParsedLog.php?id=33287847&tree=Holly
https://tbpl.mozilla.org/php/getParsedLog.php?id=33333431&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=33365147&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33431122&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=33437713&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33502097&tree=Try
https://tbpl.mozilla.org/php/getParsedLog.php?id=33548687&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=33563812&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33643246&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=33674252&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=33713327&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=33719773&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=33768231&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=33937168&tree=Mozilla-Inbound
Still sort of curious about the actual measured speed benefits of ccache for these builds, what with the fact that my reported number of failures has to be somewhere between one third and one tenth of the total number of failures.
https://tbpl.mozilla.org/php/getParsedLog.php?id=33947981&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=33966950&tree=B2g-Inbound#error60
https://tbpl.mozilla.org/php/getParsedLog.php?id=34033579&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34082940&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34100648&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34149084&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34172620&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=34181553&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34184494&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34214390&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34231938&tree=Mozilla-Inbound

Chris, this is a frequent source of wasted build time. Can we please find an owner?
Flags: needinfo?(catlee)
FYI, we may be replacing ccache with glandium's S3 cache soon, see
http://glandium.org/blog/?p=3054
http://glandium.org/blog/?p=3079
https://tbpl.mozilla.org/php/getParsedLog.php?id=34439293&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34521584&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34644994&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34653956&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=34685256&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34704017&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34744016&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34746744&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34748046&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34766570&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=34769968&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34822074&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=34945990&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34978835&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35102393&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35161423&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35182752&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=35181058&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=35240299&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35283755&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35359786&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35404126&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35561639&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35572067&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35633154&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35640784&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35824484&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=35853563&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35892066&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=35914153&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=36012444&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=36116314&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=36120213&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=36146693&tree=Fx-Team
https://tbpl.mozilla.org/php/getParsedLog.php?id=36221071&tree=Mozilla-Central
19:59 < catlee-away> https://lkml.org/lkml/2012/12/5/33
20:00 < catlee-away> 18:29:52     INFO -  INFO: Package ccache-3.1.7-moz0.i386 already installed and latest version
20:00 < catlee-away> could try http://ccache.samba.org/releasenotes.html#_ccache_3_1_9
20:00 < catlee-away> Disappearing temporary stdout/stderr files are now handled gracefully.
20:00 < catlee-away> sounds promising
Flags: needinfo?(catlee)

Updated

3 years ago
Depends on: 984062
https://tbpl.mozilla.org/php/getParsedLog.php?id=36305486&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=36329819&tree=Fx-Team
I just deployed an updated version of ccache. Hopefully that fixes this issue!
Calling this fixed!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
https://tbpl.mozilla.org/php/getParsedLog.php?id=47696105&full=1&branch=mozilla-inbound
You need to log in before you can comment on or make changes to this bug.