734764 - Queues need to be created with mode 755 so that nagios can check status

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 734728 ended up rebooting four buildbot masters in SCL1, which had nagios alerts like this afterwards
 buildbot-master04.build.scl1:Command Queue is UNKNOWN: Unhandled exception

It turned out that /dev/shm/queue and subdirs had mode 700 on cltbld:cltbld, so nagios can't come along and read the files there. A 'chmod -R 755 /dev/shm/queue' fixes it up, but we should create the dirs with the right mode.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

At face value you'd expect this to have already fixed it
 http://hg.mozilla.org/build/tools/rev/d9e1c085fa02

catlee said he'd take a look Monday.

Comment 2

[Chris AtLee [:catlee]]

very strange...all the tools checkouts I can find on this machine are up-to-date. the only thing out of place I could find was an old lib/python/buildtools/queuedir.pyc reference. maybe some script was picking that up instead of lib/python/mozilla_buildtools/queuedir.py.

I've removed the old .pyc files and the empty directory.

Comment 3

[Chris AtLee [:catlee]]

bm04 was jsut rebooted, and this happened again.

/dev/shm/queue is 0700, as are /dev/shm/queue/{commands,pulse}. The subdirectories of {commands,pulse} are 0755 though.

Comment 4

[Chris AtLee [:catlee]]

so one way to reproduce this is if you have a umask of 077. then os.makedirs(dirname, 0755) results in 0700 permissions on those directories. I'm not sure if this is actually happening or not, but one change we could make is to explicitly set the umask of buildbot and the queue consumers to 022.

Comment 5

[Chris AtLee [:catlee]]

the problem here is that buildbot is running with umask 077, and it starts before the queue processors do, so the directories get created with too-restrictive directories.

Comment 6

[Chris AtLee [:catlee]]

Attachment: force directory permissions to be correct