601623 - linux-ix-slave16 and linux-ix-slave18 are in a weird state

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

It has messed up several jobs and I have gracefully shut it down since I was unable to move buildbot.tac to off.

It was failing to remove the "tools" directory because of permission issues.

rm: cannot remove `tools/stage/post_upload.py': Read-only file system
rm: cannot remove `tools/buildfarm/breakpad/cleanup-breakpad-symbols.py': Read-only file system
rm: cannot remove `tools/buildfarm/maintenance/count_and_reboot.py': Read-only file system
...

Oct 04 08:31 	?? 	failure 	Android R7 tracemonkey build 	#195 	Failed clone_buildtools hg_pull
Oct 03 15:02 	?? 	failure 	Android R7 tracemonkey build 	#194 	Failed hg_pull
Oct 03 04:02 	?? 	exception 	Linux electrolysis nightly 	#9 	Exception rm_builddir tinderboxprint_changeset
Oct 02 09:13 	?? 	failure 	Android R7 tracemonkey build 	#193 	Failed hg_pull
Oct 02 04:02 	d292d6bb8b0f... 	failure 	Linux electrolysis nightly 	#8 	Failed compile
Oct 01 17:59 	?? 	success 	Android R7 tracemonkey build 	#192 	Build successful slave lost

[root@linux-ix-slave16 slave]# mv buildbot.tac buildbot.tac.off
mv: cannot move `buildbot.tac' to `buildbot.tac.off': Read-only file system
[root@linux-ix-slave16 slave]# ls -l buildbot.tac 
-rw-r--r-- 1 cltbld cltbld 966 Oct  1 10:58 buildbot.tac

Comment 1

[Armen [:armenzg]]

I have disabled buildbot on linux-ix-slave18 as it had been doing wacky things:
http://buildbot-master2.build.scl1.mozilla.com:8010/buildslaves/linux-ix-slave18?numbuilds=400

This machine had been going wild for longer time than linux-ix-slave16.

Oct 01 14:32 	?? 	exception 	Firefox mozilla-1.9.2 linux l10n dep 	#59 	Exception update_locale_source
Oct 01 14:13 	?? 	failure 	Android R7 tracemonkey build 	#18 	Failed hg_pull
Oct 01 14:02 	?? 	exception 	Firefox mozilla-central linux l10n dep 	#84 	Exception update_locale_source
Oct 01 13:11 	?? 	failure 	Android R7 tracemonkey build 	#17 	Failed hg_pull
Oct 01 11:47 	?? 	success 	Firefox mozilla-central linux l10n dep 	#80 	Build successful

Comment 4

[Aki Sasaki (not active)]

remounting /builds, rebooting and rebuilding seemd to help 18, which was back in production this weekend.

Comment 5

[Aki Sasaki (not active)]

18 is still rw /builds.
Just fixed 16.  Don't know why this happened, but should be fixed now.