728535 - (t-snow-r4-0007) t-snow-r4-0007 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

Sadly, I didn't say how it was broken in bug 716326, so I don't know whether that was also like https://tbpl.mozilla.org/php/getParsedLog.php?id=9433872&tree=Mozilla-Inbound where all of rm -rf tools and then rm -rf build says "rm: tools/.hg/00changelog.i: Invalid argument" about every file, and then it craps out trying to download the build, with "Cannot write to `firefox-13.0a1.en-US.mac.dmg' (Invalid argument)." but, I sort of think it was, and it probably needs its disk looked at, rather than just another reimage. Maybe.

Anyway, it's once again chewing up jobs like crazy because it only takes a couple of seconds to fail to rm and then fail to save a downloaded build.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Disabled in slavealloc and on the buildbot master. It's refusing ssh access so will needs some hands on help.

Comment 2

[Chris Cooper [:coop] (he/him)]

Putting it back in the pool. Will monitor.

Comment 4

[Mike Taylor [:bear]]

I disabled it per Philor's request in IRC and didn't realize it already had a bug (teach me to not check slavealloc *before* filing bug...)

Comment 5

[Armen [:armenzg]]

decomission?

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #5)
> decomission?

rev4 machines are (essentially) brand-new, so I certainly hope not. Please file a server ops bug to get it fixed.

Comment 7

[Chris Cooper [:coop] (he/him)]

Handing back to buildduty now that hands-on has been requested.

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156650&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11156669&tree=Mozilla-Inbound

rm: tools/.hg: Device not configured
rm: tools/.hgignore: Invalid argument
rm: tools/.hgtags: Invalid argument
rm: tools/.pylintrc: Invalid argument
rm: tools/breakpad: Device not configured
rm: tools/buildbot-helpers: Device not configured
rm: tools/buildfarm: Device not configured
rm: tools/cdmaker: Device not configured
rm: tools/clobberer: Device not configured
rm: tools/graphserver_webapp: Device not configured
rm: tools/lib: Device not configured
rm: tools/MANIFEST.in: Invalid argument
rm: tools/misc: Device not configured
rm: tools/release: Device not configured
rm: tools/scripts: Device not configured
rm: tools/setup.py: Invalid argument
rm: tools/stage: Device not configured
rm: tools/sut_tools: Device not configured
rm: tools/trychooser: Device not configured
rm: tools: Directory not empty

Comment 9

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156809&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11156805&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11156825&tree=Birch

rm: tools/.hg: Device not configured
rm: tools/.hgignore: Invalid argument
rm: tools/.hgtags: Invalid argument
rm: tools/.pylintrc: Invalid argument
rm: tools/breakpad: Device not configured
rm: tools/buildbot-helpers: Device not configured
rm: tools/buildfarm: Device not configured
rm: tools/cdmaker: Device not configured
rm: tools/clobberer: Device not configured
rm: tools/graphserver_webapp: Device not configured
rm: tools/lib: Device not configured
rm: tools/MANIFEST.in: Invalid argument
rm: tools/misc: Device not configured
rm: tools/release: Device not configured
rm: tools/scripts: Device not configured
rm: tools/setup.py: Invalid argument
rm: tools/stage: Device not configured
rm: tools/sut_tools: Device not configured
rm: tools/trychooser: Device not configured
rm: tools: Directory not empty

Comment 10

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11156872&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11156764&tree=Birch

Connecting to build.mozilla.org|10.2.74.128|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1,169 (1.1K) [application/x-sh]
installdmg.sh: Invalid argument

Cannot write to `installdmg.sh' (Invalid argument).
program finished with exit code 1

Comment 11

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157344&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11157301&tree=Firefox

Comment 12

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157557&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11157563&tree=Mozilla-Inbound

Comment 13

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11158550&tree=Fx-Team

Comment 14

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11157755&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=11157899&tree=Firefox

Comment 15

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11159447&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11159436&tree=Birch
https://tbpl.mozilla.org/php/getParsedLog.php?id=11159396&tree=Birch

Comment 16

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=11159703&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11159623&tree=Mozilla-Inbound

Comment 17

[Mike Taylor [:bear]]

disabled in slavealloc and ssh'ing in to make sure it's offline

Comment 18

[Mike Taylor [:bear]]

something is wrong with the harddrive on this host - the above activity points to a bad or full drive and df -h shows:

talos-r4-snow-007:~ cltbld$ df -h
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2   298Gi  8.3Gi  289Gi     3%    /
devfs          106Ki  106Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home

please pull this unit and check it's harddrive

Comment 19

[Jake Watkins [:dividehex]]

this slave has returned from repairs with a new hdd.  Re-imaged and back in SCL1.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Back in production

Comment 21

[[e-mail] (:bhearsum)]

Needs a reboot, and to be added back to nagios.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Rebooted via PDU after nagios reported it down.

Comment 23

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1037831)