922783 - (bld-lion-r5-050) bld-lion-r5-050 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[[e-mail] (:bhearsum)]

ping check failing, trying a pdu reboot

Comment 1

[[e-mail] (:bhearsum)]

Back in production.

Comment 2

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=32626116&tree=Fx-Team, https://tbpl.mozilla.org/php/getParsedLog.php?id=32666101&tree=B2g-Inbound, https://tbpl.mozilla.org/php/getParsedLog.php?id=32666778&tree=B2g-Inbound, all disk failures.

Disabled in slavealloc.

Comment 3

[Armen [:armenzg]]

Re-image requested.

Comment 4

[[e-mail] (:bhearsum)]

Back in production.

Comment 5

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=33932642&tree=Mozilla-Inbound (and, by the time anyone reboots it, hundreds of others)

rm: tools: Input/output error

Might wants some diagnostics on that disk.

Disabled in slavealloc, despite the fact that won't do anything.

Comment 6

[Justin Wood (:Callek)]

enabled after diags in Bug 974068

Comment 7

[Ed Morley [:emorley]]

Perma-retrying; disabled in slavealloc

https://tbpl.mozilla.org/php/getParsedLog.php?id=38008207&tree=Mozilla-Inbound

rm: tools/trychooser/index.html: Invalid argument
rm: tools/trychooser/jquery-ui.css: Invalid argument
rm: tools/trychooser/jquery.min.js: Invalid argument
rm: tools/trychooser/trychooser.css: Invalid argument
rm: tools/trychooser/trychooser.js: Invalid argument
rm: tools/trychooser/tryload.js: Invalid argument
rm: tools/trychooser: Directory not empty
rm: tools: Directory not empty
program finished with exit code 1
elapsedTime=2086.710640
========= Finished clobber build tools failed (results: 5, elapsed: 34 mins, 46 secs) (at 2014-04-17 06:04:57.571328) =========

Comment 8

[Phil Ringnalda (:philor)]

IOError in https://tbpl.mozilla.org/php/getParsedLog.php?id=48056361&tree=Mozilla-Central

Comment 9

[Phil Ringnalda (:philor)]

And in http://buildbot-master86.srv.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-beta%20build/builds/133/steps/compile/logs/stdio

Comment 10

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting PDU reboot...Failed.
Filed IT bug for reboot (bug 1067698)

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Being the only slave that hits bug 1130905 sounds more than fishy to me.

Comment 13

[Phil Ringnalda (:philor)]

The fact that we continue to allow mozharness tests to just fill up slave disks with trash until they can no longer run, and then reimage the slave to allow them to run again, seems more than fishy to me.

Nevertheless, temporarily reenabled this slave to get it a job which runs tests, to see whether the mozharness fix from bug 1130905 will cause it to properly output the old message which will tell us that the test has completely crapped it up and it has to be reimaged.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Ohai.
https://treeherder.mozilla.org/logviewer.html#?job_id=1968058&repo=fx-team

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So Jordan and I debugged that a bit and found the following problem:

[[e-mail] ~]# ls -la /Volumes/Stub
total 0
d--x--x--x  3 cltbld  admin  102  8 Feb 14:40 .
drwxrwxrwt@ 4 root    admin  136 11 Feb 13:36 ..
drwx------  4 root    admin  136  8 Feb 14:40 .Spotlight-V100

This folder has been mounted by root so it's clear that our build process which is not running as root, cannot mount anything.

Also as Jordan wrote on IRC it was not a current mount:

<jlund|buildduty> I don't think this is a mount
<jlund|buildduty> hdiutil: eject failed - No such file or directory
<jlund|buildduty> tries rm -rf
<jlund|buildduty> [[e-mail] Volumes]# rm -rf Stub
<jlund|buildduty> success

So the mozinstall problem should be done now. If it manifests again please let us know immediately. In such a case it would be good to know how this mount was initially mounted and from whom.

Comment 16

[Jordan Lund (:jlund)]

re-enabling. cc'n myself to check back before closing

Comment 17

[Jordan Lund (:jlund)]

last 7 jobs are green

Comment 18

[Alin Selagea [:aselagea]]

Back to the pool after re-image.

Comment 19

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online after been re-image.

Comment 20

[Tom Prince [:tomprince]]

This machine appears to be having memory issues:
build-script-build(56444,0x102004000) malloc: *** error for object 0x7f9c2c01c608: incorrect checksum for freed object - object was probably modified after being freed.
from
https://treeherder.mozilla.org/logviewer.html#?job_id=114117969&repo=comm-central&lineNumber=2594

Comment 21

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* comm-central: 1

Platform breakdown:
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=922783&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 22

[Tom Prince [:tomprince]]

https://treeherder.mozilla.org/logviewer.html#?job_id=115339431&repo=comm-central&lineNumber=4849

Comment 23

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Re-imaged and back online

Comment 24

[Intermittent Failures Robot]

1 failures in 822 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* comm-central: 1

Platform breakdown:
* osx-10-7: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=922783&startday=2017-07-17&endday=2017-07-23&tree=all