878114 - (talos-r4-lion-003) talos-r4-lion-003 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

Torching jobs with hdiutil problems.

https://tbpl.mozilla.org/php/getParsedLog.php?id=23635588&tree=Mozilla-Inbound

Comment 1

[John Hopkins (:jhopkins)]

Disabled in slavealloc, graceful shutdown.

Comment 2

[Armen [:armenzg]]

Needs to be setup.

Comment 3

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 4

[Phil Ringnalda (:philor)]

Torching jobs with hdiutil problems.

Comment 5

[Phil Ringnalda (:philor)]

And, as it will, a forced reboot set it right again.

Comment 6

[Kim Moir [:kmoir] ET]

Killing builds with 

Traceback (most recent call last):
  File "/usr/local/bin/hg", line 36, in <module>
    mercurial.util.setbinary(fp)
  File "/Library/Python/2.5/site-packages/mercurial/demandimport.py", line 76, in __getattribute__
    return getattr(self._module, attr)
AttributeError: 'module' object has no attribute 'setbinary'

Callek is this related to your hg changes?

Comment 7

[Justin Wood (:Callek)]

(In reply to Kim Moir [:kmoir] from comment #6)
> Killing builds with 
> 
> Traceback (most recent call last):
>   File "/usr/local/bin/hg", line 36, in <module>
>     mercurial.util.setbinary(fp)
>   File "/Library/Python/2.5/site-packages/mercurial/demandimport.py", line
> 76, in __getattribute__
>     return getattr(self._module, attr)
> AttributeError: 'module' object has no attribute 'setbinary'
> 
> Callek is this related to your hg changes?

Yep that was related to the hg changes, this one got missed for the buildbot plist change. I have corrected it and put it back in production

Comment 8

[[e-mail] (:bhearsum)]

down, trying a pdu reboot

Comment 9

[[e-mail] (:bhearsum)]

didn't work, needs IT help

Comment 10

[Ed Morley [:emorley]]

Machine is not chewing through jobs:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=talos-r4-lion-003

https://tbpl.mozilla.org/php/getParsedLog.php?id=25657422&tree=Mozilla-Inbound

{
03:59:10    ERROR - Errors received:
03:59:10     INFO - Reading from file tmpfile_stderr
03:59:10    ERROR -  hdiutil: attach failed - Device not configured
03:59:10    ERROR -  Traceback (most recent call last):
03:59:10    ERROR -    File "/Users/cltbld/talos-slave/test/build/venv/bin/mozinstall", line 9, in <module>
03:59:10    ERROR -      load_entry_point('mozInstall==1.6', 'console_scripts', 'mozinstall')()
03:59:10    ERROR -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 299, in install_cli
03:59:10    ERROR -      install_path = install(src, options.dest)
03:59:10    ERROR -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 118, in install
03:59:10    ERROR -      install_dir = _install_dmg(src, dest)
03:59:10    ERROR -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.7/site-packages/mozinstall/mozinstall.py", line 246, in _install_dmg
03:59:10    ERROR -      subprocess.call('hdiutil detach %s -quiet' % appDir,
03:59:10    ERROR -  mozinstall.mozinstall.InstallError: Failed to install "/Users/cltbld/talos-slave/test/installer.dmg"
03:59:10     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
03:59:10     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
03:59:10    ERROR - Return code: 1
03:59:10    FATAL - Halting on failure while running ['/Users/cltbld/talos-slave/test/build/venv/bin/mozinstall', '/Users/cltbld/talos-slave/test/installer.dmg', '--destination', '/Users/cltbld/talos-slave/test/build/application']
03:59:10    FATAL - Exiting 1
}

Disabled in slavealloc.

Comment 11

[Armen [:armenzg]]

Re-imaging or diagnostics are needed.

See bug 897649.

Comment 12

[Armen [:armenzg]]

The machine has been re-imaged.
We need to put it back into production.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Armen Zambrano [:armenzg] (Use needinfo flag) (Release Enginerring) (EDT/UTC-4) from comment #12)
> The machine has been re-imaged.
> We need to put it back into production.

back in production after 6 weeks of pointless idleness.

Comment 14

[[e-mail] (:bhearsum)]

back in production again

Comment 15

[Justin Wood (:Callek)]

Last job Sat Oct 19

Slaveapi says:
      "text": "Attempting SSH reboot...Failed.\nAttempting PDU reboot...Failed.\nCan't do anything else, human intervention needed."

Comment 16

[[e-mail] (:bhearsum)]

Back in production after a RAM replacement.