611128 - Set up linux-ix-slave17 for Internap

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This machine left Castro for repairs at IX and came back to Internap, see bug 601123. We need to update/check the puppet config, make sure the root password is OK and generally get it back into use.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Remove from mv-production.pp

This looks like the appropriate thing to do, with mpt-production-puppet already pointing it to scl and already in scl config.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 489689 [details] [diff] [review]
Remove from mv-production.pp

http://hg.mozilla.org/build/puppet-manifests/rev/cc64939e94cb

Clone on mv-production-puppet update.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Console

I'm not convinced this slave is healthy: see attachment for disk check on /, lots more of those in syslog for Nov 9 and 10, 120+ seconds to do puppet on boot. It's connected to sm02 to see what happens.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Yeah, definitely not well.

Data point the first:
Did a mozilla-central linux opt clobber build in staging on Nov 18, which took much longer than it should. Comparing to a nightly build (which does more work!) on linux-ix-slave40

Task                  linux-ix-slave17    nightly on OK machine
Total                 4 hrs, 31 mins      22 mins
Remove build/tools    5 mins              0 secs   (only 25MB)
Clone m-c             23 mins             4 mins   (*network dependency)
compile               3 hrs, 18 mins      8 mins   (make -j4 in both builds)

Data point the second:
Currently it's refusing ssh connections and using IPMI to redirect the console shows a lot of errors, the latest are of the form
 end_request: I/O error, dev sda, sector 777591

Gonna reopen bug 601123.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Dependent on hardware fix/investigation in dep bug.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Bug#601123 fixed; I see bug#636827 filed to track reimaging.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

The machine was re-imaged in December in bug 601123, and 636827 tracks its re-introduction into production.