735735 - (talos-r3-fed-068) talos-r3-fed-068 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Aki Sasaki (not active)]

Needs a reboot.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Getting close to a reimage ?

Comment 2

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 3

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 4

[Armen [:armenzg]]

PING down again:
[52] talos-r3-fed-068.build.scl1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%

Comment 5

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 6

[[e-mail] (:bhearsum)]

Back in production.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot again.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

The reboot got this back in production.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot.

Comment 10

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot.

Comment 12

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 13

[Rail Aliiev [:rail]]

Up and running

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Please reboot.

Comment 15

[[e-mail] (:bhearsum)]

Back in production.

Comment 16

[[e-mail] (:bhearsum)]

Back in production.

Comment 17

[Armen [:armenzg]]

Reboot needed.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Did 100s of jobs since bug 808500 but is down again, reboot time.

Comment 19

[Armen [:armenzg]]

Taking jobs.

Comment 20

[Kim Moir [:kmoir] ET]

Needs a reboot, not available on the network.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Seeing this a lot this morning:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17973936&tree=Mozilla-Esr17

exceptions.OSError: [Errno 30] Read-only file system: '/home/cltbld/talos-slave/test-pgo/build'

Comment 22

[Justin Wood (:Callek)]

Disabled in slavealloc, and I went in and forced a reboot as well.

Not a single pass in the last 400 runs

Comment 23

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #22)
> and I went in and forced a reboot as well.

Umm, odd- Seems that *not a single* login attempt, with all versions of the password, including trying as root is working

Comment 26

[Armen [:armenzg]]

Requested to be re-imaged in bug 826863.

Comment 27

[Armen [:armenzg]]

Do not forget to enable on slavealloc when we get to set it up.
Remember that it might have issues and it might burn jobs even after re-imaging.
Diagnosis might be required at that point.

Comment 28

[[e-mail] (:bhearsum)]

Back in production.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Reboot needed (bug 830154)

Comment 30

[Kim Moir [:kmoir] ET]

needs a reboot

Comment 31

[Carsten Book [:Tomcat]]

and reboot needed ,

Host Status:	
  DOWN  
 (for 1d 5h 21m 27s)
Status Information:	PING CRITICAL - Packet loss = 100%

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

And down again.

Comment 33

[Justin Wood (:Callek)]

Confirmed its back up again.

Comment 34

[Justin Wood (:Callek)]

...and back down

Comment 35

[Aki Sasaki (not active)]

Green (and a couple orange) in production.

Comment 36

[[e-mail] (:bhearsum)]

Back in production.

Comment 37

[Justin Wood (:Callek)]

Last job Oct 3

Comment 38

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 962752)

Comment 39

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 973952)

Comment 40

[[e-mail] (:bhearsum)]

Back in production.

Comment 41

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 977789)

Comment 42

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 988921)

Comment 43

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 994768)

Comment 44

[Justin Wood (:Callek)]

Disabled in slavealloc due to lack of jobs. Should be back ready for service anyway