436727 - Live Chat server is down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Majken Connor [:Kensie]]

We were all kicked off of spark and https://chat-support.mozilla.com:9091 returns a network timeout error.

Comment 1

[Dave Miller [:justdave]]

This has paged oncall three times already over 45 minutes and hasn't been answered yet.  Since I happen to be around and this is too important to let sit, I went and poked at it.

dm-chat01 kernel panicked.  I rebooted it.

This is the last thing in the log prior to the reboot:

> May 31 16:09:51 dm-chat01 kernel: mptscsih: ioc0: attempting task abort! (sc=df505300)
> May 31 16:09:51 dm-chat01 kernel: sd 0:0:0:0: 
> May 31 16:09:51 dm-chat01 kernel:         command: Write(10): 2a 00 00 06 f8 85 00 00 10 00
> May 31 16:09:53 dm-chat01 kernel: mptbase: Initiating ioc0 recovery
> May 31 16:09:54 dm-chat01 kernel: mptscsih: ioc0: Issue of TaskMgmt failed!
> May 31 16:09:54 dm-chat01 kernel: mptscsih: ioc0: task abort: FAILED (sc=df505300)
> May 31 16:09:54 dm-chat01 kernel: mptscsih: ioc0: attempting target reset! (sc=df505300)
> May 31 16:09:54 dm-chat01 kernel: sd 0:0:0:0: 
> May 31 16:09:54 dm-chat01 kernel:         command: Write(10): 2a 00 00 06 f8 85 00 00 10 00
> May 31 16:09:54 dm-chat01 kernel: mptscsih: ioc0: target reset: SUCCESS (sc=df505300)
> May 31 16:09:54 dm-chat01 kernel:  target0:0:0: Beginning Domain Validation
> May 31 16:09:54 dm-chat01 kernel:  target0:0:0: Domain Validation skipping write tests
> May 31 16:09:54 dm-chat01 kernel:  target0:0:0: Ending Domain Validation
> May 31 16:09:56 dm-chat01 kernel:  target0:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)