1072979 - HTTPS - Port 443 on etherpad.zlb.phx.mozilla.net is CRITICAL: CRITICAL - Socket timeout after 10 seconds

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.phx1.mozilla.com:

Hostname: etherpad.zlb.phx.mozilla.net
Service:  HTTPS - Port 443
State:    CRITICAL
Output:   CRITICAL - Socket timeout after 10 seconds

Runbook:  http://m.allizom.org/HTTPS+-+Port+443

Comment 1

[Ludovic Hirlimann [:Usul]]

[[e-mail]1 lhirlimann]# screen -r

        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) 
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-09-25 07:49:53.881::WARN:  EXCEPTION
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:165)  
        at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
        at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:565)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
        at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) 
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Task checkForStalePads execution failed with non-200 response: 500

Comment 2

[Ludovic Hirlimann [:Usul]]

Task checkForStalePads execution failed with non-200 response: 500
^C[2014-09-25 07:59:26.240-0700]: Shutting down...
[2014-09-25 07:59:29.240-0700]: ...done, running onshutdown.
[2014-09-25 07:59:30.543-0700]: ...done, stopping server.

Comment 3

[Ludovic Hirlimann [:Usul]]

[[e-mail]1 lhirlimann]# service etherpad start
Starting soffice: 
Waiting for service to come up ...bash: /usr/lib64/openoffice.org3/program/soffice: No such file or directory
.
Starting etherpad: 

done
[[e-mail]1 lhirlimann]#  screen -r
[detached]
[[e-mail]1 lhirlimann]#

Comment 4

[C. Liang [:cyliang]]

Closing the bug (since the service is up).  Wishing for team-pad like extension of the new Etherpad. >_<