856734 - mm-ci-master appears to be hung

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[u279076]

Matt Wobensmith noticed that he hasn't received any 17.0.5esr results in the mozmill-ci dashboard since getting candidate builds on Friday. I checked the mozmill-ci emails and there are a lot of results with no dashboard URLs. When I try to load http://mm-ci-master.qa.scl3.mozilla.com:8080/ it takes a very very long time. There is also a considerable backlog of testruns which have been in queue for 3 hours.

I'm not sure how to debug this any further. This blocks our ability to qualify 17.0.5esr and 20.0RC (due to release tomorrow AM).

Comment 1

[cmtalbert]

Rwood, time to put all that knowledge you learned from Henrik to action -- can you see if you can debug what's going on?  Perhaps the system just needs to be rebooted?

Comment 2

[Robert Wood [:rwood]]

I tried restarting just jenkins but it wouldn't respond; so I did an entire restart of master. Master/jenkins etc. and dashboard now back online.

Comment 3

[Matt Wobensmith [:mwobensmith][:matt:]]

Site is accessible and I was able to kick off a test run, but pages only load partially and site appears almost too slow to be usable.

Comment 4

[Robert Wood [:rwood]]

Hmmm... the site is running fine for me, and pages for the various slaves, etc. load fine and quickly...

Comment 5

[Matt Wobensmith [:mwobensmith][:matt:]]

Sadly, not for me. In fact, now the site connection is timing out completely.

Comment 6

[u279076]

(In reply to Matt Wobensmith from comment #5)
> Sadly, not for me. In fact, now the site connection is timing out completely.

It's nice and snappy now on my end. Matt, can you try clearing your cache?

Comment 7

[Matt Wobensmith [:mwobensmith][:matt:]]

It's up. I was someone switched off my VPN connection; now I'm back and site loads great.

Comment 8

[u279076]

Great, thank you Matt and thank you Rob for getting this fixed.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thank you Rob for taking care of it. One thing through... Have you observed the system load and processes before you restarted the whole machine?  I'm really eager to know what happened to understand the problem and so we can make sure it will not happen again. Rebooting the server is really the last resort. Thanks.