938073 - Aborted testruns on OS X 10.9 nodes after hanging

Status: RESOLVED WORKSFORME

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Andreea Matei [:AndreeaMatei]]

We have all the testruns on 10.9 OS X nodes timeout, at the moment mm-osx-109-1 is offline and we're investigating the others.

Seems only 10.9 is affected, as the other versions work fine.

From what I noticed, after mounting the build, the browser opens but no tests starts, so the browser remains like that until timeout.

Example:
http://mm-ci-master.qa.scl3.mozilla.com:8080/job/release-mozilla-esr24_remote/442

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

mm-osx-109-4 had a Firefox instance running, that was responding but not running any tests. I restarted mm-osx-109-4 and started Jenkins, which then allowed the other nodes to continue. 

I found a similar situation on mm-osx-109-2, which I took offline and tried running the tests locally. The first attempt stalled after printing "Updating branch of test repository to 'mozilla-esr24'" to the console with a Firefox instance open. I interrupted this, stopped Firefox, and successfully switched to the target branch via the command line. I ran the testrun again and it run to completion.

I am suspecting an intermittent issue with launching Firefox on 10.9. We should try to replicate this locally and add debug to identify what's happening.

I did notice that com.apple.IconServicesAgent was using over 800MB on mm-osx-109-4, which appears to be a known Mavericks issue: https://discussions.apple.com/thread/5472367?start=0&tstart=0 it may or may not be related to this issue as other nodes with high values were running successfully, and some nodes did not have such high values.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

On an additional remote testrun on mm-osx-109-2 the tests stalled during the restart tests. I will restart this node and mm-osx-109-1 to see if we can get these back online.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

All nodes are back online and appear to be operating well. We need to keep an eye on these nodes and act quickly if we see a node hanging as it can have a large impact especially when running ondemand tests. If we see this, find the node that has caused the hang (usually the earliest started build, or the one with an orphaned Firefox instance). Take the node temporarily offline and abort the stalled build. Other builds should then recover and free up the queue.

As mentioned, we need to try to replicate this on OS X 10.9 and determine the underlying issue.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's strange. Have you had a chance to take a look at the system log? I wonder if some helpful information are listed there. If it happens again I would propose to not reboot one of the affected nodes but really put it in offline mode. This should only be done for one of them.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Henrik Skupin (:whimboo) from comment #4)
> That's strange. Have you had a chance to take a look at the system log? I
> wonder if some helpful information are listed there. If it happens again I
> would propose to not reboot one of the affected nodes but really put it in
> offline mode. This should only be done for one of them.

Yes, the system log had a lot of entries in it, similar to what we see in the Jenkins console. Given all this noise it's difficult to see anything useful. As mentioned above, we did put nodes into offline mode (mm-osx-109-1 was offline for the longest) but needed to restore full capacity for desktop testing.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Could you upload parts of the log during this phase as attachment just for reference? Would be good to have something for reference and starting point of investigation.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

As said, I believe we need to attempt to replicate this locally first on an OS X 10.9 installation. Andreea has said she will try to sort this out. I don't think uploading the system log for the time of these failures is currently a good use of my time. If it occurs again I will attach the system logs, however like I say I didn't see anything there that related directly to the issue.

Comment 8

[Andreea Matei [:AndreeaMatei]]

I have upgraded mine to 10.9 and currently is running testruns. I'll reply here if I can reproduce.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I don't think that this issue is easy to reproduce in short term. You might have to run tests on it for a couple of days.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Did this happen again? If not lets get this bug closed for now.

Comment 11

[Cosmin Malutan, [:cosmin-malutan]]

I haven't seen this failure this week so I close this for now.