1258072 - Hang in Nightly with stack sampling; happens when using a Mozilla internal Jenkins Ops tool

Status: RESOLVED DUPLICATE of bug 698882

(Core :: Networking, defect, P3)

Description

[John Morrison [:jrgm]]

Attachment: NightlyHang.txt

Hi Bill,

This is that hang that I experience when using a Jenkins internal ops tool. I'm attaching a process sample from Activity Monitor.

Some notes:
- E10S is not enabled in this profile. (Although, I used to have it on, and would see similar hangs).
- Nightly entered into this hang by initially burning 250% CPU for a few minutes, and then dropped to ~0% CPU and "Not Responding" showing in Activity Monitor.
- A Quit from Activity Monitor had no effect. I had to Force Quit.

Comment 1

[John Morrison [:jrgm]]

Attachment: experienced this same type of hang again; here's the process stack sample

Comment 2

[John Morrison [:jrgm]]

Attachment: I'm on a hot streak; here's another process stack sample in the hung state

Comment 3

[John Morrison [:jrgm]]

Attachment: and another hang stack sample

Do you have enough information from these four stack samples, or shall I just keep submitting more?

Comment 4

[John Morrison [:jrgm]]

Attachment: another one bites the dust

Comment 5

[John Morrison [:jrgm]]

Attachment: And another one gone

Comment 6

[John Morrison [:jrgm]]

Attachment: And another one gone

Comment 7

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

It looks like the call to PR_SetPollableEvent is expected to be non-blocking. But in this case we're writing so much data that we block waiting for the queue to empty. This may actually be an NSPR bug, but I'll needinfo Patrick since he probably has a better idea.

Comment 8

[Patrick McManus [:mcmanus]]

your timing is pretty amazing. This is a dup of bug 698882 which has been open for years and was just merged to mozilla-central one hour ago.

retest when it hits a nightly build?

so yes, PR_SetPollableEvent uses a blocking queue which is a serious bug, and it can cause a deadlock when tons of events are generated on the socket thread.. which rarely happens - but apparently jenkins makes it happen somehow? in any event, the deadlock should be fixed by 698882 whenever it sticks (it has a uncovered several unrelated latent bugs and been backed out a few times).

Comment 9

[John Morrison [:jrgm]]

Cool. I'll see if I get this hang again. (Given my recent rate of these hangs, if I don't see it in a week or so, it probably means it's been fixed).

Comment 10

[John Morrison [:jrgm]]

So, I believe I had the same hang in the past two weeks, but I didn't have time right then to capture a trace, as I had a more pressing problem to address. Sorry. If I trigger it again, and I have a bit of time to capture the stack, I will.

Comment 11

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 12

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258