1461697 - Background hang reporter causes shutdown hangs.

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Nicolas B. Pierron [:nbp]]

During shutdown of the ASan nightly version, after keeping it open for a day, and seeing too many hangs. I see a loop (during at least ~30 minutes) of the following error messages:

> ###!!! [Child][MessageChannel] Error: (msgtype=0x2D00FA,name=PContent::Msg_AccumulateChildKeyedHistograms) Closed channel: cannot send/recv
> 
> 
> ###!!! [Child][MessageChannel] Error: (msgtype=0x2D00FE,name=PContent::Msg_RecordDiscardedData) Closed channel: cannot send/recv
> 
> 
> ###!!! [Child][MessageChannel] Error: (msgtype=0x2D0105,name=PContent::Msg_BHRThreadHang) Closed channel: cannot send/recv

My hypothesis is that the BHR is attempting to report telemetry during shutdown, and the telemetry cannot submit any of these data for reasons unknown to me, thus causing it to hang.

Simple question: Do we have a mechanism to throttle down the BHR based on the number of hangs it already attempted to report?

Comment 1

[Marion Daly [:mdaly]]

(In reply to Nicolas B. Pierron [:nbp] {backlog: 36} from comment #0)
> During shutdown of the ASan nightly version, after keeping it open for a
> day, and seeing too many hangs. I see a loop (during at least ~30 minutes)
> of the following error messages:
> 
> > ###!!! [Child][MessageChannel] Error: (msgtype=0x2D00FA,name=PContent::Msg_AccumulateChildKeyedHistograms) Closed channel: cannot send/recv
> > 
> > 
> > ###!!! [Child][MessageChannel] Error: (msgtype=0x2D00FE,name=PContent::Msg_RecordDiscardedData) Closed channel: cannot send/recv
> > 
> > 
> > ###!!! [Child][MessageChannel] Error: (msgtype=0x2D0105,name=PContent::Msg_BHRThreadHang) Closed channel: cannot send/recv
> 
> My hypothesis is that the BHR is attempting to report telemetry during
> shutdown, and the telemetry cannot submit any of these data for reasons
> unknown to me, thus causing it to hang.
> 
> Simple question: Do we have a mechanism to throttle down the BHR based on
> the number of hangs it already attempted to report?

Could we get more details on this?

Comment 2

[Marion Daly [:mdaly]]

can we get more details on this?

Comment 3

[Nicolas B. Pierron [:nbp]]

(In reply to Marion Daly [:mdaly] from comment #2)
> can we get more details on this?

I was not able to reproduce it lately by starting the ASan build and leaving it idle for a few hours. I did not try to use these ASan nightly builds for a day.

Comment 4

[Andrew McCreight [:mccr8]]

FWIW, I have a patch in bug 1467549 to disable BHR in ASan.

Comment 5

[Andrew McCreight [:mccr8]]

BHR is disabled now in ASan.