1142384 - [MTBF] System crashed, wifi keep spinning

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Stability, defect)

Description

[Paul Yang [:pyang] (away)]

Attachment: dmp file

STR: Run mtbf-test for more than 12 hours
Reproduce rate: low

Wifi spinning and can't be brought up.
Logcat keeps printing "[Parent][MessageChannel] Error: Channel error: cannot send/recv" might be a ipc error.

Comment 1

[Paul Yang [:pyang] (away)]

Version info:
Build ID               20150310162504
Gaia Revision          5af6f8d5d6161dea02002634c6d0a570a122e5dd
Gaia Date              2015-03-10 19:17:12
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/ec87adb8cf13
Gecko Version          37.0
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150310.200728
Firmware Date          Tue Mar 10 20:07:39 EDT 2015
Bootloader             L1TC100118D0

Comment 2

[Paul Yang [:pyang] (away)]

Attachment: Symbol zip file

Comment 3

[Paul Yang [:pyang] (away)]

Vincent, can you provide comment for this issue?

Comment 4

[Ting-Yu Chou [:ting] (away)]

Attachment: stack.txt

The stack generated from attachment 8576428 [details] & 8576429.

Comment 5

[Paul Yang [:pyang] (away)]

Crash ID: bp-3e3cbe1f-7e4a-4d3f-bc8a-45bb72150312

Comment 6

[Vincent Chang[:vchang][changyihsin]]

It seems wpa_supplicant and wifi driver work fine. I use start wpa_supplicant command and wpa_cli to verify it manually. 

After restart b2g, I could use settings app to turn on/off wifi, and get AP list from wpa_cli scan command. However, I still could not see the scan list shown up on the settings apps.

Not sure what's happened here, may need to put some debug logs.

Comment 7

[Paul Yang [:pyang] (away)]

Vincent - would you like to provide build or patch so that we can get more information? thanks.

Comment 8

[Ken Chang[:kenkai]]

Henry, Since Vincent isn't in Taipei, can you please check this issue? Thanks.

Comment 9

[Henry Chang [:hchang]]

(In reply to Ken Chang[:ken](OOO from 2/18 to 3/1) from comment #8)
> Henry, Since Vincent isn't in Taipei, can you please check this issue?
> Thanks.

No problem. I'll take it a look!

Comment 10

[Henry Chang [:hchang]]

I actually don't any connection between the crash and the wifi issue...

Comment 11

[Henry Chang [:hchang]]

With gecko: ec87adb8cf13 and gaia: 5af6f8d5d6161de,

The wifi never ever shows scan result and keep printing 

"W/Settings( 1700): [JavaScript Error: "Error: wifiListStart mark not found" {file: "app://settings.gaiamobile.org/shared/js/usertiming.js" line: 130}]"

Paul,

Do you also see the same message?

Comment 12

[Paul Yang [:pyang] (away)]

This bug crashed and accidentally few logs left, so can't tell if above log appeared.
Will try to reproduce in next round.

Comment 13

[Henry Chang [:hchang]]

Attachment: Bug1142384.diff

As Arthur suggested, move around the window.performance.mark('wifiListStart')
to avoid race condition.

Comment 14

[Vincent Chang[:vchang][changyihsin]]

The bug mentioned in comment 13 is going to fix and land in Bug 1146208, and the crash doesn't related to WiFi in comment 10. So we would like to drop the bug and let others people to jump in.

Comment 15

[Paul Yang [:pyang] (away)]

Look like a crash in audioTrack, Bobby do we have a chance to see this?

Comment 16

[Paul Yang [:pyang] (away)]

ni? Alastor for audioTrack related

Comment 17

[Bobby Chien]

Hi Steven, could you have comments? I saw call stack is strange, it looks not possible to crash in audioInitTask.

Comment 18

[StevenLee[:slee]]

(In reply to Bobby Chien [:bchien] from comment #17)
> Hi Steven, could you have comments? I saw call stack is strange, it looks
> not possible to crash in audioInitTask.
Agree.

1. As the call stack shows, it crashed at "libxul.so!mozilla::RefPtr<mozilla::AudioInitTask>::~RefPtr() + 0xa", but AudioInitTask is running on "CubeInit" thread, [1]. 
2. From the call stack, the crash thread should be SocketTransportService thread. 

So that I think it should not be an audio related problem.

[1] https://dxr.mozilla.org/mozilla-central/source/dom/media/AudioStream.h#422

Comment 19

[Paul Yang [:pyang] (away)]

cancel ni? per comment 18

Comment 20

[Bobby Chien]

Jason, This issue is very rarely appear. However, it looks like crashed in HTTP stack. could you help to have comment on this? Thanks.

Comment 21

[Kai-Chih Hu [:khu]]

Doug, could you help to find someone to take a look on this bug? Thanks.

Comment 22

[Shian-Yow Wu [:swu]]

There are a lot of FennecAndroid crash report pointing to the same crash signature "mozilla::RefPtr<mozilla::AudioInitTask>::~RefPtr()", in each report the crash happens in different kind of threads.  Maybe this problem is not related to specific thread.

Comment 23

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

The function name of the call stack might be wrong because the |Release| and auto pointer destructor will be optimize to one function instance. We can see a lot of functions are mapping to the same address. |AsyncLatencyLogger::Release()| and |mozilla::RefPtr<mozilla::AudioInitTask>::~RefPtr()| happen to be the first entry of the group of that kind of functions in symbol file.

I think the following call stack is more reasonable by investigating the source code.

> 0  libxul.so!nsCOMPtr_base::~nsCOMPtr_base()
> 1  libxul.so!mozilla::net::EventTokenBucket::~EventTokenBucket() [nsCOMPtr.h : 344 + 0x7]
> 2  libxul.so!mozilla::net::EventTokenBucket::~EventTokenBucket() [EventTokenBucket.cpp:ec87adb8cf13 : 133 + 0x3]
> 3  libxul.so!mozilla::net::EventTokenBucket::Release()
> 4  libxul.so!mozilla::net::nsHttpConnectionMgr::OnMsgUpdateRequestTokenBucket(int, void*) [nsRefPtr.h : 47 + 0x5]
> 5  libxul.so!mozilla::net::nsHttpConnectionMgr::nsConnEvent::Run() [nsHttpConnectionMgr.h:ec87adb8cf13 : 631 + 0xb]

Comment 24

[Doug Turner (:dougt)]

maybe garvan can take a look. bounce it back if you can't.

Comment 25

[:garvan]

I don't know this code, and plate is full ATM, so I'll have to bounce it.

Some obvious things to try would be to null check param in OnMsgUpdateRequestTokenBucket()
https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpConnectionMgr.cpp#530

Might be worth pinging Patrick McManus, author of the code:
https://hg.mozilla.org/mozilla-central/diff/ecf37b2b9a96/netwerk/protocol/http/nsHttpConnectionMgr.cpp

Is there any useful/meaningful way to sanity check "param"? Is it possible that the EventTokenBucket had its refcount drop to zero before it gets to OnMsgUpdateRequestTokenBucket. I assume there is some async behaviour in this code, perhaps that introduces that possibility.

If there is going to be guessing happening, it would great to find some way to increase to probability of this crash. Not knowing the tests involved, I don't know if they can kicked into overdrive to trigger this bug faster.

Comment 26

[bhavana bajaj [:bajaj]]

(In reply to Garvan Keeley [:garvank] from comment #25)
> I don't know this code, and plate is full ATM, so I'll have to bounce it.
> 
> Some obvious things to try would be to null check param in
> OnMsgUpdateRequestTokenBucket()
> https://dxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/
> nsHttpConnectionMgr.cpp#530
> 
> Might be worth pinging Patrick McManus, author of the code:
> https://hg.mozilla.org/mozilla-central/diff/ecf37b2b9a96/netwerk/protocol/
> http/nsHttpConnectionMgr.cpp
> 
> Is there any useful/meaningful way to sanity check "param"? Is it possible
> that the EventTokenBucket had its refcount drop to zero before it gets to
> OnMsgUpdateRequestTokenBucket. I assume there is some async behaviour in
> this code, perhaps that introduces that possibility.
> 
> If there is going to be guessing happening, it would great to find some way
> to increase to probability of this crash. Not knowing the tests involved, I
> don't know if they can kicked into overdrive to trigger this bug faster.

Paul, are we hitting this now? Or can you trigger a local run to see if we can catch the test results and get more info here?

Comment 27

[Paul Yang [:pyang] (away)]

Haven't seen this issue for long time. I can try and see in our next trigger.

Comment 28

[Doug Turner (:dougt)]

see comment #25

Comment 29

[Patrick McManus [:mcmanus]]

so that member is only supposed to be assigned on the socket thread, and the stack trace looks fine.. however I did find one place where it is assinged on the main thread and during a pref change and that could be racing against the stack trace we see.. the backtrace that is included here has only 2 seconds of uptime, so it makes sense that it is reading the startup prefs.

I'm not certain this is your issue, but its worth giving it a try

Comment 30

[Patrick McManus [:mcmanus]]

Attachment: eventtokenbucket thread management

Comment 31

[Patrick McManus [:mcmanus]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a2c807a51b57

Comment 32

[Patrick McManus [:mcmanus]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/36c4d774fa03

Comment 33

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/36c4d774fa03

Comment 34

[Ryan VanderMeulen [:RyanVM]]

Please request b2g37 approval on this patch when you get a chance.

Comment 35

[Patrick McManus [:mcmanus]]

Comment on attachment 8597632 [details] [diff] [review]
eventtokenbucket thread management

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): long standing latent bug
User impact if declined: potential startup crashes. seen in qa mtbf test
Testing completed: regression only
Risk to taking this patch (and alternatives if risky): very low. it has had a month of platform coverage
String or UUID changes made by this patch: none

Comment 36

[Bobby Chien]

As comment 34 and comment 35, ni Josh to aware last minute request for v2.2.

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/c8eb4c10b0a7