962219 - crash in RtlpWaitOnCriticalSection | RtlpDeCommitFreeBlock | winMutexEnter

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[u279076]

This bug was filed from the Socorro interface and is 
report bp-1f06ec5c-cd1d-42a1-b66a-f09f02140120.
=============================================================
0 	ntdll.dll 	RtlpWaitOnCriticalSection 	
1 	ntdll.dll 	RtlpDeCommitFreeBlock 	
2 	nss3.dll 	winMutexEnter 	db/sqlite3/src/sqlite3.c
3 	mozglue.dll 	je_malloc 	memory/mozjemalloc/jemalloc.c
4 	nss3.dll 	sqlite3_get_table 	db/sqlite3/src/sqlite3.c
5 	ESWBMon.dll 	ESWBMon.dll@0x7345 	
6 	comctl32.dll 	CreateAction 	
=============================================================
More reports:
https://crash-stats.mozilla.com/report/list?product=Firefox&signature=RtlpWaitOnCriticalSection+%7C+RtlpDeCommitFreeBlock+%7C+winMutexEnter

Strangely this crash is only showing up in the Aurora 28.0a2 topcrash reports and has spiked from #24 to #7 on Aurora in the last week. Unfortunately there are no comments nor correlations to investigate.

There are currently 152 crashes per 13 installs, so it seems this might be isolated to a handful of Aurora users. I'm not sure what could be going on here.

Comment 1

[u279076]

Ioana, can you have someone look into this to try to find steps to reproduce? I'm seeing youtube.com and livejournal.com mentioned frequently in URLs. In particular, http://alexstarc.livejournal.com.

Comment 2

[Alex Starchenko]

I've observed that issue a lot (seems lot of reports are mine). There's not much information to add, actually. The issue was observed in many different sites and looks like it's not related to any particular one (but on livejournal.com it was observed more frequently may be because it usually takes long time to load completely).
I've tried completely fresh install and start in safe mode - the issue was still there.
My installation has no specific plugins etc. There's some specific monitoring software is installed by my employer, but I think it shouldn't be an issue, because it haven't been observed previously (2-3 months ago).

Comment 3

[Ioana (away)]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #1)
> Ioana, can you have someone look into this to try to find steps to
> reproduce? I'm seeing youtube.com and livejournal.com mentioned frequently
> in URLs. In particular, http://alexstarc.livejournal.com.

Paul is working on this, so I've set him as QA contact too.

Comment 4

[Paul Silaghi, QA [:pauly]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #1)
> I'm seeing youtube.com and livejournal.com mentioned frequently
> in URLs. In particular, http://alexstarc.livejournal.com.
Tested on Aurora 28.0a2 (2014-01-22), Win 7 x86.
All I could get was a Flash crash (https://crash-stats.mozilla.com/report/index/23839207-692e-4156-ac18-0704c2140123) having many youtube videos and livejournal tabs opened.

Comment 5

[u279076]

(In reply to Alex Starchenko from comment #2)
> My installation has no specific plugins etc. There's some specific
> monitoring software is installed by my employer, but I think it shouldn't be
> an issue, because it haven't been observed previously (2-3 months ago).

It's completely possible that a recent change in Firefox is triggering a poor interaction with the monitoring software. Is there any chance you can do some testing without the monitoring software to see if it reproduces?

Comment 6

[Alex Starchenko]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #5)
> It's completely possible that a recent change in Firefox is triggering a
> poor interaction with the monitoring software. Is there any chance you can
> do some testing without the monitoring software to see if it reproduces?

Unfortunately, it's impossible. Without that software I'll not be able to reach network at all. If there's some way to provide more debug information, I would be glad to do so.

Comment 7

[u279076]

(In reply to Alex Starchenko from comment #6)
> If there's some way to provide more debug information, I would be glad to do so.

David, can you advise Alex on this?

Comment 8

[(Away)]

I think there are two separate bugs here, and our crash-stats server is confusing them into the same signature. The reports from Alex have very short call stacks with an external module on the stack. The reports from other machines have long stacks with JS and XPConnect near the top.

Alex, may I ask the name of that monitoring software? I see several third-party modules loaded, and I'm not sure which ones they are.

Comment 9

[Alex Starchenko]

(In reply to David Major [:dmajor] from comment #8)
> 
> Alex, may I ask the name of that monitoring software? I see several
> third-party modules loaded, and I'm not sure which ones they are.

It's ESCORT software (some description is available here sds.samsung.com/popup/solution/epoint.jsp). From crash information I can observe Anywall3.dll from that software. If You could provide modules names, I would be able to confirm if these modules from that software or not.

Comment 10

[(Away)]

Here are some modules that I don't recognize. I suspect they are related to each other because of the common folders, and also they all use a similar y.m.d version numbering system.

C:\Windows\incops3\ESWBMon.dll
C:\Windows\incops3\ICATCDLL.dll
C:\Windows\incops3\ICDCNL.dll
C:\Windows\incops3\ESSPD.dll
C:\Windows\incops3\ProcMon.dll
C:\Windows\pcdrm\NSCCOR03.dll
C:\Windows\pcdrm\NBID.dll
C:\Windows\pcdrm\NFD01.dll
C:\Windows\pcdrm\NSCPE.dll
C:\Windows\System32\anywall3.dll

In particular, ESWBMon.dll is the one that shows up in crash stacks.

Comment 11

[u279076]

(In reply to David Major [:dmajor] from comment #10)
> Here are some modules that I don't recognize. I suspect they are related to
> each other because of the common folders, and also they all use a similar
> y.m.d version numbering system.
> 
> C:\Windows\incops3\ESWBMon.dll
> C:\Windows\incops3\ICATCDLL.dll
> C:\Windows\incops3\ICDCNL.dll
> C:\Windows\incops3\ESSPD.dll
> C:\Windows\incops3\ProcMon.dll
> C:\Windows\pcdrm\NSCCOR03.dll
> C:\Windows\pcdrm\NBID.dll
> C:\Windows\pcdrm\NFD01.dll
> C:\Windows\pcdrm\NSCPE.dll
> C:\Windows\System32\anywall3.dll

I did some quick online research, all of these correlate to Samsung SDS.

Comment 12

[(Away)]

It's hard to say what's going on. I'm not sure if I even trust the stacks (the frame pointers don't look normal). The ESWBMon on the stack may be related or may be a red herring.

(In reply to Alex Starchenko from comment #6)
> (In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #5)
> > It's completely possible that a recent change in Firefox is triggering a
> > poor interaction with the monitoring software. Is there any chance you can
> > do some testing without the monitoring software to see if it reproduces?
> 
> Unfortunately, it's impossible. Without that software I'll not be able to
> reach network at all. If there's some way to provide more debug information,
> I would be glad to do so.

Alex, as an experiment, is it possible to disable the monitoring and then replay some previous HTTP traffic using a replay proxy? (Fiddler2, mitmproxy, web-page-replay, etc.) That should allow you to exercise browser code even if your network access is disabled.

Comment 13

[Alex Starchenko]

(In reply to David Major [:dmajor] from comment #12)
> Alex, as an experiment, is it possible to disable the monitoring and then
> replay some previous HTTP traffic using a replay proxy? (Fiddler2,
> mitmproxy, web-page-replay, etc.) That should allow you to exercise browser
> code even if your network access is disabled.

David, I've tried to do so, but currently it impossible, because it requires some kind of network admin password just to turn off that SW. I'll get back if find some way to disable it in order to reproduce the issue with replay proxy.

Comment 14

[Lukas Blakk [:lsblakk] use ?needinfo]

Does this reproduce for you on other channels?  Beta, Release, Nightly?  Though the crash is only showing up in high volume on Aurora (and you're the submitter of many of those reports) we could use your help figuring out a regression window here.

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Just because it's not topcrash on other versions doesn't mean those are unaffected, so let's wait to hear back from someone who can reproduce.

Comment 16

[(Away)]

Lukas, I'm not sure whether Alex's results will be useful for topcrash tracking. His crashes appear to be different from others in the same signature (comment 8).

But you're right that knowing whether this affects Beta or Release could help determine whether those particular crashes are a recent regression.

Comment 17

[Robert Kaiser]

Signature Summary on https://crash-stats.mozilla.com/report/list?product=Firefox&signature=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBlock+|+winMutexEnter says that the signature shows up on 26.0, 27.0b9, 28.0a2 (interestingly not 29.0a1 crash this week) and goes back to older versions as well, like 21.0, 16.0.2, and even 7.0b6 - it's also not Firefox-specific as it happens with SeaMonkey 2.23 as well (which is the current release based on Gecko/Platform 26.0).

Comment 18

[Alex Starchenko]

(In reply to Lukas Blakk [:lsblakk] from comment #14)
> Does this reproduce for you on other channels?  Beta, Release, Nightly? 
> Though the crash is only showing up in high volume on Aurora (and you're the
> submitter of many of those reports) we could use your help figuring out a
> regression window here.

Looks like it's reproducible in other versions also. I've tried todays Beta, Release and Aurora version. Using Beta and Release crash happened after longer time than with Aurora. Two reports submitted (one for Beta and another for Release) by me today with this bug number referenced in description. Looks on crash information crashes for all 3 versions looks the same for me.

(In reply to Robert Kaiser (:[e-mail]) from comment #17)
> Signature Summary on
> https://crash-stats.mozilla.com/report/
> list?product=Firefox&signature=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBl
> ock+|+winMutexEnter says that the signature shows up on 26.0, 27.0b9, 28.0a2
> (interestingly not 29.0a1 crash this week) and goes back to older versions
> as well, like 21.0, 16.0.2, and even 7.0b6 - it's also not Firefox-specific
> as it happens with SeaMonkey 2.23 as well (which is the current release
> based on Gecko/Platform 26.0).

Seems '29.0a1 crash this week' have happened just because I stopped sending reports.

Comment 19

[Lukas Blakk [:lsblakk] use ?needinfo]

Leaving the nomination for FF28 until beta 1 is out and we can assess if this is still a topcrash on windows at that time but given that this is a known crash going back to FF7 and that one person seems to have generated a lot of the crash data, I'm inclined to not track this.

Comment 20

[u279076]

Wontfix for Firefox 26 since it's reached EOL as of Tuesday. Follow are the current 7-day statistics for this signature.

0 reports for Firefox 30.0a1:
https://crash-stats.mozilla.com/query/?product=Firefox&version=Firefox%3A30.0a1&range_value=1&range_unit=weeks&date=02%2F07%2F2014+22%3A00%3A00&query_search=signature&query_type=contains&query=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBlock+|+winMutexEnter&reason=&release_channels=&build_id=&process_type=any&hang_type=any

0 reports for Firefox 29.0a2:
https://crash-stats.mozilla.com/query/?product=Firefox&version=Firefox%3A29.0a2&range_value=1&range_unit=weeks&date=02%2F07%2F2014+22%3A00%3A00&query_search=signature&query_type=contains&query=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBlock+|+winMutexEnter&reason=&release_channels=&build_id=&process_type=any&hang_type=any

28 reports for Firefox 28.0b:
https://crash-stats.mozilla.com/query/?product=Firefox&version=Firefox%3A28.0b&range_value=1&range_unit=weeks&date=02%2F07%2F2014+22%3A00%3A00&query_search=signature&query_type=contains&query=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBlock+|+winMutexEnter&reason=&release_channels=&build_id=&process_type=any&hang_type=any

26 reports for Firefox 27.0:
https://crash-stats.mozilla.com/query/?product=Firefox&version=Firefox%3A27.0&range_value=1&range_unit=weeks&date=02%2F07%2F2014+22%3A00%3A00&query_search=signature&query_type=contains&query=RtlpWaitOnCriticalSection+|+RtlpDeCommitFreeBlock+|+winMutexEnter&reason=&release_channels=&build_id=&process_type=any&hang_type=any

It's still early days since the merge so we should track for at least another few days but this looks promising.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

This no longer looks like a topcrash, Kairo can you confirm and adjust the keywords if that's correct?

Comment 22

[u279076]

(In reply to Lukas Blakk [:lsblakk] from comment #21)
> This no longer looks like a topcrash, Kairo can you confirm and adjust the
> keywords if that's correct?

This currently ranks outside of the top-300 on Nightly, #276 and dropping on Aurora, #279 and dropping on Beta, and outside of the top-300 on Release.

Looking at total reports for the last week I see no reports on Nightly, 6 reports on Aurora, 25 reports on the latest Beta, and 63 reports on Release.

Comment 23

[Saulo]

It is impossible to use Firefox with this bug... it is crashing every 5 min in may computer...

Comment 24

[u279076]

(In reply to saulo from comment #23)
> It is impossible to use Firefox with this bug... it is crashing every 5 min
> in may computer...

We've been unable to reproduce this internally. Could you please describe your system configuration and the events leading up to these crashes, particularly anything in common with each instance?

Comment 25

[Saulo]

  I have been trying to find the root cause of this problem for some time.

- There is no add ons or plugins installed;
- Not related to any particular website;
- Windows Events dont show anything;
- Tryed "Troubleshooting Information -> Reset" without success;
- I have escort software installed;

  What kind of information do you need?

Comment 26

[u279076]

(In reply to Saulo from comment #25)
> - I have escort software installed;

Could you try disabling this software to see if it has any effect?

Comment 27

[Saulo]

sorry, escort a software that I cant disable.

but I sure that this issue is relate to escort.

Comment 28

[u279076]

Unfortunately I think this is wontfix for Firefox 28, 29, and 30 at this point. At this point, I think we need an Engineer to work with the users in this bug who are able to reproduce the crash. Unless we get some traction on this soon, I'm not sure this is ever going to get resolved in our code.

Comment 29

[Robert Kaiser]

Well, this is low volume, but if we find some good way to reproduce, I guess dmajor is the person to work with people here as he already looked into this earlier for a bit.

Comment 30

[chrisrescobedo]

Hello,
I can confirm that I too was having this problem.
I could not use firefox at all since just having it open with a blank page would cause a crash in a few minutes.
The problem disappeared completely when I uninstalled the ESCORT software from my Win7x64 machine.
More information about ESCORT from Samsung's website:
http://www.sdse-samsung.com/serviceline/SmartSecurity.htm

'PC Security Solution (ESCORT): More and more corporates are continuously and frequently exposed to information leakage incidents nowadays. ESCORT as an integrated PC security solution blocks information leakages, manages security loopholes and your HW/SW assets.'

Comment 31

[Ryan VanderMeulen [:RyanVM]]

No reports in the last 28 days. Closing.