Currently the #33 topcrasher on 3.6b3.
Still trying to figure out what the actual stack is here, because I suspect that MSVC is lying to me. Here is what it shows:
strchr() Line 101 Asm
PR_SetEnv() Line 93
arena_dalloc_small() Line 4083
arena_dalloc() Line 4208
nsFactoryEntry::GetFactory() Line 3707
nsComponentManagerImpl::CreateInstanceByContractID() Line 1682
nsComponentManagerImpl::GetServiceByContractID() Line 2254
nsCOMPtr_base::assign_from_gs_contractid_with_error() Line 141
However as far as I can tell there is no way in heck that arena_dalloc_small will call PR_SetEnv, even indirectly.
So far it looks like the first two frames are correct, PR_SetEnv calling strchr, and that someone is calling PR_SetEnv with a null pointer.
More digging to follow....
If I'm doing the stack-math correctly (and I might very well not be), it looks like the caller of PR_SetEnv is something that we don't have code for in the minidump. I.e. an external dll.
Looking at 3 days worth of correlation data with modules, idmmzcc.dll is standing out:
83% (15/18) vs. 10% (1062/10465) idmmzcc.dll
95% (42/44) vs. 11% (1313/11946) idmmzcc.dll
97% (68/70) vs. 9% (901/10117) idmmzcc.dll
Nothing at all is showing up for addon correlations for two of the days, and nothing interesting on the third day.
Googling idmmzcc.dll shows that it belongs to "Internet Download Manager Module" from "Tonec Inc". Doesn't look like any type of malware.
Dbaron helped pull out some data showing that idmmzcc.dll is indeed the caller of PR_SetEnv here. Turns out that our minidumps contain information about where in memory each module is loaded.
Another interesting fact is that the string
appears higher up on the stack.
So I guess next step to contact these guys and ask them to look into fixing this.
However it is interesting to note that there was no addon correlations, so I wonder if they simply drop in a dll in our components directory, in which case beta 4 should not show this crash any more.
Henrik, could you possibly try out this software to see how they install themselves into Firefox.
The fact that there is no addon correlation in the crash data indicates that it's not installed as an addon. But since this is happening in 3.6b3 it *shouldn't* be due to them simply dropping a dll in our components directory (unless there is a bug in the whitelisting code there somewhere).
So the plan of action here is something like this:
1. Figure out how they get loaded, i.e. check what their installer does.
2. Based on that, see if we need to amend the component whitelisting code.
In parallel, we should contact the vendor and let them know about this crash. I'll shoot Damon an email about that.
Additionally, this is *possibly* related to bug 484576, though likely not.
I looked at if there's any specific versions crashing, and unfortunately it looks like the latest version is the crashiest of them all. Possibly because it's the one most heavily used.
strchr|EXCEPTION_ACCESS_VIOLATION (163 crashes)
68% (111/163) vs. 3% (2697/97540) idmmzcc.dll
0% (0/163) vs. 0% (16/97540) 188.8.131.52
0% (0/163) vs. 0% (1/97540) 184.108.40.206
0% (0/163) vs. 0% (28/97540) 220.127.116.11
0% (0/163) vs. 0% (2/97540) 18.104.22.168
0% (0/163) vs. 0% (4/97540) 22.214.171.124
0% (0/163) vs. 0% (21/97540) 126.96.36.199
2% (3/163) vs. 0% (84/97540) 188.8.131.52
1% (1/163) vs. 0% (30/97540) 184.108.40.206
0% (0/163) vs. 0% (51/97540) 220.127.116.11
1% (1/163) vs. 0% (41/97540) 18.104.22.168
2% (3/163) vs. 0% (157/97540) 22.214.171.124
7% (12/163) vs. 0% (169/97540) 126.96.36.199
0% (0/163) vs. 0% (15/97540) 188.8.131.52
1% (2/163) vs. 0% (9/97540) 184.108.40.206
0% (0/163) vs. 0% (44/97540) 220.127.116.11
1% (2/163) vs. 0% (81/97540) 18.104.22.168
4% (6/163) vs. 0% (215/97540) 22.214.171.124
13% (22/163) vs. 0% (423/97540) 126.96.36.199
1% (1/163) vs. 0% (97/97540) 188.8.131.52
36% (58/163) vs. 1% (1209/97540) 184.108.40.206
(In reply to comment #4)
> Henrik, could you possibly try out this software to see how they install
> themselves into Firefox.
> The fact that there is no addon correlation in the crash data indicates that
> it's not installed as an addon. But since this is happening in 3.6b3 it
> *shouldn't* be due to them simply dropping a dll in our components directory
> (unless there is a bug in the whitelisting code there somewhere).
I have tested this application in the last two days while working out the list of 3rd party applications (https://wiki.mozilla.org/QA/Firefox3.6/TestPlan:DLL_Blocklisting:3rd-party).
> 1. Figure out how they get loaded, i.e. check what their installer does.
The add-on is installed into %appdata%/idm and registered via a registry entry under HKCU/software/mozilla/firefox/extensions. Could that be the cause why we don't list it in the list of interested add-ons? Chris, do you have an idea?
> 2. Based on that, see if we need to amend the component whitelisting code.
No, that's not related here. The DLL is placed into the components folder of the add-on.
FWIW yesterdays interested add-ons list for 3.6b3 builds shows:
strchr|EXCEPTION_ACCESS_VIOLATION (55 crashes)
15% (8/55) vs. 1% (169/13730) firstname.lastname@example.org
Not quiet high but it's another download manager. Given that I can also see crashes with DAP installed. Could that be related to how download managers integrate into Firefox?
Strange that another download manager is crashing with the same signature. Though I'm tempted to say it's a coincidence. Or possibly the two are sharing code.
Henrik: So when you install Internet Download Manager, it shows up as a normal extension in the addon manager? Not as a plugin or something else?
(In reply to comment #8)
> Henrik: So when you install Internet Download Manager, it shows up as a normal
> extension in the addon manager? Not as a plugin or something else?
Given my document from above the integration is made via an add-on. It's not placed inside the profile or application folder but in %appdata%/idm. With the registry key which has been added the extension can be found by Firefox.
It's hard to get older versions of IDM. The only site I have found is: http://www.oldversion.com/Internet-Download-Manager.html
I'll get in contact with some reporters and ask them for an update and if it is reproducible with the latest version of IDM. I cannot reproduce this crash for now.
Henrik: The crash data indicated that the latest version was affected too, see comment 5. At this point I'm actually mostly interested to figure out why the add on doesn't show up in our addon correlation data.
You didn't actually answer my question regarding if the addon shows up in the normal firefox addon manager?
Also, if it does. Can you disable/uninstall it using the addon manager as well?
Please give me a bit time. I was playing around with those older versions and the latest one (5.18 Build 5) which is available via download on the official website. But the version of the idmmzcc.dll in that build is 220.127.116.11. So the crashes we see are with older versions of IDM. The news site of IDM (http://www.internetdownloadmanager.com/news.html) tells about some critical fixes which went into the latest versions. So it could be that the crash we are seeing here is already fixed in Build 5.
We should think about blocking older versions of this add-on.
(In reply to comment #10)
> comment 5. At this point I'm actually mostly interested to figure out why the
> add on doesn't show up in our addon correlation data.
Good question. I tried that with the crashme extension and sometimes all the extensions aren't fetched! Ted, do you know if we have a bug on that? It's really important because the list of interested add-ons is one of our most visited input data. An example you can see here:
All extensions: bp-e919ffb7-d698-4f15-ae01-2e59b2091130
No extensions: bp-7bbd38ef-ef70-4a2c-85d0-02e662091130
> You didn't actually answer my question regarding if the addon shows up in the
> normal firefox addon manager?
Yes, it shows up as a normal extension.
> Also, if it does. Can you disable/uninstall it using the addon manager as well?
I can disable it but an uninstallation is not possible from Firefox side. You have to disable the browser integration within the IDM options panel.
Mossop wrote the add-on collection piece. The only thing I can offer is that if we crash on startup, we might not get to that point, so we might not send any information about installed extensions.
Oh, those are both "crash me now" crashes. I don't know, you'd have to ask Mossop.
(In reply to comment #13)
> Oh, those are both "crash me now" crashes. I don't know, you'd have to ask
Yes, for that example. Does it differ from other "normal" crash situations? I don't think so. Shall we better move to another bug? I can file one.
I filed bug 531870. Thanks for the discovery Henrik!
Oh, and if we do blocklist here, we should probably blocklist *up to* version 6.8, if that's possible. Recent crash stats look like this:
strchr|EXCEPTION_ACCESS_VIOLATION (126 crashes)
80% (101/126) vs. 10% (2101/21747) idmmzcc.dll
0% (0/126) vs. 0% (16/21747) 18.104.22.168
0% (0/126) vs. 0% (11/21747) 22.214.171.124
0% (0/126) vs. 0% (10/21747) 126.96.36.199
0% (0/126) vs. 0% (30/21747) 188.8.131.52
0% (0/126) vs. 0% (24/21747) 184.108.40.206
0% (0/126) vs. 0% (7/21747) 220.127.116.11
8% (10/126) vs. 0% (30/21747) 18.104.22.168
6% (7/126) vs. 0% (14/21747) 22.214.171.124
0% (0/126) vs. 0% (69/21747) 126.96.36.199
0% (0/126) vs. 1% (133/21747) 188.8.131.52
0% (0/126) vs. 0% (30/21747) 184.108.40.206
0% (0/126) vs. 0% (2/21747) 220.127.116.11
0% (0/126) vs. 0% (17/21747) 18.104.22.168
1% (1/126) vs. 0% (82/21747) 22.214.171.124
10% (13/126) vs. 1% (140/21747) 126.96.36.199
2% (3/126) vs. 1% (147/21747) 188.8.131.52
0% (0/126) vs. 0% (80/21747) 184.108.40.206
2% (3/126) vs. 3% (720/21747) 220.127.116.11
51% (64/126) vs. 2% (539/21747) 18.104.22.168
So there's several older version too that are crashing, not just the latest.
I just had a whole succession of these crashes, on attempted startup, and I DO NOT have Internet Download Manager.
In my case, the culprit ended up being HTML Validator 0.8.6.1: http://users.skynet.be/mgueury/mozilla/
Frame Module Signature [Expand] Source
0 mozcrt19.dll strchr strchr.asm:101
iu: please file a new bug, and please use https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg to try to get a better stack. ted: can we arrange to trap .dmp's where the call stack length of the crashing thread is 1?
No. But we keep .dmp files for all crashes for a few days now, you just need to log in to Socorro and click the "raw dump" tab to download the original.
Filed bug 541946 on the HTML Validator crash.
I think we can now CLOSE this bug; IDM has been updated many times in the last two years with lots of bugfixes regarding browser integration.
Feel free to REOPEN if this bug is still reproducible.