1290403 - Crash in chtbrkg.dll@0x2e1f0

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox :: Other, defect, P3)

Description

[Marco Castelluccio [:marco]]

Attachment: Correlations

This bug was filed from the Socorro interface and is 
report bp-abacab92-1441-41b4-8db7-ea9de2160729.
=============================================================

This is a new signature (first appeared on 2016-07-17), which is a top beta crasher (#32).

A sizeable chunk of crashes have the addon "@90B817C8-8A5C-413B-9DDD-B2C61ED6E79A", which, from a quick search, is probably malware.

~35% have the addon "Internet Download Manager integration".

I think we can't blocklist the library, as it is a Winsock LSP (it would require bug 1238735).

Can we blocklist the "@90B817C8-8A5C-413B-9DDD-B2C61ED6E79A" addon?

Comment 1

[Andrew Williamson [:eviljeff]]

Is there an indication of what the add-on is doing that is causing the crash?  We'd like to feedback to the developer what they can fix if possible to lessen the chance of the add-on just being redistributed under a different guid.

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

The add-on is on AMO (unlisted, ID 711032) and looks like search greyware. I'll leave it to the admin reviewers to determine if the add-on should be blocked.

I don't see binaries or references to binaries in the add-on (after a quick look), so blocking it wouldn't have any effect in the crashes.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'chtbrkg.dll@0x2e1f0':
 - nightly(version 50):1 crash from 2016-06-06.
 - aurora (version 49):0 crashes from 2016-06-07.
 - beta   (version 48):585 crashes from 2016-06-06.
 - release(version 47):676 crashes from 2016-05-31.
 - esr    (version 45):3 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       1       0       0       0       0       0       0
 - aurora        0       0       0       0       0       0       0
 - beta        312     263       0       0       0       0       0
 - release     419     232       3       0       0       0       0
 - esr           0       3       0       0       0       0       0

Affected platform: Windows

Comment 5

[[:philipp]]

this lsp module may be part of the pup/malware dubbed "adskip":
https://threatinfo.net/9387709741_adskip-exe-exe-the-executable-has-pe-anomalies-could-be-a-false-positive/

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'chtbrkg.dll@0x30ba1':
 - nightly (version 51): 1 crash from 2016-08-01.
 - aurora  (version 50): 13 crashes from 2016-08-01.
 - beta    (version 49): 463 crashes from 2016-08-02.
 - release (version 48): 461 crashes from 2016-07-25.
 - esr     (version 45): 0 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       0       1       0
 - aurora        7       0       0
 - beta        220      88      13
 - release     173     111      39
 - esr           0       0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser     Content   Plugin
 - nightly
 - aurora  #86
 - beta    #62
 - release #109
 - esr

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'chtbrkg.dll@0x19311':
 - nightly (version 52): 12 crashes from 2016-09-19.
 - aurora  (version 51): 33 crashes from 2016-09-19.
 - beta    (version 50): 979 crashes from 2016-09-20.
 - release (version 49): 1861 crashes from 2016-09-05.
 - esr     (version 45): 0 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly      12       0
 - aurora       33       0
 - beta        979       0
 - release    1861       0
 - esr           0       0

Affected platform: Windows

Crash rank on the last 7 days:
           Browser   Content     Plugin
 - nightly #69
 - aurora  #29
 - beta    #8
 - release #17       #3168
 - esr

Comment 8

[[:philipp]]

[Tracking Requested - why for this release]:
this issue is now accounting for 10% of startup crashes on 49.0.1 and 20% of startup crashes on 50.0b. due to the volume, maybe we could look into any short-term solutions here other than generally going for bug 1238735?

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Andreas, does it look fairly certain these crashes are from adskip?

We could write something on SUMO that explains how to detect and uninstall Adskip. 
Can we also replicate the issue, to make sure those instructions work?

Comment 10

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Cornel, can we try to replicate this with Adskip on our Win 10 machines that are on the isolated network? If it reproduces, we should also try to see if uninstalling Adskip fixes the problem.

Comment 11

[Andreas Wagner [:TheOne] [use NI]]

Do we still think this crash is related to the add-on mentioned in comment #0?

I can't see any reference to the adskip executable from the add-on.

Comment 12

[Marco Castelluccio [:marco]]

(In reply to Andreas Wagner [:TheOne] from comment #11)
> Do we still think this crash is related to the add-on mentioned in comment
> #0?
> 
> I can't see any reference to the adskip executable from the add-on.

I don't think so, the addon was correlated to the signature that at the time
was the top one (chtbrkg.dll@0x2e1f0).
The crashes with the signatures that are currently the top ones are happening
so early at startup that we basically don't have any info about addons (only
~2% of crashes have an 'addons' field != null).

The addons should probably still be blocked, since it is adware, but I don't
think blocking it would solve this crash.

Comment 13

[Andreas Wagner [:TheOne] [use NI]]

Ok, thanks! I'm afraid I can't help much in this case.

Comment 14

[guigs [:guigs] PST (please needinfo me!)]

I asked in the support forum today for the community to ask people to uninstall it if they see it in the crash reports. 

https://support.mozilla.org/en-US/forums/contributors/712155?last=70566&page=2#post-70566

I am waiting for joni to come back online for if its in scope of an article.

Comment 15

[Joni Chan]

We can write a troubleshooting article for this, but we'll need some more info to make sure we do it right and not further confuse users:

*Is there any way the average user can tell whether or not a crash is caused by this particular malware? If not, users will probably end up going to the support forums anyway, so a canned response might be a better solution.

*Who would be the right person to review our draft?

Comment 16

[Mihai Boldan, Desktop QA [:mboldan]]

I've tried to reproduce this crash on Firefox 50.0b6, under Windows 10 x64 and under Windows 7 x64, without any luck.

Here are the steps performed in order to reproduce the crash:
1. Installed "Internet Download Manager integration" add-on on Firefox 18, since the add-on is not available on the latest builds. 
2. Firefox autoupdated only to build 44.0.1, so I closed the working session and started Firefox 50.0b6 with the same profile from step 1, in order to have the "Internet Download Manager integration" add-on installed.
3. I've installed Adskip from http://blog.adskipbrowser.com/watch%EF%BC%9A-how-to-install-adskip-only-3-steps-4/.
4. I've accessed various websites with ads (that got blocked by Adskip) and I haven't encountered any crash after a few hours of testing.

Am I missing something here? Has anyone managed to reproduce this crash?

Comment 17

[Marco Castelluccio [:marco]]

Mihai, could you install this addon (https://addons.mozilla.org/en-US/firefox/addon/crash-me-now-simple/)
force a crash of the browser and paste here the URL for your crash report?

This way we can check if chtbrkg.dll was injected in Firefox.

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Mihai, could you please help with Marco's comment 17? Thanks!

Comment 19

[Mihai Boldan, Desktop QA [:mboldan]]

Here is the URL from the crash report: bp-b421026b-0fa1-493d-93c5-9066a2161017 . 
Please let me know if I can help any further with the investigation.

Comment 20

[Marco Castelluccio [:marco]]

Looks like the DLL is not being injected in your case.

Have you also installed this: http://www.internetdownloadmanager.com/download.html?

Comment 21

[Mihai Boldan, Desktop QA [:mboldan]]

(In reply to Marco Castelluccio [:marco] from comment #20)
> Looks like the DLL is not being injected in your case.
> 
> Have you also installed this:
> http://www.internetdownloadmanager.com/download.html?

Here is the URL from the forced crash report (used Crash me) after installing http://www.internetdownloadmanager.com/download.html: bp-43591798-5294-430f-a47a-2c0a92161017 .
Note that the Adskip app is working and the Blocked ads no. is increasing while navigating through Firefox. 

Firefox browser is still not crashing while performing various actions.

Comment 22

[Marco Castelluccio [:marco]]

The DLL is still not there. So it must be coming from somewhere else.
Perhaps the Adskip you installed is not the Adskip malware, but a legit software.

Comment 23

[Mihai Boldan, Desktop QA [:mboldan]]

I've installed Adskip from http://blog.adskipbrowser.com/watch%EF%BC%9A-how-to-install-adskip-only-3-steps-4/ . 
Do you have another link from where I could install the Adskip malware, since all I have found is how to uninstall the Adskip app/virus?

Comment 24

[Marco Castelluccio [:marco]]

Unfortunately I don't know where we can find it. It's also possible that they are the same thing but that the effects appear after a while.

Comment 25

[Marco Castelluccio [:marco]]

Have you also installed this: http://www.internetdownloadmanager.com/download.html?

If you haven't, can you try to install it and then make Firefox crash again?

Comment 26

[Mihai Boldan, Desktop QA [:mboldan]]

(In reply to Marco Castelluccio [:marco] from comment #25)
> Have you also installed this:
> http://www.internetdownloadmanager.com/download.html?
> 
> If you haven't, can you try to install it and then make Firefox crash again?

Yes, I've tested also with  http://www.internetdownloadmanager.com/download.html installed (see Comment 21).
I will perform another set of test in order to reproduce this issue and if I manage to reproduce it, I will notify you.
Please let me know if you have any new ideas.

Comment 27

[Marco Castelluccio [:marco]]

(In reply to Mihai Boldan, QA [:mboldan] from comment #26)
> (In reply to Marco Castelluccio [:marco] from comment #25)
> > Have you also installed this:
> > http://www.internetdownloadmanager.com/download.html?
> > 
> > If you haven't, can you try to install it and then make Firefox crash again?
> 
> Yes, I've tested also with 
> http://www.internetdownloadmanager.com/download.html installed (see Comment
> 21).
> I will perform another set of test in order to reproduce this issue and if I
> manage to reproduce it, I will notify you.
> Please let me know if you have any new ideas.

Sorry, I forgot I had already asked you and you had already replied :)
It will be impossible to reproduce this crash unless we find a way to get your machine affected.

Comment 28

[Liz Henry (:lizzard) (relman/hg->git project)]

jsavage: sorry I missed this. I'm not sure we know that adskip/Internet Download manager is the main cause. I'm happy to take a look at your draft.

Comment 29

[Liz Henry (:lizzard) (relman/hg->git project)]

Nicholas or bz..... Can you help us find a developer to investigate further?

Comment 30

[Liz Henry (:lizzard) (relman/hg->git project)]

(This is the #1 topcrash so far in 50 on release, and it was a topcrash all through 49.)

Comment 31

[Boris Zbarsky [:bzbarsky]]

Aren't we basically at the point where we know this is malware and we need bug 1238735 to block it?

Comment 32

[Nicholas Nethercote [inactive]]

I don't have any insight beyond what bz already said, sorry.

Comment 33

[Nicholas Nethercote [inactive]]

I've asked aklotz to look at bug 1238735 again.

Comment 34

[Liz Henry (:lizzard) (relman/hg->git project)]

Noting this is the #1 browser crash in beta 51.

Comment 35

[alex_mayorga]

¡Hola Charles!

From http://www.internetdownloadmanager.com/support/about_us.html :

"Internet Download Manager Corp. is a subsidiary of Tonec Inc. that develops Internet Applications since 1990."

Are you aware that your product might be being used to distribute malware that severely affects the stability of Firefox?

Could you please review and correct this?

¡Gracias!
Alex

Comment 36

[Charles Jones]

Hi Alex

We as Internet Download Manager vendor do not know anything about chtbrkg.dll, and we have no relationship to it. We would like to assist you, but we do not know how to cause this crash, and do not understand the relationship with our extension. What should we specifically install, and what links do we need to visit?

I noticed a link to a crash report in 19th & 21st comments where our extension was installed

https://crash-stats.mozilla.com/report/index/b421026b-0fa1-493d-93c5-9066a2161017

But it's a different crash, and it has a resolved status. So it's not clear what relationship it has to this subject.

Regarding chtbrkg.dll LSP, we can advise the following.

The file has neither version number, nor clear vendor name. Does it have a digital signature? Maybe this is a malware.

I would advise Firefox to show a warning like a suspicious "chtbrkg.dll" DLL from an unknown vendor listens your network traffic, and it is known to cause crashes in Firefox. Would you like Firefox to uninstall it?

When you have administrative rights in your installer, you can uninstall this DLL from LSP chain.

LSP is an old and buggy technology, and Microsoft does not recommend to use it anymore. Most respectable software vendors got rid of their LSPs in 2010, and 2011.

The problem is when you have a chain of providers, you need to remove a node and change the chain. There are LSP providers, which are installed incorrectly, not following Microsoft's recommendations. So you need to be careful in such cases, because when you delete LSP provider incorrectly, you may break the chain, and leave your users without the Internet. Maybe it would be easier just to delete this LSP when there are no other LSPs in the chain? This should be a most frequent situation.

Thank you

Regards

Charles Jones
Software engineer

Tonec Inc.

Comment 37

[Marco Castelluccio [:marco]]

I think someone is downloading a cracked version of IDM and is getting affected by the virus. So there's nothing IDM can do.

Comment 38

[Marco Castelluccio [:marco]]

By the way, the volume of this signature reduced more or less at the same time of the increase in BaseThreadInitThunk (bug 1322554).
I think the two crashes might be related (there are also several comments mentioning IDM in the other signature).

Comment 39

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 40

[Wayne Mery (:wsmwk)]

We have an example of *one* user in Thunderbird bp-8f4704ec-d55a-45a0-a35d-478862170405  chtbrkg.dll@0x1c3e1.
Appeared on rader because user crashed 30 time on startup.  no extensions listed in socorro tab

Comment 41

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Hello, I'm asking your help with an experiment with making decisions on bugs. You've been needinfo'ed on this bug. I'd like you to take one action to help this bug make progress toward a decision. The things you can do include:

* If you know or have a good guess of which product and component this bug belongs to, change the product and component of the bug
* If you know of the right person to ask about this bug, redirect the needinfo to them
* If you cannot reproduce the bug, close it

Comment 42

[Dão Gottwald [:dao]]

Is this even still an issue? How much has the volume gone done since four months ago / comment 38?

Comment 43

[Marco Castelluccio [:marco]]

The volume is much lower, but not negligible (almost in the top-50).

Comment 44

[Jim Mathies [:jimm]]

blocklist candidate

Comment 45

[Marco Castelluccio [:marco]]

This DLL can't be blocklisted unless we implement LSP blocklisting (bug 1238735).

Comment 46

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.

Comment 47

[Wayne Mery (:wsmwk)]

Marco
The signatures changed from chtbrkg.dll@<address> [1] to (mostly) chtbrkg.dll | WSARecv | <something>
Crash volume is approximately the same.

[1] https://crash-stats.mozilla.com/search/?signature=~chtbrkg.dll&product=Firefox&date=>%3D2018-12-01T00%3A43%3A00.000Z&date=<2019-01-01T00%3A43%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

[2] https://crash-stats.mozilla.com/search/?signature=~chtbrkg.dll&product=Firefox&date=>%3D2019-01-01T00%3A43%3A00.000Z&date=<2019-02-01T00%3A43%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Comment 48

[Marco Castelluccio [:marco]]

I filed https://github.com/mozilla/relman-auto-nag/issues/491 to handle better cases like this.

Comment 49

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.