838568 - OOM crash in gbmzh_abn.dll or gbmzh_uni.dll or gbmzh_bb.dll

Status: RESOLVED WONTFIX

(Firefox :: Extension Compatibility, defect)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-68a824b1-49b8-4b6a-8aee-a61792130206 .
============================================================= 

Those crashes started at 2013-02-01, and the gbmzh_abn one is now #37 in 18.* in yesterday's data.

mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | gbmzh_abn.dll@0x14b63|EXCEPTION_BREAKPOINT (443 crashes)
    100% (443/443) vs.   0% (727/174704) gbmzh_abn.dll (2.12.2.1)

More crashes at https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const%2A%20const%29%20%7C%20mozalloc_handle_oom%28unsigned%20int%29%20%7C%20moz_xmalloc%20%7C%20gbmzh_abn.dll%400x14b63

mozalloc_abort(char const* const) | mozalloc_handle_oom(unsigned int) | moz_xmalloc | gbmzh_uni.dll@0x14b63|EXCEPTION_BREAKPOINT (108 crashes)
    100% (108/108) vs.   0% (835/174704) gbmzh_uni.dll (2.12.2.4)

More crashes at: https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const%2A%20const%29%20%7C%20mozalloc_handle_oom%28unsigned%20int%29%20%7C%20moz_xmalloc%20%7C%20gbmzh_uni.dll%400x14b63

The gbmzh_*.dll names sounds like it would be a similar product to bug 704125, so I guess GAS Tecnologia is also behind it - also, the comments look like they're also Portuguese or Spanish (I have a hard time telling the difference, sorry).

Felipe and Sergio, I realize this might not be Banco do Brazil but maybe others, but you helped in bug 704125 and surely understand the comments a bit better (see the "more crashes" links above and their comments tabs), do you have any clue what those libs might be and how we might be able to contact the company producing this?

Comment 1

[Marcia Knous [:marcia]]

Adding Reuben to the bug as well, as he can help with the comments which seem to be in Portuguese.

Comment 2

[Sergio Oliveira Campos]

Looking at the dll names I would say it's from the banks ABN and Unibanco. I'm gonna check the reports later today to make sure. Anyway it's gonna be almost impossible to get it fixed in Brazil right before Carnaval.

Comment 3

[Sergio Oliveira Campos]

Almost as I said! *_abn.dll it's used by the Santander and *_uni.dll by Itaú and both of them are major players.

Kairo, do you have the contact of someone in GAS? At this point I'm not sure if we should contact the banks or GAS...

Comment 4

[Robert Kaiser]

(In reply to Sergio Oliveira Campos from comment #3)
> Kairo, do you have the contact of someone in GAS? At this point I'm not sure
> if we should contact the banks or GAS...

No, I don't have any contact, but I think it might actually be better to contact them as given the address in the two DLLs is the same, it probably is a problem across their code for different banks.

Comment 5

[Sergio Oliveira Campos]

Ok! They have a phone number in their website. I can try to call tomorrow and see if I can get to talk to a developer or a tech lead.

Comment 6

[Robert Kaiser]

Sergio, that would be great!

Comment 7

[Sergio Oliveira Campos]

Just spoke with Marlos from GAS and sent the bug URL to him. He said their developers and support team would be monitoring our stats and trying to find out what is wrong.

I've also asked him to try to give us some feedback on their progress either by email or Bugzilla.

Comment 8

[Robert Kaiser]

Thanks a lot, Sergio, that's great! I love how being a world-spanning community helps us in such cases! :)

Comment 9

[Sergio Oliveira Campos]

It's good to know that I'm been helpful ;)

Just to let you guys know, today it's first day of Carnaval so don't wait for any progress before February 18th. This is Brazil... :)

Comment 10

[Sergio Oliveira Campos]

Few minutes ago Marlos contacted me saying that they couldn't reproduce the crash.
I told him that all the crashes happened in 32bits systems (x86) and more than 90% on Windows 7 and XP (got that from crash stats).

Anyway this might not be enough. How can I get some of the URLs that the users were accessing when the crash happened.

If you can't disclose this info you can send it to my email or directly to GAS ([e-mail]).

Comment 11

[Robert Kaiser]

Top URLs:

gbmzh_abn.dll@0x14b63

99 	https://www.facebook.com/
81 	http://www.facebook.com/
50 	http://www.google.com.br/
45 	about:blank
41 	http://www.uol.com.br/
31 	http://www.globo.com/
23 	https://www.google.com.br/
18 	https://www.facebook.com/login.php?login_attempt=1
14 	https://mail.google.com/mail/u/0/?shva=1#inbox
13 	about:sessionrestore
11 	http://www.santander.com.br/
11 	http://www.facebook.com/?ref=tn_tnmn
11 	http://www.ig.com.br/


gbmzh_uni.dll@0x14b63

33 	http://www.itau.com.br/
27 	http://www.facebook.com/
21 	https://www.facebook.com/
17 	about:blank
16 	http://www.google.com.br/
15 	http://www.uol.com.br/
13 	https://www.google.com.br/
7 	https://www.facebook.com/login.php?login_attempt=1


gbmzh_bb.dll@0x14b63

22 	http://www.facebook.com/
15 	https://www.facebook.com/
13 	http://www.uol.com.br/
12 	http://www.bb.com.br/
7 	about:blank
6 	http://www.globo.com/
6 	http://www.google.com.br/
5 	about:sessionrestore

There more URLs with fewer hits than those listed, but not too sure if they will help much anyhow.

Almost all info we have is available in public crash-stats.

Comment 12

[Scoobidiver (away)]

It's #35 top browser crasher in 19.0.2.

It's correlated to one version:
    100% (225/225) vs.   0% (433/139851) gbmzh_bb.dll (2.12.2.45)

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char+const*+const%29+|+mozalloc_handle_oom%28unsigned+int%29+|+moz_xmalloc+|+gbmzh_bb.dll%400x14d03

Comment 13

[Reuben Morais [:reuben]]

Looking at bug 704125 and how users see this as a problem in the browser, we should seriously consider contacting the banks in some "official" way, and ask them not to use plugins. Hopefully in a way that would get picked up by the press.

Anyway, looking at the URLs, the plugin is apparently doing bad things even when it isn't in the foreground? Can we blacklist the problematic version?

Comment 14

[Scoobidiver (away)]

(In reply to Reuben Morais [:reuben] from comment #13)
> Can we blacklist the problematic version?
It's the latest one so users won't be able to login in to their bank.

Comment 15

[Robert Kaiser]

The gbmzh_abn.dll@0x15079 signature has exploded over the weekend in 19, it's #14 on https://crash-stats.mozilla.com/topcrasher/byversion/Firefox/19.0.2/3/browser now - and gbmzh_bb.dll@0x14d03 follows at #29. Not seen in 20 or higher yet, though.

Comment 16

[Scoobidiver (away)]

There are no crashes in 20.0 and above but 17.0.5esr is still affected.

Comment 17

[Scoobidiver (away)]

There are a few crashes in 20.0 and 20.0.1. It's currently #71 browser crasher in 20.0.1.

Comment 18

[Robert Kaiser]

gbmzh_bb.dll@0x15006 and gbmzh_uni.dll@0x15006 have been rising on 20 in the last few days, now at #54 and #70 in yesterday's overall (including plugins) 20.* topcrashes.

Comment 19

[Scoobidiver (away)]

With combined signatures, it's #16 top browser crasher in 20.0.1.

The affected versions are:
gbmzh_uni.dll 	2.12.3.8 		
gbmzh_bb.dll 	2.12.3.9
{87F8774F-B485-47E2-A755-A40A8A5E8873} 	2.12.3.8.200 	
{87F8774F-B485-47E2-A755-A40A8A5E886C} 	2.12.3.9.200

Comment 20

[Lukas Blakk [:lsblakk] use ?needinfo]

At #16, and if this is only on 20, there's no reason to track since we'll just wait for FF21 at this point.  Please nominate for 21 if the signature shows up/increases in volume there.

Comment 21

[Scoobidiver (away)]

(In reply to [e-mail] from comment #20)
> Please nominate for 21 if the signature shows up/increases in volume there.
It's now #14 top browser crasher in 20.0.1 (over the last week) and #6 over the last three days so the rising tendency is confirmed.

Comment 22

[Robert Kaiser]

(In reply to [e-mail] from comment #20)
> At #16, and if this is only on 20, there's no reason to track since we'll
> just wait for FF21 at this point.  Please nominate for 21 if the signature
> shows up/increases in volume there.

This will probably never show up in beta versions. It look like GAS is only making those compatible with release some time after we have released. In any case, the only ones who can fix this are the people at GAS who are creating those libraries. We need to contact them (once again).

Comment 23

[Sergio Oliveira Campos]

I've just sent another email to GAS and will try to reach them by phone in a couple hours.

I believe that last time we got this solved we blacklisted a version of their plugin, that caused the bank customers to massively call their banks and them the banks contacted GAS.

I totally understand that blocking will affect hundreds of users (including myself) but it has been about 80 days since I first contacted them and no actions were taken so far.

What we could do from the community side is to write blog posts saying that we are blocking GAS plugin with the proper explanation and than spread it using Facebook and Twitter.

Comment 24

[bhavana bajaj [:bajaj]]

(In reply to Sergio Oliveira Campos from comment #23)
> I've just sent another email to GAS and will try to reach them by phone in a
> couple hours.
> 
> I believe that last time we got this solved we blacklisted a version of
> their plugin, that caused the bank customers to massively call their banks
> and them the banks contacted GAS.
> 
> I totally understand that blocking will affect hundreds of users (including
> myself) but it has been about 80 days since I first contacted them and no
> actions were taken so far.
> 
> What we could do from the community side is to write blog posts saying that
> we are blocking GAS plugin with the proper explanation and than spread it
> using Facebook and Twitter.

Thanks for the outreach here.Let's wait for a couple of days to see GAS gets back, please keep us posted with any latest information you may have.Else we will consider the blog post approach you suggested above if nothing moves in this bug.

Comment 25

[Sergio Oliveira Campos]

GAS contacted me yesterday. They have said that the new version of the plugin was delivered more than a month ago to the banks but each one of them have a different qa process that can take few months before the release. Marlon also said that their manager is following this bug and the crash stats and he was surprised that he is not commenting here in the bug.

I've highlighted how bad the bug was and asked them to please reach the banks explaining the situation and asked to please interact more with us here in Bugzilla.

Comment 26

[Rafael Leandro]

Hi everyone,
My name's Rafael Leandro, I'm development manager at GAS Tecnologia.

Since we received the information about this issue on Firefox 18, we started a hard work in order to find out the 
solution. Unfortunately our Dev, QA and Lab teams have not reproduced the occurrence. In parallel we've 
created a new automated test with FF 18 and gbmzh version 2.12.2 to keep constantly (24x7) navigation in the most 
access urls reported, only after one week we got a crash in this enviroment which we immediately solved in version 2.12.3. This crash came to be caused after a change occurred in Firefox 18. The same code change doesn't exists in FF 17 and 
lower. After solution applied we kept the same test for weeks, adding in this moment FF 19 and after FF 20 and we never more got any crash (the automated test is still running).

Unfortunately, the occurrence seems to continue with this new version (2.12.3), according to this forum (Comment 19).
We are keeping our teams focused in this issue, however we didn't find any problem with this version yet.

Meanwhile, if any of you have more information, maybe an environment where the issue occurs consistently, or a step-by-step instructions to reproduce, please send us for verification. It would be extremely helpfull. Unluckily, all current informations on mozilla crash-stats doesn't help us enough.

In addition to efforts with version 2.12, we're releasing to our clients a newest version (3.x) containing (among other benefits) a new code architecture. Because of this new architecture, occurrences specifically like this will no longer happen. However, we must wait all internal homologation process (with its specific phases) before our clients put it in full production. Probably it will take a while.

Some clients - like Caixa and Santander - already have this new version and at the moment are distributing partially.

Comment 27

[Robert Kaiser]

Rafael, thanks for that information and looking forward to seeing if those issues go away with your 3.x versions. In the mean time, I can give you some URLs from the current crop of crashes with Firefox 20:

mozalloc_abort(char const* const) | moz_xmalloc | gbmzh_bb.dll@0x15006
124 	https://www.facebook.com/
73 	http://www.facebook.com/
64 	http://www.uol.com.br/
56 	http://www.google.com.br/
54 	about:blank
45 	http://www.bb.com.br/
41 	https://www.google.com.br/
26 	http://www.globo.com/
19 	https://www.facebook.com/home.php
18 	http://www.ig.com.br/
13 	https://mail.google.com/mail/?shva=1#inbox
12 	http://fast.globo.demdex.net/dest3.html?d_nsid=0#http%3A%2F%2Fwww.globo.com%2F
12 	https://mail.google.com/mail/u/0/?shva=1#inbox
11 	http://www.bb.com.br/portalbb/home29,116,116,1,1,1,1.bb
11 	https://sc.imp.live.com/content/dam/imp/surfaces/mail_signin/v3/mai/PT-BR.html
11 	https://www.facebook.com/?ref=tn_tnmn
10 	about:sessionrestore
10 	http://www.facebook.com/home.php
10 	http://www.facebook.com/?ref=tn_tnmn

mozalloc_abort(char const* const) | moz_xmalloc | gbmzh_uni.dll@0x15006 
151 	https://www.facebook.com/
103 	http://www.facebook.com/
91 	http://www.itau.com.br/
76 	http://www.google.com.br/
58 	http://www.uol.com.br/
56 	https://www.google.com.br/
48 	about:blank
42 	http://www.globo.com/
24 	https://bankline.itau.com.br/GRIPNET/bklcom.dll
20 	about:newtab
19 	http://www.itau.com.br/index.htm
19 	https://mail.google.com/mail/u/0/?shva=1#inbox
18 	http://fast.globo.demdex.net/dest3.html?d_nsid=0#http%3A%2F%2Fwww.globo.com%2F
17 	https://www.facebook.com/?ref=tn_tnmn
17 	https://mail.google.com/mail/?shva=1#inbox
16 	https://banklineplus.itau.com.br/GRIPNET/bklcom.dll
15 	http://www.ig.com.br/
14 	https://www.facebook.com/login.php?login_attempt=1
13 	http://www.facebook.com/?ref=tn_tnmn
12 	about:sessionrestore
12 	https://www.facebook.com/home.php
11 	http://www.sonico.com/chatroom.php?room=amistad_pt
10 	http://mail.mailig.ig.com.br/mail/?AuthEventSource=SSO#inbox

Also, you can see specific instances of this in the reports tabs of the following links and user comments in the "Comments" tabs:
https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const*%20const%29%20|%20moz_xmalloc%20|%20gbmzh_uni.dll%400x15006
https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort%28char%20const*%20const%29%20|%20moz_xmalloc%20|%20gbmzh_bb.dll%400x15006


And here's an analysis of crashes per installation:

breakpad=> SELECT version,COUNT(*) as crashes,COUNT(DISTINCT client_crash_date - install_age  * interval '1 second') as installations FROM reports WHERE product='Firefox' AND signature LIKE '%.dll@0x15006' AND date_processed BETWEEN '2013-03-24' AND '2013-05-01' GROUP BY version;
 version | crashes | installations 
---------+---------+---------------
 20.0    |      16 |             6
 20.0.1  |    8491 |          5241
(2 rows)

So this is seeing ~1.6 crashes per installation within a week (on Firefox 20.0.1).

Comment 28

[Alex Keybl [:akeybl]]

(In reply to Rafael Leandro from comment #26)
> In addition to efforts with version 2.12, we're releasing to our clients a
> newest version (3.x) containing (among other benefits) a new code
> architecture. Because of this new architecture, occurrences specifically
> like this will no longer happen. However, we must wait all internal
> homologation process (with its specific phases) before our clients put it in
> full production. Probably it will take a while.

Let us know when your newer versions are in the wild so that we can check back on the stability of newer versions. Untracking since the fix won't happen on our side.

Comment 29

[Rafael Leandro]

(In reply to Robert Kaiser (:[e-mail]) from comment #27)

Thanks for your informations Robert,
Since there isn't a pattern of websites, probably the problem isn't related to any specific url. I'd already seen tab "Comments" and there isn't specific information that help to find out the problem.

Did you get crashes in your environment with gbmzh? Can you share maybe a VM with us?

Comment 30

[Rafael Leandro]

(In reply to Alex Keybl [:akeybl] from comment #28)

> Let us know when your newer versions are in the wild so that we can check
> back on the stability of newer versions. Untracking since the fix won't
> happen on our side.

Hi Alex,
Look, we found one unexpected change in FF 18 code that affected our product and because of this we needed to change gbmzh (to v2.12.3), not exactly because we had a problem in gbmzh. Maybe there are others code changes in new FF that affects our product, we don't know yet. It would be nice to perform a check on changes in FF 18+ along with a dump of this crash. 

Look, the same gbmzh binary file, exactly the same code, with just gecko sdk updates, works nice in FF 15, 16, 17 but in FF 18+ crashed, why? We didn't get this crash in advance because it's hard to reproduce as I said. The question is: What exactly has changed? What changes are coming?

For example, the last code change in FF 18 we saw that affected our product was in the proxy code. This code in FF 18+ is different from the code in FF 17-. Unfortunately this change affected us.

I'm telling this because I think it's important you keep traking this issue in your side in order to try to find any improvement point in FF. I think we can not rule out the idea of ​​being a problem in Firefox that is affecting our product.

Each client has its own time to publish a new version, depends entirely of them. As I said, probably it will take a while. However, since two clients are distributing it partially (few users), there are computers with it installed. Currently version 3.2.0.2.

If you have any environment (virtual machines for example) with constant crashes, please send us for analysis. We can open a FTP account for it. Do you?

Regards!

Comment 31

[Scoobidiver (away)]

(In reply to Rafael Leandro from comment #30)
> Maybe there are others code changes in new FF that affects our product, we don't know 
> yet.
Changes impacting extensions are tracked in the compatibility section of the add-on blog: https://blog.mozilla.org/addons/category/compatibility/
Firefox 18 was there. See https://blog.mozilla.org/addons/2012/12/28/compatibility-for-firefox-18/

Comment 32

[Robert Kaiser]

(In reply to Rafael Leandro from comment #29)
> Did you get crashes in your environment with gbmzh? Can you share maybe a VM
> with us?

Sergio Oliveira Campos, a Mozilla community member who is on this bug report as well, might have seen it on his own machine from what I can gather - but we don't have any setup in our internal QA that even would have access to those modules.
All we have is the data sent to us via the crash reporting function that Firefox includes, and a good part of that data is public at the crash-stats.mozilla.com locations I posted in comment #27. We do have actual minidumps of those reports, but due to those containing private data of users, we have very strict policies on giving anyone insight into those. Still, you should be able to get a lot of information out of the public reports From looking into a few of the gbmzh_*.dll@0x15006 reports' modules lists, it looks like the current crashes with Firefox 20 are happening with gbmzh_uni.dll 2.12.3.8 or gbmzh_bb.dll 2.12.3.9 right now.

Comment 33

[Scoobidiver (away)]

There are currently no crashes in 21.0 with 15% of 20.0.1 users that have upgraded to 21.0.

It's still a top crasher in 17.0.6esr. Extension's binaries can't be compatible with both 17.0ESR and 21.0.

Comment 34

[Benjamin Smedberg]

This remain a pretty high issue for OOM crashes. It's not showing up on topcrash lists because there are many signatures involved.

Rafael, the prevalence of issues here basically indicates that your DLLs are leaking memory or something like that. The minidumps here are only going to give you the allocation locations, but not help figure out why this is happening. Could you please let me know whether you already know what's going on, or can use profiling to figure it out?

User crash comments show that they don't understand that it is your addon code which is causing these crashes: https://crash-stats.mozilla.com/search/?date=%3C%3D2013-11-26+21%3A00%3A00&signature=~gbmzh&user_comments=!__null__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=user_comments

Unless we can find a solution to this problem, we need to consider blocking the DLLs/addons here.

Comment 35

[Rafael Leandro]

Hi Benjamin,
I can't see the link "https://crash-stats.mozilla.com/search/?date=%3C%3D2013-11-26+21%3A00%3A00&signature=~gbmzh&user_comments=!__null__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=user_comments", when I access it I receive the message "No results were found"

Do you have information about versions of our product and Firefox? Anyway, a link with all information would be better, if possible.

Comment 36

[Benjamin Smedberg]

The crash-stats server was down for temporary maintenance at the time. The link works again, and you should be able to modify the query to suit your needs. Here is a new one that breaks down by Firefox version:

https://crash-stats.mozilla.com/search/?signature=~gbmzh&_facets=version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=user_comments

It's interesting to me that these crashes happen primarily with the Firefox 17 ESR release and somewhat with Firefox 20 (which is not up-to-date) but I don't see reports from our current release version of Firefox, which is Firefox 25.

Can you briefly explain why this DLL is loaded into Firefox at all? Is it via a Firefox extension or some other means? If it's from an extension, perhaps you should add a version check in your extension so that it doesn't load into older versions of Firefox which have this issue?

For users who are experiencing this issue, do you have instructions on how to disable the software in Firefox or uninstall it completely so that they can browse?

Comment 37

[Robert Kaiser]

If you set a needinfo, please also actually specify what info you request. That said, I'm personally not working on crashes any more.

Comment 38

[Jorge Villalobos [:jorgev] (he/him)]

Mass-closing old Extension Compatibility bugs that relate to legacy add-ons or NPAPI plug-ins. If you think this bug is still valid, please reopen or comment.

Sorry for the bug spam, and happy Friday!