1519123 - High proportion of out-of-memory crashes for users with Ghostery addon installed

Status: RESOLVED INVALID

(External Software Affecting Firefox :: Other, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is
report bp-085f1bcd-cb0e-45f7-a6ea-dee460190109.

Top 20 crash in 64 without a bug, not sure where to bucket it but it is called out as a plugins crash: https://bit.ly/2slyhhW. All versions of Windows are affected. This crash has been around in previous versions of Firefox, but not since around Firefox 50. The flash version in almost all of the crash reports shows as [blank].

One correlation to Ghostery, but it is less than 40%:
(36.36% in signature vs 01.90% overall) Addon "Ghostery – Privacy Ad Blocker" Version = 8.2.6 [47.34% vs 01.91% if process_type = null]

Top 7 frames of crashing thread:

0 xul.dll xul.dll@0xa51f05
1 xul.dll xul.dll@0x12c89f2
2 xul.dll xul.dll@0x8e3490
3 xul.dll xul.dll@0x8e2a3e
4 @0x16fc5cf5
5 @0x231ac8ef
6 @0x1701100f

=============================================================

Comment 1

[[:philipp]]

bug 1515487 changed how signatures for dll files are created. it trimmed the @0x... crashing address from all dll files in crashing signature.
that's why all those crashes show up under [@ xul.dll] now, when before they'd be spread out across countless different signatures. so these crashes aren't a regression per se.

Comment 2

[Trevor Skywalker]

I think these crash signatures are related.
(xul.dll)
(xul.dll | do_main)

All Windows versions are affected.
The Flash Version is [blank] for 100% of the crashes.
And the Ghostery – Privacy Ad Blocker is true for 40% of the crashes.

(40.07% in signature vs 02.12% overall) Addon "Ghostery – Privacy Ad Blocker" = true

Adding Crash Signature:
xul.dll | do_main

Comment 3

[Marcia Knous [:marcia]]

Adding the regression keyword so this gets into triage. This is visible in 66 beta so good to understand why it might be happening beyond the 40% correlation to Ghostery.

Comment 4

[Trevor Skywalker]

Top Crashers for Firefox 65.0
Top 300 Crashing Signatures. 7 days ago

#14 0.82% -0.02% xul.dll | do_main 311 311 0 0 289 0 2019-01-02

#17 0.67% -0.02% xul.dll 256 256 0 0 254 0 2019-01-02

Top Crashers for Firefox 65.0.1
Top 50 Crashing Signatures. 7 days ago

#10 0.8% new xul.dll | do_main 582 582 0 0 469 0 2019-01-02

#13 0.76% new xul.dll 557 557 0 0 456 0 2019-01-02

Top Crashers for Firefox 66.0b7
Top 50 Crashing Signatures. 7 days ago

#11 0.85% new xul.dll 68 68 0 0 58 0 2019-01-02

Top Crashers for Firefox 66.0b8
Top 50 Crashing Signatures. 7 days ago

#6 0.92% new xul.dll 67 67 0 0 61 0 2019-01-02

Comment 5

[[:philipp]]

good find! when the we don't have proper symbols for xul.dll in a crash report that's usually a symptom of a low memory situation according to https://bugzilla.mozilla.org/show_bug.cgi?id=1528668#c4

so we might want to reach out to ghostery and heave them look into optimising their memory footprint - close to 80% of browser crashes we're seeing from users on 32bit systems who have ghostery installed are oom crashes which is a strikingly high percentage:
https://crash-stats.mozilla.com/search/?addons=~ghostery.com&cpu_arch=x86&release_channel=release&product=Firefox&version=65.0&version=65.0.1&process_type=browser&date=%3E%3D2019-01-01#facet-signature

Comment 6

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Alexander, can you pass this along, put us in contact with the right engineers for Ghostery? Thanks!

Comment 7

[alexander]

Hi. Passed request to Ghostery team. They will react soon

Comment 8

[Christopher Tino]

Hey all, we've been trying to reproduce these crashes for the last week but haven't had much luck. Are we sure that this is specific to Ghostery? When looking at the crash report results that Philipp posted, I see similar numbers when filtering by other extension ids. For instance, using ABP's id:

https://crash-stats.mozilla.com/search/?addons=~d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d&cpu_arch=x86&release_channel=release&product=Firefox&version=65.0&version=65.0.1&process_type=browser&date=%3E%3D2019-01-01T00%3A00%3A00.000Z&date=%3C2019-03-07T18%3A33%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 9

[[:philipp]]

the search query without specifying the presence of any extensions results in about 20% clear cut out-of memory browser crashes, which we could take as a kind of baseline for firefox.
with adblock plus added to the query this number rises to 45% of all browser crashes being oom crashes for those users - with ghostery that number is in the range of 80% though, so extraordinarily high...

the correlation of [@ xul.dll] and [@ xul.dll | do_main] crash signatures to ghostery is also getting picked up independently by the tool at https://mozilla.github.io/stab-crashes/addon_related_signatures.html - ghostery is present in 40% of these (oom-related) crashes, while less than 1% of the userbase has the extension installed in the first place.

as with all out of memory issues in general it is quite difficult to pin-point the problem to a particular piece of code, as the resource use might creep up slowly over time or just in particular situations. but the available data shows that there is a strong correlation to ghostery, so it'd certainly be beneficial to vet the extension code and look into ways of minimizing its memory footprint.

Comment 10

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

This is a huge number of crashes, bad in 65 and just as bad in 66 but I'm still assuming that isn't new but is because of several signatures now appearing combined under this one as described in comment 1.

Comment 11

[Christopher Tino]

Thanks for clarifying. I see the correlation now looking at those four crash signatures. We'll keep digging and see if we can figure out what's causing this.

Comment 12

[Serge Zarembsky]

Trying to crash Firefox with Ghostery on Win7, 32bit, I has not yet succeeded, but noticed that about 95% of all reported crashes (Ghostery or no Ghostery) are on PCs with multiple processors (cpu count > 1 in the query). For OOM signatures with Ghostery installed it is even 99%. Thought that this info may be of interest in general.

Comment 13

[Serge Zarembsky]

The basic search I used was: https://crash-stats.mozilla.com/search/?addons=~ghostery.com&cpu_arch=x86&release_channel=release&cpu_count=%3E1&product=Firefox&version=65.0&version=65.0.1&platform=Windows&process_type=browser&date=%3E%3D2019-01-01T00%3A00%3A00.000Z&date=%3C2019-03-07T18%3A33%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 14

[Liz Henry (:lizzard Please n-i to RyanVM, jcristau, or pascal)]

Still 1000/week crashes for 66 release: https://crash-stats.mozilla.com/search/?addons=~ghostery.com&cpu_arch=x86&release_channel=release&cpu_count=%3E1&product=Firefox&version=66.0.3&platform=Windows&process_type=browser&date=%3E%3D2019-04-16T16%3A00%3A00.000Z&date=%3C2019-04-23T16%3A00%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 15

[Marcia Knous [:marcia]]

Christopher and Serge - We are still seeing high volume crashes on both 66 release and 67 beta. Have you been able to identify anything that might be causing this? Thanks.

Comment 16

[Christopher Tino]

We pushed a hotfix this week with a few optimizations but we're still trying to isolate the problem. We haven't been able to consistently reproduce the crashes on our test machines. Any insight there would be helpful.

That said, we have made overall performance improvements to our ad-block code which will land in the next release and should help memory consumption.

Comment 17

[Marcia Knous [:marcia]]

Came across a few Mac crashes on nightly that all have Ghostery and are crashing with Moz Crash reason MOZ_CRASH(Unhandlable OOM while clearing document dependent slots.): https://bit.ly/2Yq5qa1

Comment 18

[Marcia Knous [:marcia]]

This crash is currently #10 overall in the 67.0.4 release.

Comment 19

[Christopher Tino]

We've continued to dig into this and may have found the issue. There was an Array concat() as part of our onBeforeRequest handler that seemed to be eating up memory. We've removed that call and pushed a new release (v8.4.2) which is pending AMO review.

Comment 20

[[:philipp]]

thank you for the update, we're looking forward to the improvements.

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Christopher Tino from comment #19)

We've continued to dig into this and may have found the issue. There was an Array concat() as part of our onBeforeRequest handler that seemed to be eating up memory. We've removed that call and pushed a new release (v8.4.2) which is pending AMO review.

So this bug can be closed?

Comment 22

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:haik, since the bug has high severity, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 23

[Haik Aftandilian [:haik]]

Comments indicate this was fixed in Ghostery back in 2019. Using crash-stats to search for OOMs and aggregating by addons no longer shows a higher correlation for ghostery.