Open Bug 1519123 Opened 11 months ago Updated 4 months ago

High proportion of out-of-memory crashes for users with Ghostery addon installed

Categories

(External Software Affecting Firefox :: Other, defect, critical)

x86
Windows
defect
Not set
critical

Tracking

(firefox-esr60 affected, firefox64 wontfix, firefox65 wontfix, firefox66 wontfix, firefox67 wontfix, firefox68 affected, firefox69 affected)

Tracking Status
firefox-esr60 --- affected
firefox64 --- wontfix
firefox65 --- wontfix
firefox66 --- wontfix
firefox67 --- wontfix
firefox68 --- affected
firefox69 --- affected

People

(Reporter: marcia, Unassigned, NeedInfo)

Details

(Keywords: crash)

Crash Data

This bug was filed from the Socorro interface and is
report bp-085f1bcd-cb0e-45f7-a6ea-dee460190109.

Top 20 crash in 64 without a bug, not sure where to bucket it but it is called out as a plugins crash: https://bit.ly/2slyhhW. All versions of Windows are affected. This crash has been around in previous versions of Firefox, but not since around Firefox 50. The flash version in almost all of the crash reports shows as [blank].

One correlation to Ghostery, but it is less than 40%:
(36.36% in signature vs 01.90% overall) Addon "Ghostery – Privacy Ad Blocker" Version = 8.2.6 [47.34% vs 01.91% if process_type = null]

Top 7 frames of crashing thread:

0 xul.dll xul.dll@0xa51f05
1 xul.dll xul.dll@0x12c89f2
2 xul.dll xul.dll@0x8e3490
3 xul.dll xul.dll@0x8e2a3e
4 @0x16fc5cf5
5 @0x231ac8ef
6 @0x1701100f

=============================================================

bug 1515487 changed how signatures for dll files are created. it trimmed the @0x... crashing address from all dll files in crashing signature.
that's why all those crashes show up under [@ xul.dll] now, when before they'd be spread out across countless different signatures. so these crashes aren't a regression per se.

Keywords: regression
OS: Windows 7 → Windows

I think these crash signatures are related.
(xul.dll)
(xul.dll | do_main)

All Windows versions are affected.
The Flash Version is [blank] for 100% of the crashes.
And the Ghostery – Privacy Ad Blocker is true for 40% of the crashes.

(40.07% in signature vs 02.12% overall) Addon "Ghostery – Privacy Ad Blocker" = true

Adding Crash Signature:
xul.dll | do_main

Crash Signature: [@ xul.dll] → [@ xul.dll] [@ xul.dll | do_main ]

Adding the regression keyword so this gets into triage. This is visible in 66 beta so good to understand why it might be happening beyond the 40% correlation to Ghostery.

Top Crashers for Firefox 65.0
Top 300 Crashing Signatures. 7 days ago

#14 0.82% -0.02% xul.dll | do_main 311 311 0 0 289 0 2019-01-02

#17 0.67% -0.02% xul.dll 256 256 0 0 254 0 2019-01-02

Top Crashers for Firefox 65.0.1
Top 50 Crashing Signatures. 7 days ago

#10 0.8% new xul.dll | do_main 582 582 0 0 469 0 2019-01-02

#13 0.76% new xul.dll 557 557 0 0 456 0 2019-01-02

Top Crashers for Firefox 66.0b7
Top 50 Crashing Signatures. 7 days ago

#11 0.85% new xul.dll 68 68 0 0 58 0 2019-01-02

Top Crashers for Firefox 66.0b8
Top 50 Crashing Signatures. 7 days ago

#6 0.92% new xul.dll 67 67 0 0 61 0 2019-01-02

Keywords: top50, topcrash

good find! when the we don't have proper symbols for xul.dll in a crash report that's usually a symptom of a low memory situation according to https://bugzilla.mozilla.org/show_bug.cgi?id=1528668#c4

so we might want to reach out to ghostery and heave them look into optimising their memory footprint - close to 80% of browser crashes we're seeing from users on 32bit systems who have ghostery installed are oom crashes which is a strikingly high percentage:
https://crash-stats.mozilla.com/search/?addons=~ghostery.com&cpu_arch=x86&release_channel=release&product=Firefox&version=65.0&version=65.0.1&process_type=browser&date=%3E%3D2019-01-01#facet-signature

Crash Signature: [@ xul.dll] [@ xul.dll | do_main ] → [@ xul.dll] [@ xul.dll | do_main ] [@ xul.dll | _PR_NativeRunThread | pr_root ] [@ OOM | small]
Component: Plug-ins → Other
Product: Core → External Software Affecting Firefox
Summary: Crash in xul.dll → High proportion of out-of-memory crashes for users with Ghostery addon installed
Version: 65 Branch → unspecified

Alexander, can you pass this along, put us in contact with the right engineers for Ghostery? Thanks!

Flags: needinfo?(alexander)

Hi. Passed request to Ghostery team. They will react soon

Flags: needinfo?(alexander)

Hey all, we've been trying to reproduce these crashes for the last week but haven't had much luck. Are we sure that this is specific to Ghostery? When looking at the crash report results that Philipp posted, I see similar numbers when filtering by other extension ids. For instance, using ABP's id:

https://crash-stats.mozilla.com/search/?addons=~d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d&cpu_arch=x86&release_channel=release&product=Firefox&version=65.0&version=65.0.1&process_type=browser&date=%3E%3D2019-01-01T00%3A00%3A00.000Z&date=%3C2019-03-07T18%3A33%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

the search query without specifying the presence of any extensions results in about 20% clear cut out-of memory browser crashes, which we could take as a kind of baseline for firefox.
with adblock plus added to the query this number rises to 45% of all browser crashes being oom crashes for those users - with ghostery that number is in the range of 80% though, so extraordinarily high...

the correlation of [@ xul.dll] and [@ xul.dll | do_main] crash signatures to ghostery is also getting picked up independently by the tool at https://mozilla.github.io/stab-crashes/addon_related_signatures.html - ghostery is present in 40% of these (oom-related) crashes, while less than 1% of the userbase has the extension installed in the first place.

as with all out of memory issues in general it is quite difficult to pin-point the problem to a particular piece of code, as the resource use might creep up slowly over time or just in particular situations. but the available data shows that there is a strong correlation to ghostery, so it'd certainly be beneficial to vet the extension code and look into ways of minimizing its memory footprint.

This is a huge number of crashes, bad in 65 and just as bad in 66 but I'm still assuming that isn't new but is because of several signatures now appearing combined under this one as described in comment 1.

Thanks for clarifying. I see the correlation now looking at those four crash signatures. We'll keep digging and see if we can figure out what's causing this.

Trying to crash Firefox with Ghostery on Win7, 32bit, I has not yet succeeded, but noticed that about 95% of all reported crashes (Ghostery or no Ghostery) are on PCs with multiple processors (cpu count > 1 in the query). For OOM signatures with Ghostery installed it is even 99%. Thought that this info may be of interest in general.

Christopher and Serge - We are still seeing high volume crashes on both 66 release and 67 beta. Have you been able to identify anything that might be causing this? Thanks.

Flags: needinfo?(serge)
Flags: needinfo?(ctino)

We pushed a hotfix this week with a few optimizations but we're still trying to isolate the problem. We haven't been able to consistently reproduce the crashes on our test machines. Any insight there would be helpful.

That said, we have made overall performance improvements to our ad-block code which will land in the next release and should help memory consumption.

Flags: needinfo?(ctino)

Came across a few Mac crashes on nightly that all have Ghostery and are crashing with Moz Crash reason MOZ_CRASH(Unhandlable OOM while clearing document dependent slots.): https://bit.ly/2Yq5qa1

This crash is currently #10 overall in the 67.0.4 release.

We've continued to dig into this and may have found the issue. There was an Array concat() as part of our onBeforeRequest handler that seemed to be eating up memory. We've removed that call and pushed a new release (v8.4.2) which is pending AMO review.

thank you for the update, we're looking forward to the improvements.

You need to log in before you can comment on or make changes to this bug.