Closed Bug 1261015 Opened 10 years ago Closed 10 years ago

GC-related crashes due to Roboform Addon

Categories

(Core :: General, defect)

Product:
Core
Component:
General
Version:
45 Branch
Platform:
x86
Windows NT
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox45 --- wontfix
firefox46 --- wontfix
firefox47 --- wontfix
firefox48 --- affected
firefox49 --- affected
firefox-esr45 --- wontfix
firefox50 --- affected

People

(Reporter: philipp, Unassigned)

References

Details

(Keywords: crash)

Crash Data

This bug was filed from the Socorro interface and is report bp-00fe3e21-ef05-4ca1-91ee-ccb402160325. ============================================================= Crashing Thread (0) Frame Module Signature Source 0 xul.dll JS::Heap<JSObject*>::~Heap<JSObject*>() js/public/RootingAPI.h 1 xul.dll js::ctypes::CClosure::Finalize js/src/ctypes/CTypes.cpp 2 xul.dll FinalizeTypedArenas<JSObject> js/src/jsgc.cpp 3 xul.dll FinalizeArenas js/src/jsgc.cpp 4 xul.dll js::gc::ArenaLists::forceFinalizeNow(js::FreeOp*, js::gc::AllocKind, js::gc::ArenaLists::KeepArenasEnum, js::gc::ArenaHeader**) js/src/jsgc.cpp 5 xul.dll js::gc::ArenaLists::queueForegroundObjectsForSweep(js::FreeOp*) js/src/jsgc.cpp 6 xul.dll js::gc::GCRuntime::beginSweepingZoneGroup() js/src/jsgc.cpp 7 xul.dll js::gc::GCRuntime::beginSweepPhase(bool) js/src/jsgc.cpp 8 xul.dll js::gc::GCRuntime::gcCycle(bool, js::SliceBudget&, JS::gcreason::Reason) js/src/jsgc.cpp 9 xul.dll js::gc::GCRuntime::collect(bool, js::SliceBudget, JS::gcreason::Reason) js/src/jsgc.cpp 10 xul.dll js::gc::GCRuntime::gc(JSGCInvocationKind, JS::gcreason::Reason) js/src/jsgc.cpp 11 xul.dll JS_GC(JSRuntime*) js/src/jsapi.cpp 12 xul.dll nsXREDirProvider::DoShutdown() toolkit/xre/nsXREDirProvider.cpp 13 xul.dll xul.dll@0x1d25383 14 xul.dll nsAString_internal::ReplacePrep(unsigned int, unsigned int, unsigned int) xpcom/string/nsTSubstring.cpp 15 xul.dll nsLocalFile::AppendInternal(nsString const&, bool) xpcom/io/nsLocalFileWin.cpp 16 xul.dll nsLocalFile::Append(nsAString_internal const&) xpcom/io/nsLocalFileWin.cpp crashes with this signature are around in volume since a few firefox versions, in 45 it's making up ~0.2% of crashes. in addon correlation data, this looks to be strongly related to the presence of the Roboform Toolbar add-on from https://addons.mozilla.org/addon/750: JS::Heap<T>::~Heap<T>|EXCEPTION_ACCESS_VIOLATION_READ (128 crashes) 98% (125/128) vs. 0% (374/84733) {22119944-ED35-4ab1-910B-E619EA06A115}
I've contacted the developers about this.
And what version of RoboForm is this? I do not see any indication, and we had a lot of cases where people with old version have problems.
this is the version specific drill-down... JS::Heap<T>::~Heap<T>|EXCEPTION_ACCESS_VIOLATION_READ (128 crashes) 98% (125/128) vs. 0% (374/84733) {22119944-ED35-4ab1-910B-E619EA06A115} (Roboform Toolbar for Firefox, https://addons.mozilla.org/addon/750) 1% (1/128) vs. 0% (1/84733) 7.9.11.1 0% (0/128) vs. 0% (1/84733) 7.9.11.5 1% (1/128) vs. 0% (2/84733) 7.9.12.2 0% (0/128) vs. 0% (1/84733) 7.9.15.0 0% (0/128) vs. 0% (1/84733) 7.9.15.5 18% (23/128) vs. 0% (69/84733) 7.9.15.9 61% (78/128) vs. 0% (240/84733) 7.9.16.9 13% (17/128) vs. 0% (50/84733) 7.9.18.0 1% (1/128) vs. 0% (2/84733) 7.9.7.133 2% (2/128) vs. 0% (3/84733) 7.9.7.5 1% (1/128) vs. 0% (3/84733) 7.9.8.5 1% (1/128) vs. 0% (1/84733) 7.9.9.1
The only thing it can be is RoboForm2Go. it uses binary APIs that change from version to version. we are thinking about discontinuing it for that reason. do you have a path to RoboForm.dll, it would show us whether this is fixed or 2go distribution
I had this with Firefox 46 and Roboform Desktop 7.9.18.5: https://crash-stats.mozilla.com/report/index/a499eba6-7d42-4caa-930b-c3cec2160503 Refreshing the tab with Twitter or something. No idea if Roboform was really part of the crash or not.. crash report just pointed to this.
Crash volume for signature 'JS::Heap<T>::~Heap<T>': - nightly (version 50): 14 crashes from 2016-06-06. - aurora (version 49): 32 crashes from 2016-06-07. - beta (version 48): 1454 crashes from 2016-06-06. - release (version 47): 6789 crashes from 2016-05-31. - esr (version 45): 486 crashes from 2016-04-07. Crash volume on the last weeks: Week N-1 Week N-2 Week N-3 Week N-4 Week N-5 Week N-6 Week N-7 - nightly 1 0 0 3 2 5 3 - aurora 2 4 8 6 5 3 1 - beta 229 200 192 264 256 174 65 - release 1068 1008 983 987 1058 950 358 - esr 61 54 52 50 47 51 37 Affected platform: Windows
status-firefox47: --- → affected
status-firefox48: --- → affected
status-firefox49: --- → affected
status-firefox50: --- → affected
status-firefox-esr45: --- → affected
We will investigate. Yes, RF2Go uses one or two binary APIs, that may change from version to version. BTW, we will discontinue RF2Go in RF ver 8, this being one of the reasons. Will be only 'installed' RoboForm starting from ver 8.
Should we blocklist the addon? When do you plan to discontinue it?
Flags: needinfo?(vm)
Current version of RoboForm is 7.9.20. So If you blacklist version 7.9.11 with a clear path to upgrade this would be OK. user ca get the latest version of RoboForm here: http://www.roboform.com/download
Unfortunately most of the crashes (more than 80%) are occurring with version 7.9.18.
Current version of RoboForm is 7.9.20. SO if you provide a clear upgrade path, you can block 7.9.18 too. Users should be directed to http://www.roboform.com/download to upgrade
Depends on: 1293964
We are seeing a few crashes where the addon version is different than the DLL version, do you know why? For example, https://crash-stats.mozilla.com/report/index/035f3a12-7400-4854-a52b-2f0092160805. The addon version is 7.9.18.0, the DLL version is 7.9.20.5.
(In reply to Marco Castelluccio [:marco] from comment #12) > We are seeing a few crashes where the addon version is different than the > DLL version, do you know why? > > For example, > https://crash-stats.mozilla.com/report/index/035f3a12-7400-4854-a52b- > 2f0092160805. The addon version is 7.9.18.0, the DLL version is 7.9.20.5. Actually, in most crashes in the last few days (582 out of 588) the addon version is different than the DLL version.
This appears to be a genuine crash, not related to Rf2Go and attachment to browser from outside. We cannot reproduce it in our testing, so we will do more analysis. Do you have more detailed crash files (maybe with memory)?
(In reply to Vadim Maslov from comment #14) > This appears to be a genuine crash, not related to Rf2Go and attachment to > browser from outside. It's strange because most (basically all) of the crashes have Roboform installed and have the roboform.dll DLL in the stack trace. > We cannot reproduce it in our testing, so we will do more analysis. Is it normal that there's a mismatch between the addon version and the DLL version? Shouldn't the version of the addon and the DLL be the same for a given user? > Do you have more detailed crash files (maybe with memory)? Mark, would you be willing to share information from your crash report with Roboform? The crash report contains private information.
Flags: needinfo?(bugzilla)
(In reply to Vadim Maslov from comment #11) > Current version of RoboForm is 7.9.20. > > SO if you provide a clear upgrade path, you can block 7.9.18 too. > > Users should be directed to > http://www.roboform.com/download > to upgrade 7.9.18 was the last version submitted for signing under the current guid {22119944-ED35-4ab1-910B-E619EA06A115} - are you using a different add-on id?
My latest crash of this was https://crash-stats.mozilla.com/report/index/2fbde40d-9fe4-4fe6-a7eb-cc3882160728 with Firefox 47.0.1 and Roboform Toolbar 7.9.18.0, assuming it was the cause. I am currently on Firefox 48 and Roboform Desktop 7.9.20.5 (which it thinks is the newest) and Toolbar 7.9.18.0 and have not had this crash on it yet. You can share this with Roboform. I'm sure whatever information there is they already got from support tickets I have filed with them over the years. It never was reproducible. Some seeming random series of events caused it.
Flags: needinfo?(bugzilla)
Benjamin, how can we proceed here? The reporter agreed to share his crash data in comment 17.
Flags: needinfo?(benjamin)
please send it to me at @vm@siber.com ideally we want crashes from several users to establish a pattern
(In reply to Vadim Maslov from comment #19) > please send it to me at @vm@siber.com > > ideally we want crashes from several users to establish a pattern In the meantime, could you reply to the questions in comment 15 and comment 16?
Actually our Firefox extension may lag behind 2-3 versions from RoboForm.DLL. This is because Mozilla made it extra hard to submit new versions of extensions. And our extension is compatible with the newer DLL. So difference is version number of RoboForm.DLL and RoboForm extension does not mean nothing. As to this crash, we cannot reproduce it, so it is caused by some combination of factors that we do not have.
Marco, since you have user permission you may send minidumps as requested. The fundamental issue here is that Roboform is sticking a DLL into the Firefox process. Even avoiding XPCOM and using ctypes, this is an obvious risk, and will not be allowed as we move from old-style addons to webextensions. Roboform should move to a more stable technology stack such as using native messaging ASAP.
Flags: needinfo?(benjamin)
418 crash reports have been analyzed and the following versions have been found: - {22119944-ed35-4ab1-910b-e619ea06a115} - 7.9.16.9: 25 - 7.9.12.2: 1 - 7.9.8.5: 1 - 7.9.15.9: 30 - 7.9.18.0: 343 - roboform.dll - 7.9.17.5: 20 - 7.9.8.5: 1 - 7.9.19.7: 207 - 7.9.16.7: 29 - 7.9.19.5: 16 - 7.9.18.5: 31 - 7.9.20.5: 64 - 7.9.18.0: 1 - 7.9.15.5: 2 - 7.9.12.2: 1
So we need actual crash files with memory. So far we did not get anything that would allow us to reproduce this crash. Our own crash collector does not catch it, as it is not C++ part.
(In reply to Vadim Maslov from comment #24) > So we need actual crash files with memory. > > So far we did not get anything that would allow us to reproduce this crash. > > Our own crash collector does not catch it, as it is not C++ part. Vadim, I've just sent you the two reports that Mark agreed to share with you.
I did not receive any reports. Can you tell me what email they will be coming from.
Our developers today say that they fixed it in RF extension for Firefox ver 7-9-20-5 that was just submitted to FF gallery. Since we could not reproduce it, this is only a tentative fix and we need to see how it behaves in the wild.
Correction: Since FF gallery does not have auto-update of extension, we will have to do new RF ver 7-9-21 that will install new RF extension into FF. You can get it here: http://www.roboform.com/dist/RoboForm-Setup-7-9-21-0.exe We will release it tomorrow or on Friday
I've installed this version and will update if it happens again. I have not had one of these crashes since using Firefox 48. Last one was with 47.0.1 as I mentioned earlier.
(In reply to Vadim Maslov from comment #26) > I did not receive any reports. > > Can you tell me what email they will be coming from. I've sent it to vm@siber.com with my email address. (In reply to Vadim Maslov from comment #27) > Our developers today say that they fixed it in RF extension for Firefox ver > 7-9-20-5 > that was just submitted to FF gallery. > > Since we could not reproduce it, this is only a tentative fix and we need to > see how it behaves in the wild. Still seeing crashes with 7.9.20.5, nothing yet with 7.9.21. Has 7.9.21 been released? 293 crash reports have been analyzed and the following versions have been found: - {22119944-ed35-4ab1-910b-e619ea06a115} - 7.9.16.9: 22 - 7.9.15.9: 25 - 7.9.11.1: 1 - 7.9.18.0: 231 - roboform.dll - 7.9.17.5: 18 - 7.9.19.7: 83 - 7.9.16.7: 25 - 7.9.19.5: 3 - 7.9.18.5: 15 - 7.9.11.1: 3 - 8.0.5.0: 1 - 7.9.18.0: 1 - 7.9.20.5: 108 - 8.1.0.0: 1
7.9.21 has been released yesterday, Aug 25. it contains fix for Firefox crashes.
(In reply to Vadim Maslov from comment #31) > 7.9.21 has been released yesterday, Aug 25. > > it contains fix for Firefox crashes. What is the latest version of the addon (and the version of the DLL) that is affected?
This bug can appear up to a version before release, that is, up to ver 7-9-20. Fixed in ver 7-9-21, versions are both of extension and URL.
For now, 7.9.21 doesn't seem to be affected, so your fix probably worked. 1245 crash reports have been analyzed and the following versions have been found: - {22119944-ed35-4ab1-910b-e619ea06a115} - 7.9.15.0: 4 - 7.9.12.2: 1 - 7.9.16.9: 108 - 7.9.18.0: 931 - 7.9.7.5: 5 - 7.9.10.1: 1 - 7.9.14.4: 2 - 7.9.13.5: 1 - 7.9.15.9: 124 - roboform.dll - 7.9.17.5: 96 - 7.9.19.7: 372 - 7.9.16.7: 105 - 7.9.19.5: 27 - 7.9.18.5: 97 - 7.9.20.5: 371 - 7.9.18.0: 7 - 7.9.7.5: 5 - 7.9.15.5: 9 - 8.1.0.0: 2 - 7.9.14.6: 1 - 7.9.16.6: 6 - 7.9.14.4: 2 - 7.9.10.1: 1 - 7.9.0.0: 2 - 7.9.13.5: 1 - 7.9.7.133: 1
Other two signatures that are related to RoboForm are 'js::gc::StoreBuffer::unputCell' (~80% of crashes with this signature have the addon) and 'JS::HeapObjectPostBarrier' (100% of crashes with this signature have the addon).
Crash Signature: [@ JS::Heap<T>::~Heap<T>] → [@ JS::Heap<T>::~Heap<T>] [@ JS::HeapObjectPostBarrier ] [@ js::gc::StoreBuffer::unputCell ]
Summary: crash in JS::Heap<T>::~Heap<T> correlated to Roboform Addon → GC-related crashes due to Roboform Addon
Crash Signature: [@ JS::Heap<T>::~Heap<T>] [@ JS::HeapObjectPostBarrier ] [@ js::gc::StoreBuffer::unputCell ] → [@ JS::Heap<T>::~Heap<T>] [@ JS::HeapObjectPostBarrier ] [@ js::gc::StoreBuffer::unputCell ] [@ js::ctypes::ClosureInfo::~ClosureInfo ]
Resolved by blocklisting older versions of the Roboform Addon.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: needinfo?(vm)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.