This bug was filed from the Socorro interface and is report bp-fe261ba9-7cca-4858-aa1c-295520170828. ============================================================= Seen while looking at crash stats. #37 top crash in beta 6, correlated to a RelevantKnowledge dll: http://bit.ly/2wN4u63
those modules got added to our dll blocklist in bug 1277846 - apparently they found a way to work around that now...
can you help taking a look if there's something actionable in these reports?
the issue is occurring with a range of different signatures: https://crash-stats.mozilla.com/search/?signature=~ls.dll%400x&signature=~ls64.dll%400x adding a few more to this bug report...
Andym or David, anyone available from your team to investigate or re-do the blocklisting? It is getting to be high volume on beta, so this may turn into a worse issue on release. Thanks.
Based on brief research about this, it seems we should just re-do the blocklisting. I'm not sure how this is being loaded or why the blocklisting in bug 1277846 isn't holding though.
(/me agrees with andym. removes NI.)
Any idea how they are circumventing the blocklist?
dmajor, can you help with the blocklisting?
This may be related to our underlying problems with dll injection described in bug 1380335. Sounds like dmajor and aklotz can't take this bug. I am asking for help on the stability mailing list in case we can figure out some kind of workaround. Adam, can you try contacting someone at RelevantKnowledge? Thanks.
Reaching out over email to a couple paths, left a voicemail with a possible phone number for them as well.
Too busy with a11y+e10s to analyze this, sorry.
Looking at the "module" pings, we have ~20x more users on Release than on Beta with this DLL, but in percentage we have 2x less users on Release than on Beta.
Thanks Marco. So, we know to expect a crash spike for these signatures on 56 release, and we can't do much about it at the moment. We'll keep trying to contact the company. I don't consider this a release blocker for 56. I will add this the release notes for 56 as a known issue and link to the SUMO page on removing unwanted software (https://support.mozilla.org/kb/troubleshoot-firefox-issues-caused-malware)
Marcia or philipp. I heard back from RelevantKnowledge support. They said they are unaware of any current issues and asked if we can provide specific details of the crash to help investigate. I'm not sure if there's enough information in this report as is, or if a summary would be helpful to them.
We could use steps to reproduce. :philipp is this something you can take a look at?
because this software is commonly referred to as malicious i'd be uneasy of trying to reproduce the crash myself or working closer with the vendor to troubleshoot it (like sharing crash dumps, even if an affected user would agree to that). ultimately the kind of injection they are doing is no longer supported or allowed as per https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/
(In reply to Adam Stevenson [:adamopenweb] from comment #14) > Marcia or philipp. I heard back from RelevantKnowledge support. They said > they are unaware of any current issues and asked if we can provide specific > details of the crash to help investigate. > > I'm not sure if there's enough information in this report as is, or if a > summary would be helpful to them. 1) This vendor is injecting a DLL into our process space. The DLL names is 'rlls.dll'. https://forums.malwarebytes.com/topic/183518-removal-instructions-for-relevantknowledge/ 2) This DLL inserts itself into call stacks associated with networking through an api hook added by this DLL. https://crash-stats.mozilla.com/report/index/d0dc1167-2917-47e6-b414-f14990170920 3) This DLL commonly crashes when handling these api hook callbacks. https://crash-stats.mozilla.com/search/?signature=~rlls.dll&product=Firefox&date=%3E%3D2017-09-13T16%3A48%3A00.000Z&date=%3C2017-09-20T16%3A48%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature Due to this we blocked this DLL from injecting back in the Firefox 48 release, which solved the browser stability problems (bug 1277846). The vendor apparently updated their injection method such that now the DLL is back in our process space and causing stability issues again.
Thanks :philipp and :jimm. Will update when I hear back from them.
No word back from this company. Using the search from comment 3, there are around 300 crashes per week with these signatures on 56.0. On 56.0.1 (64 bit migration) still a few crashes. I don't think we change anything dramatically here by the migration. Wontfix for 56.
Sorry Liz for not updating this report. Last contact they said: "We have figured out the reason for the crashes and no longer need any additional information. We are working on a fix and should have one implemented in the near future."
Jim asked Adam to reach out to RelevantKnowledge again because this crash seems worse in Beta 57 than Firefox 56.0. This crash affects all 32-bit Windows versions.
rlls64.dll/pmls64.dll are the corresponding 64bit variants of this crashing module
Thanks Chris. Follow up has been sent.
They have identified what they believe will likely fix this issue and will be testing a build internally next week. If all goes well they should be able to roll out the fix worldwide by December.