Crash in rlls.dll@0x6dd6a

NEW
Unassigned

Status

External Software Affecting Firefox
Other
P2
critical
3 months ago
27 days ago

People

(Reporter: marcia, Unassigned)

Tracking

(Blocks: 1 bug, {crash})

unspecified
x86
Windows
crash

Firefox Tracking Flags

(relnote-firefox 56+, firefox55 wontfix, firefox56+ wontfix, firefox57 affected, firefox58 affected)

Details

(crash signature)

(Reporter)

Description

3 months ago
This bug was filed from the Socorro interface and is 
report bp-fe261ba9-7cca-4858-aa1c-295520170828.
=============================================================

Seen while looking at crash stats. #37 top crash in beta 6, correlated to a RelevantKnowledge dll: http://bit.ly/2wN4u63

Comment 1

3 months ago
those modules got added to our dll blocklist in bug 1277846 - apparently they found a way to work around that now...
See Also: → bug 1277846
(Reporter)

Updated

3 months ago
Crash Signature: [@ rlls.dll@0x6dd6a] → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca]

Comment 2

3 months ago
can you help taking a look if there's something actionable in these reports?
Flags: needinfo?(ccorcoran)

Comment 3

3 months ago
the issue is occurring with a range of different signatures: https://crash-stats.mozilla.com/search/?signature=~ls.dll%400x&signature=~ls64.dll%400x
adding a few more to this bug report...
Crash Signature: [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98…
Flags: needinfo?(ccorcoran)
Andym or David, anyone available from your team to investigate or re-do the blocklisting? It is getting to be high volume on beta, so this may turn into a worse issue on release.  Thanks.
Flags: needinfo?(ddurst)
Flags: needinfo?(amckay)

Comment 5

3 months ago
Based on brief research about this, it seems we should just re-do the blocklisting. I'm not sure how this is being loaded or why the blocklisting in bug 1277846 isn't holding though.
Flags: needinfo?(amckay)

Comment 6

2 months ago
(/me agrees with andym. removes NI.)
Flags: needinfo?(ddurst)
Any idea how they are circumventing the blocklist?
Flags: needinfo?(aklotz)
dmajor, can you help with the blocklisting?
Flags: needinfo?(dmajor)
This may be related to our underlying problems with dll injection described in bug 1380335.
Sounds like dmajor and aklotz can't take this bug. 

I am asking for help on the stability mailing list in case we can figure out some kind of workaround. 

Adam, can you try contacting someone at RelevantKnowledge? Thanks.
status-firefox55: affected → wontfix
tracking-firefox56: --- → +
Flags: needinfo?(astevenson)
Reaching out over email to a couple paths, left a voicemail with a possible phone number for them as well.
Flags: needinfo?(astevenson)
Too busy with a11y+e10s to analyze this, sorry.
Flags: needinfo?(aklotz)
Looking at the "module" pings, we have ~20x more users on Release than on Beta with this DLL, but in percentage we have 2x less users on Release than on Beta.
Thanks Marco. So, we know to expect a crash spike for these signatures on 56 release, and we can't do much about it at the moment. We'll keep trying to contact the company. I don't consider this a release blocker for 56. 

I will add this the release notes for 56 as a known issue and link to the SUMO page on removing unwanted software (https://support.mozilla.org/kb/troubleshoot-firefox-issues-caused-malware)
relnote-firefox: --- → 56+
Marcia or philipp. I heard back from RelevantKnowledge support. They said they are unaware of any current issues and asked if we can provide specific details of the crash to help investigate. 

I'm not sure if there's enough information in this report as is, or if a summary would be helpful to them.
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(madperson)
We could use steps to reproduce. :philipp is this something you can take a look at?
Flags: needinfo?(dmajor)

Comment 16

2 months ago
because this software is commonly referred to as malicious i'd be uneasy of trying to reproduce the crash myself or working closer with the vendor to troubleshoot it (like sharing crash dumps, even if an affected user would agree to that).
ultimately the kind of injection they are doing is no longer supported or allowed as per https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/
Flags: needinfo?(madperson)

Comment 17

2 months ago
(In reply to Adam Stevenson [:adamopenweb] from comment #14)
> Marcia or philipp. I heard back from RelevantKnowledge support. They said
> they are unaware of any current issues and asked if we can provide specific
> details of the crash to help investigate. 
> 
> I'm not sure if there's enough information in this report as is, or if a
> summary would be helpful to them.

1) This vendor is injecting a DLL into our process space. The DLL names is 'rlls.dll'.

https://forums.malwarebytes.com/topic/183518-removal-instructions-for-relevantknowledge/

2) This DLL inserts itself into call stacks associated with networking through an api hook added by this DLL.

https://crash-stats.mozilla.com/report/index/d0dc1167-2917-47e6-b414-f14990170920

3) This DLL commonly crashes when handling these api hook callbacks.

https://crash-stats.mozilla.com/search/?signature=~rlls.dll&product=Firefox&date=%3E%3D2017-09-13T16%3A48%3A00.000Z&date=%3C2017-09-20T16%3A48%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Due to this we blocked this DLL from injecting back in the Firefox 48 release, which solved the browser stability problems (bug 1277846).

The vendor apparently updated their injection method such that now the DLL is back in our process space and causing stability issues again.
Thanks :philipp and :jimm. Will update when I hear back from them.
Flags: needinfo?(mozillamarcia.knous)

Updated

2 months ago
Blocks: 1306406
No word back from this company.  Using the search from comment 3, there are around 300 crashes per week with these signatures on 56.0.   On 56.0.1 (64 bit migration) still a few crashes.  I don't think we change anything dramatically here by the migration. Wontfix for 56.
status-firefox56: affected → wontfix
status-firefox57: --- → affected
status-firefox58: --- → affected
Sorry Liz for not updating this report. Last contact they said:

"We have figured out the reason for the crashes and no longer need any additional information. We are working on a fix and should have one implemented in the near future."

Updated

a month ago
Priority: -- → P2
Crash Signature: [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98… → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98…
Jim asked Adam to reach out to RelevantKnowledge again because this crash seems worse in Beta 57 than Firefox 56.0.

This crash affects all 32-bit Windows versions.
Crash Signature: [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98… → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98…
OS: Windows 7 → Windows

Comment 22

a month ago
rlls64.dll/pmls64.dll are the corresponding 64bit variants of this crashing module
Thanks Chris. Follow up has been sent.
They have identified what they believe will likely fix this issue and will be testing a build internally next week. If all goes well they should be able to roll out the fix worldwide by December.
You need to log in before you can comment on or make changes to this bug.