Open Bug 1394550 Opened 2 years ago Updated 8 months ago

Crash in rlls.dll@0x6dd6a

Categories

(External Software Affecting Firefox :: Other, defect, P3, critical)

x86
Windows
defect

Tracking

(relnote-firefox 56+, firefox55 wontfix, firefox56+ wontfix, firefox57 wontfix, firefox58 wontfix, firefox59+ wontfix, firefox60+ wontfix, firefox61 wontfix, firefox62 affected, firefox63 affected)

Tracking Status
relnote-firefox --- 56+
firefox55 --- wontfix
firefox56 + wontfix
firefox57 --- wontfix
firefox58 --- wontfix
firefox59 + wontfix
firefox60 + wontfix
firefox61 --- wontfix
firefox62 --- affected
firefox63 --- affected

People

(Reporter: marcia, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: crash, Whiteboard: inj+)

Crash Data

This bug was filed from the Socorro interface and is 
report bp-fe261ba9-7cca-4858-aa1c-295520170828.
=============================================================

Seen while looking at crash stats. #37 top crash in beta 6, correlated to a RelevantKnowledge dll: http://bit.ly/2wN4u63
those modules got added to our dll blocklist in bug 1277846 - apparently they found a way to work around that now...
See Also: → 1277846
Crash Signature: [@ rlls.dll@0x6dd6a] → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca]
can you help taking a look if there's something actionable in these reports?
Flags: needinfo?(ccorcoran)
the issue is occurring with a range of different signatures: https://crash-stats.mozilla.com/search/?signature=~ls.dll%400x&signature=~ls64.dll%400x
adding a few more to this bug report...
Crash Signature: [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98c] [@ …
Flags: needinfo?(ccorcoran)
Andym or David, anyone available from your team to investigate or re-do the blocklisting? It is getting to be high volume on beta, so this may turn into a worse issue on release.  Thanks.
Flags: needinfo?(ddurst)
Flags: needinfo?(amckay)
Based on brief research about this, it seems we should just re-do the blocklisting. I'm not sure how this is being loaded or why the blocklisting in bug 1277846 isn't holding though.
Flags: needinfo?(amckay)
(/me agrees with andym. removes NI.)
Flags: needinfo?(ddurst)
Any idea how they are circumventing the blocklist?
Flags: needinfo?(aklotz)
dmajor, can you help with the blocklisting?
Flags: needinfo?(dmajor)
This may be related to our underlying problems with dll injection described in bug 1380335.
Sounds like dmajor and aklotz can't take this bug. 

I am asking for help on the stability mailing list in case we can figure out some kind of workaround. 

Adam, can you try contacting someone at RelevantKnowledge? Thanks.
Flags: needinfo?(astevenson)
Reaching out over email to a couple paths, left a voicemail with a possible phone number for them as well.
Flags: needinfo?(astevenson)
Too busy with a11y+e10s to analyze this, sorry.
Flags: needinfo?(aklotz)
Looking at the "module" pings, we have ~20x more users on Release than on Beta with this DLL, but in percentage we have 2x less users on Release than on Beta.
Thanks Marco. So, we know to expect a crash spike for these signatures on 56 release, and we can't do much about it at the moment. We'll keep trying to contact the company. I don't consider this a release blocker for 56. 

I will add this the release notes for 56 as a known issue and link to the SUMO page on removing unwanted software (https://support.mozilla.org/kb/troubleshoot-firefox-issues-caused-malware)
Marcia or philipp. I heard back from RelevantKnowledge support. They said they are unaware of any current issues and asked if we can provide specific details of the crash to help investigate. 

I'm not sure if there's enough information in this report as is, or if a summary would be helpful to them.
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(madperson)
We could use steps to reproduce. :philipp is this something you can take a look at?
Flags: needinfo?(dmajor)
because this software is commonly referred to as malicious i'd be uneasy of trying to reproduce the crash myself or working closer with the vendor to troubleshoot it (like sharing crash dumps, even if an affected user would agree to that).
ultimately the kind of injection they are doing is no longer supported or allowed as per https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/
Flags: needinfo?(madperson)
(In reply to Adam Stevenson [:adamopenweb] from comment #14)
> Marcia or philipp. I heard back from RelevantKnowledge support. They said
> they are unaware of any current issues and asked if we can provide specific
> details of the crash to help investigate. 
> 
> I'm not sure if there's enough information in this report as is, or if a
> summary would be helpful to them.

1) This vendor is injecting a DLL into our process space. The DLL names is 'rlls.dll'.

https://forums.malwarebytes.com/topic/183518-removal-instructions-for-relevantknowledge/

2) This DLL inserts itself into call stacks associated with networking through an api hook added by this DLL.

https://crash-stats.mozilla.com/report/index/d0dc1167-2917-47e6-b414-f14990170920

3) This DLL commonly crashes when handling these api hook callbacks.

https://crash-stats.mozilla.com/search/?signature=~rlls.dll&product=Firefox&date=%3E%3D2017-09-13T16%3A48%3A00.000Z&date=%3C2017-09-20T16%3A48%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Due to this we blocked this DLL from injecting back in the Firefox 48 release, which solved the browser stability problems (bug 1277846).

The vendor apparently updated their injection method such that now the DLL is back in our process space and causing stability issues again.
Thanks :philipp and :jimm. Will update when I hear back from them.
Flags: needinfo?(mozillamarcia.knous)
No word back from this company.  Using the search from comment 3, there are around 300 crashes per week with these signatures on 56.0.   On 56.0.1 (64 bit migration) still a few crashes.  I don't think we change anything dramatically here by the migration. Wontfix for 56.
Sorry Liz for not updating this report. Last contact they said:

"We have figured out the reason for the crashes and no longer need any additional information. We are working on a fix and should have one implemented in the near future."
Priority: -- → P2
Crash Signature: RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6db39] [@ rlls.dll@0x6f8d8] [@ rlls.dll@0x6f964] [@ rlls.dll@0x6f8fc] [@ rlls.dll@0x3ad5] [@ rlls.dll@0x1df91] → RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6db39] [@ rlls.dll@0x6f8d8] [@ rlls.dll@0x6f964] [@ rlls.dll@0x6f8fc] [@ rlls.dll@0x3ad5] [@ rlls.dll@0x1df91] [@ rlls.dll@0x6f8ac] [@ rlls.dll@0x6e083]
Jim asked Adam to reach out to RelevantKnowledge again because this crash seems worse in Beta 57 than Firefox 56.0.

This crash affects all 32-bit Windows versions.
Crash Signature: RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6db39] [@ rlls.dll@0x6f8d8] [@ rlls.dll@0x6f964] [@ rlls.dll@0x6f8fc] [@ rlls.dll@0x3ad5] [@ rlls.dll@0x1df91] [@ rlls.dll@0x6f8ac] [@ rlls.dll@0x6e083] → RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6db39] [@ rlls.dll@0x6f8d8] [@ rlls.dll@0x6f964] [@ rlls.dll@0x6f8fc] [@ rlls.dll@0x3ad5] [@ rlls.dll@0x1df91] [@ rlls.dll@0x6f8ac] [@ rlls.dll@0x6e083] [@ RtlFreeHeap | HeapFree | rlls.dl…
OS: Windows 7 → Windows
rlls64.dll/pmls64.dll are the corresponding 64bit variants of this crashing module
Thanks Chris. Follow up has been sent.
They have identified what they believe will likely fix this issue and will be testing a build internally next week. If all goes well they should be able to roll out the fix worldwide by December.
David, I haven't heard anything. Sent another message, will update when I hear back.
Flags: needinfo?(astevenson)
They ran into an unexpected issue and were delayed from the planned December release. The build is finished now and they expect to release in the next couple of weeks, no later than the end of the month.
See Also: → 1429426
crashes with those malicious modules are becoming more frequent during the 59.0b cycle (4% of browser crashes on 59.0b5).
Adam, noting it's past end of January but we are still seeing these crashes on the rise. 
4% of browser crashes is very high. Can we make another stab at blocklisting?
Flags: needinfo?(astevenson)
Thanks Liz, that's not good. I'll follow up again.
Flags: needinfo?(astevenson)
Blocks: 1435797
No longer blocks: injecteject
Priority: P2 → P3
Whiteboard: inj+
From our contact:
"We completed testing the fix build and began rolling it out to our panel in segmented upgrades yesterday. We expect it to be released to 100% of the panel by the end of the month, if no issues are noted in the roll out."
Currently 3.3% of browser crashes in beta 59.
If there's a fixed version being rolled out, when can we blocklist older DLL versions?
(In reply to Ryan VanderMeulen [:RyanVM] from comment #33)
> If there's a fixed version being rolled out, when can we blocklist older DLL
> versions?

Unfortunately it sounds like our blocklisting isn't effective against this DLL.
:aklotz, could you take a look?
Flags: needinfo?(aklotz)
In every one of those crash reports I looked at, user32BeforeBlocklist was set to 1.

I couldn't query for that in the aggregate because that annotation is not whitelisted, but it looks to me like something is loading user32 early, which runs Windows hooks before our blocklist has a chance to initialize.

Once the blocklist initializes, we would catch it.
Flags: needinfo?(aklotz)
Early 59.0 release shows we are still getting a few of these crashes. It isn't a top crash currently though.
yes, so far the numbers on release are much lower than what i'd have expected from the pattern during the 59.0b cycle :-)
Crash Signature: rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] → rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] [@ mozilla::net::nsHttpConnection::OnReadSegment ]
Added [@ rlls.dll@0x21e6 ], as this is showing up in early b3/b4 crash stats.
Crash Signature: rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] [@ mozilla::net::nsHttpConnection::OnReadSegment ] → rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] [@ rlls.dll@0x21e6] [@ mozilla::net::nsHttpConnection::OnReadSegment ]
Crash Signature: rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] [@ rlls.dll@0x21e6] [@ mozilla::net::nsHttpConnection::OnReadSegment ] → rlls.dll@0x6f2d9][@ rlls.dll@0x3ad3] [@ RtlFreeHeap | rlls.dll@0x6f2d9] [@ rlls.dll@0x5ea7] [@ rlls.dll@0x5242f] [@ rlls.dll@0x21e6] [@ mozilla::net::nsHttpConnection::OnReadSegment ] [@ @0x0 | send | mozilla::net::nsHttpConnection::OnReadSegment ]
Adding a signature that was seen in 62 beta. There was also one crash in 61.
Crash Signature: [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pmls64.dll@0x9b98c] [@ … → [@ rlls.dll@0x6dd6a] [@ rlls.dll@0x6c5ca] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | rlls.dll@0x6f2d9] [@ pmls.dll@0x6dd6a] [@ pmls.dll@0x21e6] [@ rlls64.dll@0x9b98c] [@ RtlpLowFragHeapFree | RtlFreeHeap | HeapFree | pmls.dll@0x6f2d9] [@ pml…
Most of the crashes in beta 62 seem to be on one signature, https://crash-stats.mozilla.com/signature/?signature=%400x0%20%7C%20send%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnection%3A%3AOnReadSegment#summary
I'm emailing the support address for RelevantKnowledge, asking them to take a look.
Crash Signature: mozilla::net::nsHttpConnection::OnReadSegment ] → mozilla::net::nsHttpConnection::OnReadSegment ] [@ PR_Close | rlls.dll@0x400ac]
Crash Signature: mozilla::net::nsHttpConnection::OnReadSegment ] [@ PR_Close | rlls.dll@0x400ac] → mozilla::net::nsHttpConnection::OnReadSegment ] [@ PR_Close | rlls.dll@0x400ac] [@ rlls.dll | mozilla::net::nsSocketInputStream::Read] [@ rlls64.dll | `anonymous namespace'::checkHandshake] [@ pmls.dll | mozilla::net::nsSocketInputStream::Read] …
You need to log in before you can comment on or make changes to this bug.