1394550 - Crash in rlls.dll@0x6dd6a

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox :: Other, defect, P3)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-fe261ba9-7cca-4858-aa1c-295520170828.
=============================================================

Seen while looking at crash stats. #37 top crash in beta 6, correlated to a RelevantKnowledge dll: http://bit.ly/2wN4u63

Comment 1

[[:philipp]]

those modules got added to our dll blocklist in bug 1277846 - apparently they found a way to work around that now...

Comment 2

[[:philipp]]

can you help taking a look if there's something actionable in these reports?

Comment 3

[[:philipp]]

the issue is occurring with a range of different signatures: https://crash-stats.mozilla.com/search/?signature=~ls.dll%400x&signature=~ls64.dll%400x
adding a few more to this bug report...

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Andym or David, anyone available from your team to investigate or re-do the blocklisting? It is getting to be high volume on beta, so this may turn into a worse issue on release.  Thanks.

Comment 5

[Andy McKay]

Based on brief research about this, it seems we should just re-do the blocklisting. I'm not sure how this is being loaded or why the blocklisting in bug 1277846 isn't holding though.

Comment 6

[David Durst [:ddurst]]

(/me agrees with andym. removes NI.)

Comment 7

[Marco Castelluccio [:marco]]

Any idea how they are circumventing the blocklist?

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

dmajor, can you help with the blocklisting?

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

This may be related to our underlying problems with dll injection described in bug 1380335.
Sounds like dmajor and aklotz can't take this bug. 

I am asking for help on the stability mailing list in case we can figure out some kind of workaround. 

Adam, can you try contacting someone at RelevantKnowledge? Thanks.

Comment 10

[Adam Stevenson [:adamopenweb]]

Reaching out over email to a couple paths, left a voicemail with a possible phone number for them as well.

Comment 11

[(No longer employed by Mozilla) Aaron Klotz]

Too busy with a11y+e10s to analyze this, sorry.

Comment 12

[Marco Castelluccio [:marco]]

Looking at the "module" pings, we have ~20x more users on Release than on Beta with this DLL, but in percentage we have 2x less users on Release than on Beta.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

Thanks Marco. So, we know to expect a crash spike for these signatures on 56 release, and we can't do much about it at the moment. We'll keep trying to contact the company. I don't consider this a release blocker for 56. 

I will add this the release notes for 56 as a known issue and link to the SUMO page on removing unwanted software (https://support.mozilla.org/kb/troubleshoot-firefox-issues-caused-malware)

Comment 14

[Adam Stevenson [:adamopenweb]]

Marcia or philipp. I heard back from RelevantKnowledge support. They said they are unaware of any current issues and asked if we can provide specific details of the crash to help investigate. 

I'm not sure if there's enough information in this report as is, or if a summary would be helpful to them.

Comment 15

[Selena Deckelmann :selenamarie :selena]

We could use steps to reproduce. :philipp is this something you can take a look at?

Comment 16

[[:philipp]]

because this software is commonly referred to as malicious i'd be uneasy of trying to reproduce the crash myself or working closer with the vendor to troubleshoot it (like sharing crash dumps, even if an affected user would agree to that).
ultimately the kind of injection they are doing is no longer supported or allowed as per https://blog.mozilla.org/addons/2017/01/24/preventing-add-ons-third-party-software-from-loading-dlls-into-firefox/

Comment 17

[Jim Mathies [:jimm]]

(In reply to Adam Stevenson [:adamopenweb] from comment #14)
> Marcia or philipp. I heard back from RelevantKnowledge support. They said
> they are unaware of any current issues and asked if we can provide specific
> details of the crash to help investigate. 
> 
> I'm not sure if there's enough information in this report as is, or if a
> summary would be helpful to them.

1) This vendor is injecting a DLL into our process space. The DLL names is 'rlls.dll'.

https://forums.malwarebytes.com/topic/183518-removal-instructions-for-relevantknowledge/

2) This DLL inserts itself into call stacks associated with networking through an api hook added by this DLL.

https://crash-stats.mozilla.com/report/index/d0dc1167-2917-47e6-b414-f14990170920

3) This DLL commonly crashes when handling these api hook callbacks.

https://crash-stats.mozilla.com/search/?signature=~rlls.dll&product=Firefox&date=%3E%3D2017-09-13T16%3A48%3A00.000Z&date=%3C2017-09-20T16%3A48%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Due to this we blocked this DLL from injecting back in the Firefox 48 release, which solved the browser stability problems (bug 1277846).

The vendor apparently updated their injection method such that now the DLL is back in our process space and causing stability issues again.

Comment 18

[Adam Stevenson [:adamopenweb]]

Thanks :philipp and :jimm. Will update when I hear back from them.

Comment 19

[Liz Henry (:lizzard) (relman/hg->git project)]

No word back from this company.  Using the search from comment 3, there are around 300 crashes per week with these signatures on 56.0.   On 56.0.1 (64 bit migration) still a few crashes.  I don't think we change anything dramatically here by the migration. Wontfix for 56.

Comment 20

[Adam Stevenson [:adamopenweb]]

Sorry Liz for not updating this report. Last contact they said:

"We have figured out the reason for the crashes and no longer need any additional information. We are working on a fix and should have one implemented in the near future."

Comment 21

[Chris Peterson [:cpeterson]]

Jim asked Adam to reach out to RelevantKnowledge again because this crash seems worse in Beta 57 than Firefox 56.0.

This crash affects all 32-bit Windows versions.

Comment 22

[[:philipp]]

rlls64.dll/pmls64.dll are the corresponding 64bit variants of this crashing module

Comment 23

[Adam Stevenson [:adamopenweb]]

Thanks Chris. Follow up has been sent.

Comment 24

[Adam Stevenson [:adamopenweb]]

They have identified what they believe will likely fix this issue and will be testing a build internally next week. If all goes well they should be able to roll out the fix worldwide by December.

Comment 25

[David Bolter [:davidb] (NeedInfo me for attention)]

Adam any confirmation on that rollout?

(Some crashes in nightly https://crash-stats.mozilla.com/signature/?date=%3C2018-01-08T18%3A47%3A52%2B00%3A00&date=%3E%3D2018-01-01T18%3A47%3A52%2B00%3A00&product=Firefox&version=59.0a1&signature=rlls64.dll%400x114683)

Comment 26

[Adam Stevenson [:adamopenweb]]

David, I haven't heard anything. Sent another message, will update when I hear back.

Comment 27

[Adam Stevenson [:adamopenweb]]

They ran into an unexpected issue and were delayed from the planned December release. The build is finished now and they expect to release in the next couple of weeks, no later than the end of the month.

Comment 28

[[:philipp]]

crashes with those malicious modules are becoming more frequent during the 59.0b cycle (4% of browser crashes on 59.0b5).

Comment 29

[Liz Henry (:lizzard) (relman/hg->git project)]

Adam, noting it's past end of January but we are still seeing these crashes on the rise. 
4% of browser crashes is very high. Can we make another stab at blocklisting?

Comment 30

[Adam Stevenson [:adamopenweb]]

Thanks Liz, that's not good. I'll follow up again.

Comment 31

[Adam Stevenson [:adamopenweb]]

From our contact:
"We completed testing the fix build and began rolling it out to our panel in segmented upgrades yesterday. We expect it to be released to 100% of the panel by the end of the month, if no issues are noted in the roll out."

Comment 32

[Liz Henry (:lizzard) (relman/hg->git project)]

Currently 3.3% of browser crashes in beta 59.

Comment 33

[Ryan VanderMeulen [:RyanVM]]

If there's a fixed version being rolled out, when can we blocklist older DLL versions?

Comment 34

[Marco Castelluccio [:marco]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #33)
> If there's a fixed version being rolled out, when can we blocklist older DLL
> versions?

Unfortunately it sounds like our blocklisting isn't effective against this DLL.
:aklotz, could you take a look?

Comment 35

[(No longer employed by Mozilla) Aaron Klotz]

In every one of those crash reports I looked at, user32BeforeBlocklist was set to 1.

I couldn't query for that in the aggregate because that annotation is not whitelisted, but it looks to me like something is loading user32 early, which runs Windows hooks before our blocklist has a chance to initialize.

Once the blocklist initializes, we would catch it.

Comment 36

[Liz Henry (:lizzard) (relman/hg->git project)]

Early 59.0 release shows we are still getting a few of these crashes. It isn't a top crash currently though.

Comment 37

[[:philipp]]

yes, so far the numbers on release are much lower than what i'd have expected from the pattern during the 59.0b cycle :-)

Comment 38

[Marcia Knous [:marcia]]

Added [@ rlls.dll@0x21e6 ], as this is showing up in early b3/b4 crash stats.

Comment 39

[Marcia Knous [:marcia]]

Adding a signature that was seen in 62 beta. There was also one crash in 61.

Comment 40

[Liz Henry (:lizzard) (relman/hg->git project)]

Most of the crashes in beta 62 seem to be on one signature, https://crash-stats.mozilla.com/signature/?signature=%400x0%20%7C%20send%20%7C%20mozilla%3A%3Anet%3A%3AnsHttpConnection%3A%3AOnReadSegment#summary
I'm emailing the support address for RelevantKnowledge, asking them to take a look.

Comment 41

[Marcia Knous [:marcia]]

A few signatures still have crashes, but in such trivial volume this is not worth keeping open.