1335080 - RoboSizer software causes startup crashes on Windows 10

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[[:philipp]]

this a spin-off of bug 1322554 where we are tracking an increase in startup crashes with the signature [@ BaseThreadInitThunk ], which are probably malware related.

through contact with one of the crashing users, we learned that installing RoboSizer (http://www.robosizer.com/download.html) leads to consistent startup crashes on windows 10 which are reproducible. the recent spike in crashes with this signature will be caused by some different external factor though.

(In reply to Marco Castelluccio [:marco] from comment #38)
> I've contacted one of the users who experienced the crash. He said he
> downloaded "Robosizer" from http://www.robosizer.com/download.html, which
> redirected to http://www.snapfiles.com/get/robosizer.html, which redirected
> to http://www.snapfiles.com/php/sfdwnld.php?id=112684&a=7144899&loc=2.
> 
> However, he was able to simply remove it and, after that, Firefox started
> working again.

(In reply to Marco Castelluccio [:marco] from comment #39)
> He also said: "I wonder if Robosizer works in Windows 10 machines. I have
> another desktop pc running Win7, and Robosizer works as it should, without
> causing problems.".

(In reply to [:philipp] from comment #40)
> Created attachment 8831726 [details]
> RoboSizer_Setup.zip
> 
> i can corroborate the observations of the user - installing the robosizer
> tool (see .zip) on win10 caused crashing or silent failures when trying to
> launch firefox or thunderbird, whereas on windows 7 it caused no apparent
> issues.

Comment 1

[Marco Castelluccio [:marco]]

From bug 1422554 comment 44:

(In reply to Ting-Yu Chou [:ting] from comment #44)
> Robosizer has NtCreateThreadEx() called with invalid function pointer with
> stack:
> 
> : kd> !wow64exts.k
> Walking Native Stack... 
>  # Child-SP          RetAddr           Call Site
> 00 ffff8701`eceb32e8 fffff801`f90c2fab nt!PspCreateThread
> 01 ffff8701`eceb32f0 fffff801`f8d5e393 nt!NtCreateThreadEx+0x1ef
> 02 ffff8701`eceb3a90 00007fff`76e17784 nt!KiSystemServiceCopyEnd+0x13
> 03 00000000`03a9e848 00000000`50061790 ntdll!NtCreateThreadEx+0x14
> 04 00000000`03a9e850 00000000`500610de wow64!Wow64NtCreateThreadEx+0xc8
> 05 00000000`03a9e8d0 00000000`50056ea5 wow64!whNtCreateThreadEx+0x41e
> 06 00000000`03a9eab0 00000000`50131cf7 wow64!Wow64SystemServiceEx+0x155
> 07 00000000`03a9f370 00000000`5006bfa1 wow64cpu!ServiceNoTurbo+0xb
> 08 00000000`03a9f420 00000000`5005cbb0 wow64!RunCpuSimulation+0xf311
> 09 00000000`03a9f4a0 00007fff`76dea10d wow64!Wow64LdrpInitialize+0x120
> 0a 00000000`03a9f750 00007fff`76de9fae ntdll!_LdrpInitialize+0x109
> 0b 00000000`03a9f7d0 00000000`00000000 ntdll!LdrInitializeThunk+0xe
> Walking Guest (WoW) Stack... 
>  # ChildEBP          RetAddr           
> 00 03b9f134 10008c3d ntdll_776c0000!NtCreateThreadEx+0xc
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 01 03b9f82c 76f49fb3 EasyHook32!RhIsX64Process+0x23d
> 02 (Inline) -------- combase!CMessageCall::CleanupMonitorLock+0x5
> [d:\rs1\onecore\com\combase\dcomrem\call.cxx @ 578]
> 03 03b9fc60 00000000 combase!CMessageCall::~CMessageCall+0x43
> [d:\rs1\onecore\com\combase\dcomrem\call.cxx @ 522]
> 
> I am not sure how does Robosizer use EasyHook, but after I replace
> "C:\Program Files (x86)\RoboSizer\EasyHook32.dll" with the latest version
> (2.7.6035.0), Firefox *does not* crash.

Perhaps we should blocklist old versions of EasyHook.

Comment 2

[Ting-Yu Chou [:ting] (away)]

(In reply to Marco Castelluccio [:marco] from comment #1)
> From bug 1422554 comment 44:
> 
> (In reply to Ting-Yu Chou [:ting] from comment #44)
> > Robosizer has NtCreateThreadEx() called with invalid function pointer with

Sorry, but I was wrong. The function pointer is actually valid. The true story is:

The function pointer for NtCreateThreadEx() points to the EasyHook injected code Injection_ASM_x86() in Firefox process:

  public Injection_ASM_x86@0
  Injection_ASM_x86@0 PROC
  ; no registers to save, because this is the thread main function
  ; save first param (address of hook injection information)

  	mov esi, dword ptr [esp + 4]
  	
  ; call LoadLibraryW(Inject->EasyHookPath);
  	push dword ptr [esi + 8]
  	
  	call dword ptr [esi + 40] ; LoadLibraryW@4
  	mov ebp, eax
  	test eax, eax
  	je HookInject_FAILURE_A
  	
  ; call GetProcAddress(eax, Inject->EasyHookEntry);
  	push dword ptr [esi + 24]
  	push ebp
  	call dword ptr [esi + 56] ; GetProcAddress@8  // [1]
  	test eax, eax
  	je HookInject_FAILURE_B

The crash happens when the code tries to call GetProcAddress() [1] by the function pointer it retrieves earlier in RhInjectLibrary(), because the function pointer is actually AcLayers!NS_Armadillo::APIHook_GetProcAddress(), which is not existed in Firefox process as AcLayers.dll is not loaded.

EasyHook has fixed this in v2.7.5292.0 [2]:

  EasyHook crashing targets, RhInjectLibrary on Win7 64bit, 32bit process, 32bit injection dll + launcher application = problem - now determines the address of GetProcAddress etc by reading the remote process’ PE image

> Perhaps we should blocklist old versions of EasyHook.

I am not sure how could we do this as EasyHook is loaded in Robosizer not Firefox.

[1] https://github.com/EasyHook/EasyHook/blob/master/DriverShared/ASM/HookSpecific_x86.asm#L219
[2] http://easyhook.github.io/downloads.html

Comment 3

[Ting-Yu Chou [:ting] (away)]

(In reply to Ting-Yu Chou [:ting] from comment #2)
> The crash happens when the code tries to call GetProcAddress() [1] by the
> function pointer it retrieves earlier in RhInjectLibrary(), because the

Just to be clearer, RhInjectLibrary() runs in the robosizer process.

Comment 4

[Marco Castelluccio [:marco]]

(In reply to Ting-Yu Chou [:ting] from comment #2)
> > Perhaps we should blocklist old versions of EasyHook.
> 
> I am not sure how could we do this as EasyHook is loaded in Robosizer not
> Firefox.

It is injected in Firefox though, see for example https://crash-stats.mozilla.com/report/index/4a606a82-d6d9-4275-bbec-6f8352170130#tab-modules.
Isn't our blocklist able to deal with this case?

Comment 5

[Ting-Yu Chou [:ting] (away)]

You're right, we should be able to block that. Just note the EasyHook32.dll that Robosizer uses seems does not have a version specified.

Comment 6

[Marco Castelluccio [:marco]]

(In reply to Ting-Yu Chou [:ting] from comment #5)
> You're right, we should be able to block that. Just note the EasyHook32.dll
> that Robosizer uses seems does not have a version specified.

Yeah, that's a pain :(
Perhaps we can block based on the debug ID.

Comment 7

[Ting-Yu Chou [:ting] (away)]

Is there a way to understand how many of the crash does include EasyHook32.dll in the modules?

Comment 8

[Marco Castelluccio [:marco]]

(In reply to Ting-Yu Chou [:ting] from comment #7)
> Is there a way to understand how many of the crash does include
> EasyHook32.dll in the modules?

Just 33 in the past 7 days: https://gist.github.com/marco-c/ab2ee4eef3b96292c0ad141f7bae1811.

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Marco, can you look again? This crash signature still seems to be on the rise in beta 53. Might be worth blocking EasyHook.

Comment 10

[Marco Castelluccio [:marco]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #9)
> Marco, can you look again? This crash signature still seems to be on the
> rise in beta 53. Might be worth blocking EasyHook.

During the past week, 49 crashes with the EasyHook DLL.

Comment 11

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 12

[[:philipp]]

i have verified that this particular reproducible variant of the [@ BaseThreadInitThunk ] crash got successfully mitigated with the fix that landed in bug 1322554.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Bug 1322554 was backed out from Trunk already and is soon to be backed out from Aurora as well.

Comment 14

[[:philipp]]

when i revisit it, this is fixed for 32bit with the relanding of Bug 1322554.
for win64 there's a followup in bug 1369188.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Pretty unlikely we're going to invest any time in trying to fix this on ESR52 still.