1665351 - Flash Player plug-in/browser crashes on 32-bit builds

Status: VERIFIED FIXED

(Core Graveyard :: Plug-ins, defect, P1)

Description

[Alin Ilea, Desktop Test Engineering]

Attachment: Flash crash.mp4

Affected versions:

Nightly 82.0a1 (2020-09-15)(32-bit)

Tested on:

Windows 10 x64

Preconditions:

Have Adobe Flash Player installed.

Steps:

1. Launch Firefox Nightly (32-bit) with a new profile and go to one of the following websites:
   https://www.y8.com/games/orion_sandbox
   http://www.zombo.com/
   http://homestarrunner.com/
2. Click "Run Adobe Flash" and click Allow.

Actual result:

The Adobe Flash plugin crash or the browser crashes.

Expected result:

The plugin should work properly.

Regression Range:

This is a recent regression, here is the pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=93d8458a0c4dc91b24c34e2f446c76d80384c948&tochange=ec66ca239374d1f9fc6769d5f7dac1f794cfa439

Note: Only Nightly 82.0a1 (32-bit) is affected. 64-bit build is not affected. Sometimes the browser crashes after 2-3 seconds after allow the flash player to run.

My crash reports:

https://crash-stats.mozilla.org/report/index/7fa5f6d2-8e94-4c47-b218-678bd0200916
https://crash-stats.mozilla.org/report/index/7e93bb78-657c-47d1-8925-998190200916
https://crash-stats.mozilla.org/report/index/c41e6882-5afa-4ecd-bd61-138370200916

Severity suggestion:

S1 or S2 because its a blocker for all websites that are using the Flash Player plugin, on latest Nightly 32-bit builds.

Comment 1

[Jim Mathies [:jimm]]

Hey David, any chance that clang update triggered this? I don't see anything else related to graphics in the range.

Comment 2

[(Away)]

The regression range points to the second attempt to land the clang update, but that change had also briefly landed a few days before that. Here is a nightly with the first attempt: https://archive.mozilla.org/pub/firefox/nightly/2020/08/2020-08-28-15-31-26-mozilla-central/firefox-82.0a1.en-US.win32.zip If it doesn't crash, then it's probably not the compiler change. (Make sure the build doesn't update itself while testing.)

Comment 3

[(Away)]

Also, the machine appears to have some questionable software installed, which is injecting two DLLs with different random names on each startup: z_jeihso.dll, m_sjvjmtme.dll, q_axdnux.dll, d_wfnqlaxn.dll, p_dhllau.dll , c_holblonf.dll.

Comment 4

[Alin Ilea, Desktop Test Engineering]

Attachment: 2020-09-16_18h03_02.mp4

It crashes using the build from comment 2 as well. Here is the crash report for the crash from the screen recording using the build from comment 2.
https://crash-stats.mozilla.org/report/index/ddadde95-a7a2-4ca2-bf2f-ed3ad0200916

Comment 5

[(Away)]

Is it possible to test on a different machine or fresh VM that doesn't have this injected software?

Comment 6

[Alin Ilea, Desktop Test Engineering]

Sure, I just tested on a different machine, a laptop, and the result is the same. Here is the crash report from the laptop:
https://crash-stats.mozilla.org/report/index/82145ccc-8c76-40fc-a4e0-399fc0200916

Comment 7

[(Away)]

It's interesting that the comment 6 signature is different from comment 1 (which itself links to two different signatures).

Would you be willing to collect 4 or 5 more crashes on the laptop? It would help to see how widely scattered these crashes are.

Comment 8

[Laurence "GreenReaper" Parry]

I run Win64 and I got a repeatable Flash-linked Firefox termination, with a BEX reported, Bug 1665486. Not sure if related, but Flash usually works.

Comment 9

[Alin Ilea, Desktop Test Engineering]

(In reply to :dmajor from comment #7)

It's interesting that the comment 6 signature is different from comment 1 (which itself links to two different signatures).

Would you be willing to collect 4 or 5 more crashes on the laptop? It would help to see how widely scattered these crashes are.

Sure, here are another 4 crashes on my laptop:

https://crash-stats.mozilla.org/report/index/44300827-de44-4681-9ece-d60810200917
https://crash-stats.mozilla.org/report/index/025958cc-7b75-4bc2-b3ce-f82b70200917
https://crash-stats.mozilla.org/report/index/520c23b0-c60f-48c4-a24e-b49c30200917
https://crash-stats.mozilla.org/report/index/e7355139-f8c7-4d9b-92af-f55730200917

Comment 10

[Tania Maity (:tmaity)]

Noticed this bug has no severity set yet although QA has suggested a severity. @dmajor could you please help?

Severity suggestion:
S1 or S2 because its a blocker for all websites that are using the Flash Player plugin.

Comment 11

[(Away)]

I give you my blessing to set whatever severity value you feel is appropriate. I don't often work with these numbers so you have better experience than me.

Comment 12

[(Away)]

Collecting all the reports so far:

First machine
https://crash-stats.mozilla.org/report/index/7fa5f6d2-8e94-4c47-b218-678bd0200916 -- [@ mozilla::layers::TextureClient::CreateForDrawing ]
https://crash-stats.mozilla.org/report/index/7e93bb78-657c-47d1-8925-998190200916 -- [@ mozilla::UniquePtr<T>::reset ]
https://crash-stats.mozilla.org/report/index/c41e6882-5afa-4ecd-bd61-138370200916 -- [@ mozilla::UniquePtr<T>::reset ]

First machine, second try
https://crash-stats.mozilla.org/report/index/ddadde95-a7a2-4ca2-bf2f-ed3ad0200916 -- [@ RetainedDisplayList::DeleteAll ]

Laptop
https://crash-stats.mozilla.org/report/index/82145ccc-8c76-40fc-a4e0-399fc0200916 -- [@ hsw::convolve_vertically ]

Laptop, second try
https://crash-stats.mozilla.org/report/index/44300827-de44-4681-9ece-d60810200917 -- [@ mozilla::UniquePtr<T>::reset ]
https://crash-stats.mozilla.org/report/index/025958cc-7b75-4bc2-b3ce-f82b70200917 -- [@ mozilla::UniquePtr<T>::reset ]
https://crash-stats.mozilla.org/report/index/520c23b0-c60f-48c4-a24e-b49c30200917 -- [@ mozilla::UniquePtr<T>::reset ]
https://crash-stats.mozilla.org/report/index/e7355139-f8c7-4d9b-92af-f55730200917 -- [@ mozilla::UniquePtr<T>::reset ]

This is worrying. Although UniquePtr::reset is a common one, the fact that there are several others makes me suspect that this won't be straightforward to track down. :-(

Comment 13

[(Away)]

Ok, better news: I think the crash signatures are a red herring and the underlying problem is that mozilla::plugins::FunctionHook needs to disable CFG in 32-bit builds for the same reason as bug 1598119.

I am trying out a fix locally.

Comment 14

[(Away)]

Attachment: Disable CFG for plugin hook stub calls

As we saw in bug 1598119, 32-bit nop-space patches aren't compatible with clang 11's CFG because they return to the second instruction of the hooked function.

The FunctionHooks for plugins were pulling raw function pointers out of the interceptor stubs, so they didn't get the benefit of the stub's operator() that already has the CFG annotation.

As a bandaid, this patch marks all users of BasicFunctionHook::OriginalFunction() with the CFG annotation as well. A more thorough fix might be to somehow pass through to the stub's operator(), but we need something before merge day and I'm not confident in being able to do that regression-free in time.

Comment 15

[(Away)]

Alin, could you please check that https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DxPVy1XQSpyZnD09ECCGJw/runs/0/artifacts/public/build/target.zip still crashes, and https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ED46YP2DQ9-R5cLqVdcEqQ/runs/0/artifacts/public/build/target.zip is fixed? Thanks!

Comment 16

[Alin Ilea, Desktop Test Engineering]

Indeed, using the first link still crashes, here is the crash report:
https://crash-stats.mozilla.org/report/index/fd39f1e6-5b68-40ec-baa2-811cd0200918
Using the second link it work as expected, no crash, the flash player plugin works ok.
Thank you.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c8401590bab
Disable CFG for plugin hook stub calls r=handyman

Comment 18

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/9c8401590bab

Comment 19

[Alin Ilea, Desktop Test Engineering]

Verified - Fixed in latest Nightly 82.0a1 (32-bit) (build id: 20200920213416). The flash player plugin works accordingly, no crash occurs.