1328838 - Crashes due to AMD OpenCL when working with pictures

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect, P3)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-7db0e6f7-ead4-43e1-a2f7-127932161128.
=============================================================

there is a number of crash reports with signatures containing references to the amdocl*.dll module - they are spread out among various signatures (attaching the most common ones to the bug report).
these crashes mainly affect windows 10, windows 7 and windows 8.1 and have been going on for a while - they are happening on 32bit and 64bit builds.

many comments of affected users indicate that they where somehow interacting with pictures (downloading, uploading, copying...): http://bit.ly/2hVuiUS

some of those reports like https://crash-stats.mozilla.com/report/index/b6d53bcd-66a9-4017-a57e-1c4f52170102 have the amd related module amf-wic-jpeg-decoder64.dll in the stack, so it might also be related...

The ids of the most affected amd graphics adapters are:
1 	0x9874 	536 	28.66 %
2 	0x6611 	191 	10.21 %
3 	0x9851 	159 	8.50 %
4 	0x675f 	107 	5.72 %
5 	0x9830 	75 	4.01 %

Comment 1

[Paul Blinzer]

The drivers where this issue occurs are long outdated, any driver released since mid-last year should address the problem. The root cause is in the amf-wic-jpeg-decoder*.dll. Recommended update is to the Crimson ReLive Edition 16.12.x or newer.

Comment 2

[Marco Castelluccio [:marco]]

Is this something we can work around? (other than suggesting users to update)

Comment 3

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 4

[Jim Mathies [:jimm]]

blocklist candidate. every version of this has issues, we should just eject all of them.

Comment 6

[Marco Castelluccio [:marco]]

(In reply to kyle.plumadore from bug 1403741 comment 4)
> This looks like a corrupt OpenCL installation on the system. This hardware
> doesn't support the amd vp9 decoder, but the amdopencl driver will still be
> loaded to query the hardware name. It appears as if this basic opencl query
> is crashing in the amd opencl driver, likely due to some mismatch of opencl
> components.
> 
> My guess would be that this happened due to the user doing something
> "abnormal" e.g. partly installing amd drivers, or installing an old opencl
> driver on top of a newer driver. We've seen similar crashes when users
> manually install the AMD APP development sdk, which comes with it's own
> version of amdocl64.dll. (We actually shipped a workaround for this case in
> a more recent driver version)
> 
> I don't suppose we have access to a failing system to get more info? I would
> be interested to know whether doing something as simple as opening a command
> prompt and typing "clinfo" would also crash. It would also be interesting to
> know the full path of the amdopencl64.dll module that's loaded into the
> firefox process.

(In reply to Jim Mathies [:jimm] from comment #4)
> blocklist candidate. every version of this has issues, we should just eject
> all of them.

I'm afraid if we blocked these DLLs regardless of version we could prevent VP9 decoding acceleration from working for some users.

Comment 7

[Marco Castelluccio [:marco]]

Low crash volume, wontfix for 57.

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

Very low volume crash. Wontfix for 58.

Comment 9

[Gabriele Svelto [:gsvelto]]

Cleaning up the signatures as many report no crashes anymore. This is still happening and it's still strongly correlated with AMD graphics hardware. I couldn't find crashes with different graphics vendors under these signatures.

Comment 10

[Gabriele Svelto [:gsvelto]]

The correlation was a little too strong: this isn't an issue with the graphics driver, it's a CPU bug. The vast majority of the crashes comes from AMD Family 22 processors (i.e. Jaguar and friends) which have been a never-ending source of CPU-related issues. The rest of the crashes are coming from AMD Family 21 processors (Bulldozer derivatives) and those have also popped up from time to time. Long story short: don't try to fix this bug, it's not our fault.

Comment 11

[Paul Blinzer]

(In reply to Gabriele Svelto [:gsvelto] from comment #10)

The correlation was a little too strong: this isn't an issue with the graphics driver, it's a CPU bug. The vast majority of the crashes comes from AMD Family 22 processors (i.e. Jaguar and friends) which have been a never-ending source of CPU-related issues. The rest of the crashes are coming from AMD Family 21 processors (Bulldozer derivatives) and those have also popped up from time to time. Long story short: don't try to fix this bug, it's not our fault.

With this pattern, it is likely a GPU driver related problem. The affected systems mentioned typically are matched with integrated or discrete AMD GPUs that are supported by the (now legacy for years) driver with the aforementioned OpenCL image processing. The last released driver for these products has that removed and is the recommended approach to address the issue as mentioned earlier in the bug list. I suspect the reports are from systems that have not updated to the last supported driver for these legacy products. I'd expect if affected systems are upgraded then the issue is addressed.

Comment 12

[Gabriele Svelto [:gsvelto]]

I thought about that given the modules on the stacks but there's two things that don't match the idea of a driver bug:

• We have crashes with very recent driver versions such as this one which is for version 21.5.2 released in 2021
• We don't have crashes from users with a combination of newer processors and older graphics cards

If it were purely a driver issue we'd see the crashes clump to a subset of driver versions and not include recent ones. Additionally we'd see more combinations of hardware being affected, not just two specific CPU families.

Comment 13

[Greg Stoll :gstoll]

This crash still has a decent volume, and the user comments on the crashes are pretty angry...

:gsvelto, do you think if we blocked this DLL just for AMD Family 21 or 22 this would fix these crashes? Or would it just crash somewhere else? (note that I don't know how possible this is to do, just want to explore some options)

Note for the future - this would also be a candidate for user notification of crashes and directing them to a SUMO page, although unless we have a workaround this won't be very helpful...

Comment 14

[Gabriele Svelto [:gsvelto]]

Many comments mention that this crash happens when attaching a file... they don't say it explicitly but I suppose that this means that they happen within the file picker. That would explain why we crash on a thread that doesn't appear to be ours, maybe the Windows shell is spawning a thread on our behalf to create a preview of the picture or something like that and invoking this DLL in the process? If that's the case then I suppose that blocking it shouldn't affect users, but we cannot be certain unless we try.

Comment 15

[Gabriele Svelto [:gsvelto]]

This comment suggest that we can probably safely block the DLL and users will get a black preview - which we don't care about. The solution is to upgrade the driver so yeah, this would be a great candidate for a "smart" crash reporter to inform the user about the fix. Anyway, I agree we should block it (both the 64- and 32-bit versions).

Comment 16

[Greg Stoll :gstoll]

Windows DLLBlocklist request form

1. How were we aware of the problem?
   Crash reports

2. What is a suspicious product causing the problem?
   AMD JPEG decoder

3. Is the product downloadable? If so, do we have a local repro?
   It requires a few specific AMD CPUs, so I haven't tried reproing

4. Which OS versions does the problem occur on?
   Happens on Windows 10.

5. Which process types does the problem occur on?
   Just in the parent process. Although with the out-of-process file picker turned on, we may need to come back and block this in that process as well.

6. What is the maximum version of the module in the crash reports?
   1.1.0.0 for the 64-bit version; however based on AMD comments we should block all versions of these.

7. Is the issue fixed by a newer version of the product?
   No

8. Do we have data about the module in the third-party-module ping?
   We don't seem to.

9. Do we know how the module is loaded?
   No.

10. Describe your conclusion.
    We should block all versions of these two DLLs in the parent process.

Comment 17

[Greg Stoll :gstoll]

Attachment: Bug 1328838 - block old AMD JPEG decoder DLLs r=gsvelto

Comment 18

[Molly Howell (she/her) (mostly inactive)]

An idle suggestion: in lieu of a "smart" crash reporter, might we want to add a release note telling people to update their driver if they see these blank image previews?

Comment 19

[Greg Stoll :gstoll]

Yes, that's a great idea, thanks!

Release Note Request (optional, but appreciated)
[Why is this notable]: It may cause image previews in the file dialog to show up black.
[Affects Firefox for Android]: No
[Suggested wording]: "Some machines with older AMD CPUs may see image thumbnails incorrectly render as all black in file dialogs. If this is the case, updating the graphics driver should address this issue."
[Links (documentation, blog post, etc)]:

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/037b4a8e4edc
block old AMD JPEG decoder DLLs r=gsvelto

Comment 21

[Cristian Tuns]

Backed out for causing build bustages in WindowsDllBlocklistA11yDefs.h.stub

• Backout link
• Push with failures
• Failure Log
• Failure line: NameError: name 'DllBLocklistEntry' is not defined

Comment 22

[Greg Stoll :gstoll]

sorry, typo :-/

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f5b9bd13761f
block old AMD JPEG decoder DLLs r=gsvelto

Comment 24

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/f5b9bd13761f

Comment 25

[Donal Meehan [:dmeehan]]

Thanks, added the relnote as a known issue to the Fx122 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx122 release notes