1587421 - OpenH264 Plugin stops working after Update to Catalina (Mac OS 10.15)

Status: RESOLVED FIXED

(Core :: Audio/Video: GMP, defect, P1)

Description

[martin-ramm]

Attachment: mac-ff-open264.png

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

• Firefox is installed on a Mac device running Mac OS 10.14
• Confirm that the OpenH264 Video Codec V 1.8.1 plugin is installed
• Update the OS to 10.15
• Go to a WebRTC meeting, and try to share your webcam or screen

Actual results:

• I can see the stream locally.
• The stream is not sent, i.e. the counterpart in the meeting won't receive the stream
• The attached window opened. It reads, translated from German:
  "libgmpopenh264.dylib" can't be opened, because the developer can't be verified.
  macOS cannot verify that this App is malware free.
  Firefox downloaded this file on the 3rd June 2019.

Expected results:

• No error message should be displayed when sending a stream
• My counterpart in the meeting should receive the stream

Workaround that worked:

• Close FF
• Remove the 1.8.1 folder, where the extension is located
• Open FF again and force it to re-download the extension.

Comment 1

[martin-ramm]

The User Agent of the effected FF system is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:69.0) Gecko/20100101 Firefox/69.0

Comment 2

[Marcia Knous [:marcia]]

Martin - thanks for filing - I wondered what you are using for your WebRTC meeting - this will help us try to reproduce the issue. Thanks.

Comment 3

[[:philipp]]

could this basically be bug 1558924 for the cisco plugin?

Comment 4

[Nils Ohlmeier [:drno]]

I can repro with 71 on Catalina.
Yes this only affects WebRTC calls which require H264.

Comment 5

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Based on the comment 0 it sounds like this is due to us not having cleared the quarantine attribute. However, if we redownload the plugin, then it works (i.e. the attribute is cleared). My suspicion is that if we downloaded the plugin in an older version of Firefox that was not clearing the attribute then we'll get into this situation.

Suggested mitigation: we ship a dummy update to Firefox versions that have the clear attribute code. The update points to the same plugin files on the remote server, but that will force another download which should clear the quarantine attribute.

Comment 6

[[:philipp]]

i assume balrog would be able to gate such a gmp-update to versions containing the patch from bug 1566523 (firefox 69.0+, 60.9.0esr and 68.1.0esr+ on macos)?

Comment 7

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

(In reply to [:philipp] from comment #6)

i assume balrog would be able to gate such a gmp-update to versions containing the patch from bug 1566523 (firefox 69.0+, 60.9.0esr and 68.1.0esr+ on macos)?

I believe so.

Comment 8

[Nils Ohlmeier [:drno]]

I can also confirm that starting with a fresh profile in 69 and 70, which results in downloading OpenH264 freshly from the servers, works just fine. So I think that supports the theory that re-downloading the existing 1.8.1 release should fix this issue.

Comment 9

[Haik Aftandilian [:haik]]

(In reply to Bryce Seager van Dyk (:bryce) from comment #5)

Based on the comment 0 it sounds like this is due to us not having cleared the quarantine attribute. However, if we redownload the plugin, then it works (i.e. the attribute is cleared). My suspicion is that if we downloaded the plugin in an older version of Firefox that was not clearing the attribute then we'll get into this situation.

Suggested mitigation: we ship a dummy update to Firefox versions that have the clear attribute code. The update points to the same plugin files on the remote server, but that will force another download which should clear the quarantine attribute.

Clearing my needinfo. I agree with Bryce's evaluation above and testing shows that's what's happening. Bryce is addressing this on bug 1587533.

Comment 10

[Liz Henry (:lizzard) (relman/hg->git project)]

That bug led to another which led to bug 1587537. Looks like release engineering is handling that.

Comment 11

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I believe this should be mitigated in 70+ and ESR68.3+ by changes to our balrog rules. There are some changes to the fallback downloader that I'm still looking at having uplifted, but this should be broadly addressed.

Resolving. Please reopen and/or let me know if I've overlooked anything we need do further.

Comment 12

[Dan Callahan [:callahad]]

Attachment: Screen Shot 2022-05-06 at 16.42.08.png

I experienced this last week on a fresh install of macOS Monteray (12.3.1).

As expected, the dylib did have the com.apple.quarantine xattr set.

US English error message:

"libgmpopenh264.dylib" can't be opened because Apple cannot check it for malicious software.

This software needs to be updated.
Contact the developer for more information.

Firefox downloaded this file on May 3, 2022.

Comment 13

[Haik Aftandilian [:haik]]

@Dan, could you add which version of Firefox you were running?

Also, did you experience any problems with the site you were trying to use?

I experienced this last week on a fresh install of macOS Monteray (12.3.1).

So a new install that erased the original contents and then Firefox was installed?

Comment 14

[Dan Callahan [:callahad]]

Hi Haik!

could you add which version of Firefox you were running?

It would've been whatever was Beta (macOS, aarch64) on 2022-05-03... so either the last beta of 100 or the first beta of 101. More likely 100, since I don't think I had yet restarted the browser that day.

did you experience any problems with the site you were trying to use?

If I recall correctly, I believe this popped up when joining a Zoom call from the browser... but my memory is a bit fuzzy (sorry; was in a rush to get back to Doing Other Things, so I just took the screenshot, checked xattrs, cleared com.apple.quarantine, and restarted). I can't recall if it actually caused issues with the Zoom call or not... I think I reflexively restarted the browser instead of thoroughly investigating what was broken.

So a new install that erased the original contents and then Firefox was installed?

Yes: Completely new computer, completely new Firefox profile. Nothing restored from backups.

Comment 15

[:Felipe Gomes (needinfo for replies!)]

I'm also getting the same error as Dan. I can repro it here: https://meetings.dialpadbeta.com/room/initech/5rnc2znk2x , but only after joining the meeting, not while in the interstitial "Join" page. The dialog pops up about 5 times (even after you cancel).

I'm on MacOS Monterey (12.3), using an Intel Mac, and Nightly 102. The dialog says the file was downloaded at March 17, which is when I set up this machine. It was a fresh new install at the time, although there have been one or two macOS updates (12.2 -> 12.3) since then.

Comment 16

[Haik Aftandilian [:haik]]

I'll be looking into this. It sounds like a fresh install of Firefox on a new macOS installation is the trigger.

@Felipe, do you know which version of libgmpopenh264.dylib you're running into this with? The find command below should reveal the version in each profile.

$ find ~/Library/Application\ Support/Firefox/Profiles/ -name libgmpopenh264.dylib

Comment 17

[:Felipe Gomes (needinfo for replies!)]

Hi Haik,

the folder name says 1.8.1.1. The file gmpopenh264.info contains this:

Name: gmpopenh264
Description: GMP Plugin for OpenH264.
Version: 1.8.1
APIs: encode-video[h264], decode-video[h264]

Comment 18

[Haik Aftandilian [:haik]]

I haven't been able to reproduce this so far using a fresh macOS 12 VM. I am continuing to investigate.

Comment 19

[Onur Cipe]

Hi,

I experience the same issue on macOS v12.4 and Firefox v101.0 (64-bit).

Comment 20

[iwc.evan]

Can confirm I'm also seeing the same issue. Running MacOS 12.4 (Monterey)

find ~/Library/Application\ Support/Firefox/Profiles/ -name libgmpopenh264.dylib
/Users/<user>/Library/Application Support/Firefox/Profiles//4n6o78nm.default-release/gmp-gmpopenh264/1.8.1.1/libgmpopenh264.dylib

The issue occurs even after deleting the Firefox Application Support folder and re-installing Firefox.

Comment 21

[Haik Aftandilian [:haik]]

Thanks for the additional reports.

This appears to be a regression introduced by bug 1746675 late in 2021 due to a renaming copy/paste error. Bug 1746675 changed GMPExtractorWorker.js to use IOUtils.removeMacXAttr() when it should have used IOUtils.delMacXAttr() added just before in bug 1736331. Using the wrong function name doesn't result in an error because GMPExtractorWorker.js gracefully handles the function being missing because it's coded to be platform-generic only attempting to clear the quarantine attribute on macOS.

The fix will be to change GMPExtractorWorker.js code to call IOUtils.delMacXAttr() and it would be safer to use #if defined(XP_MACOSX) instead of checking for the existence of the function first. We should quickly land the rename fix and uplift it to Beta.

An additional complication is that we should add code to handle cases where the plugins have already been downloaded without clearing the attribute and now will fail to run. This could be a follow up fix.

Hence this is a new bug not caused by the same problem we hit back in 2019 when 10.15 was being released and this bug was filed. I will file a new bug to handle this problem and link it here.

---

For users hitting this problem now, one workaround would be to clear the quarantine attribute on the dylib file by running xattr -d com.apple.quarantine <filename>. Or using the following find and exec commands to remove the attribute on the two supported plugins. This has yet to be verified.

$ find ~/Library/Application\ Support/Firefox/Profiles/ -name libgmpopenh264.dylib -exec xattr -d com.apple.quarantine {} \;
$ find ~/Library/Application\ Support/Firefox/Profiles/ -name libwidevine.dylib -exec xattr -d com.apple.quarantine {} \;

Comment 22

[Haik Aftandilian [:haik]]

Bug 1773207 will cover fixing the version of this problem that started with Firefox 97 due to the rename issue on bug 1746675.