1856972 - Unable to load CPS (French Healthcare Card) PKCS#11 security module in FF119b2-4 in macOS (Intel & ARM all affected)

Status: VERIFIED FIXED

(Firefox :: Security, defect, P1)

Description

[BPER_ILEX]

Steps to reproduce:

We try to load the CPS PKCS#11 library manually :

1°) Open firefox 114 beta 4
2°) Go to Settings > Privacy & Security
3°) In the 'Security' section, click the button 'Security devices'
4° ) In the security modules dialog, click the button 'Load module'
5°) In the 'Load module' dialog, browse to location where resides the CPS PKCS#11 module (cps3_pkcs11_osx.dylib) in /usr/local/lib
6°) Validate the 'Load module' dialog

=================================

The same library is loaded without any problem in Firefox 118 release.

Actual results:

An error message is displayed saying that the module couldn't be loaded

Expected results:

The CPS module should have been loaded without any error messages
The CPS module should have been listed in the security modules dialog as a new module

Comment 1

[BPER_ILEX]

Sorry, in the steps description at step 1°), read 'firefox 119 b 4' instead of 'firefox 114 beta 4'

Comment 2

[BPER_ILEX]

(In reply to BPER_ILEX from comment #1)

Sorry, in the steps description at step 1°), read 'firefox 119 beta 4' instead of 'firefox 114 beta 4'

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Can you use https://mozilla.github.io/mozregression/ to narrow down when this stopped working? Thanks!

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(Also, presumably you're loading this module because Firefox doesn't work for you by default? (what is the value of security.osclientcerts.autoload in about:config?))

Comment 5

[Robert Relyea]

Do you have access to modutil (nss-tools). I don't know if they get built by firefox. It would be good to see what the error message is from modutil. I don't know of any NSS changes that would have affected an old PKCS #11 module.

I think this is a MAC?

If you don't have access to modutil, you can pull the NSS sources and use gmake to build NSS and the nss tools. I don't have access to a mac myself. We should try to reproduce this with modutil and then debug here the load is failing (are we failing in dlopen, or are we failing afterwards as we try to query the module).

bob

Comment 6

[BPER_ILEX]

Hello, the security.osclientcerts.autoload parameter is set to 'true' as it is a fresh install of Firefox.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox119 (beta). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned.

:dmehic, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 8

[Donal Meehan [:dmeehan]]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #3)

Can you use https://mozilla.github.io/mozregression/ to narrow down when this stopped working? Thanks!

Thanks :BPER_ILEX for the report and the additional information in Comment 6.
Would it be possible for you to run mozregression?
This will help narrow down the bug that introduced this regression.

Comment 9

[BPER_ILEX]

We tested mozregression specifiying that a new profile should be created at each Nighty build dowload
With this configuration we DON'T reproduce the problem for August/September nightlies.

Normally, the defect should have been found at a early september nightly, isn't it ?

On the other hand, if we specify the profile of the FF119b4 in mozgression, all nightlies fail for
our test of manually loading our PKCS11 module

Comment 10

[Dana Keeler (she/her) (use needinfo) [:keeler]]

What other modules do you have loaded in the profile that doesn't work?
Also, just to confirm, Firefox doesn't work for you by default without loading this module?

Comment 11

[Alvaro]

We tried two PKCS#11 modules with the same results. Unable to load the module manualy (Preferences->security devices->load module)
After a fresh Firefox 119 beta installation on a macOS without previous Firefox installation, Firefox is unable to load a PKC#11 module.

Our security device (PKCS#11) on Firefox is used by professional french healthcare in order to access securely to web services that requiring user smartcard authentication.

Contrary to BPER_ILEX statement : with or without specifing a profile in mozgression, we don't reproduce the problem with every tested nightlies.

The problem has appeared on macOS since the first 119 beta.

Installing 118beta try to load a security devices module > OK > Unload Module > About Firefox > Restart & update to 119b6 > load a security devices module > FAILED !

Is there any other way to catch the problem when trying to load a new security device without mozgression?
We lookuped at our PKCS#11 modules logs, the library is never called.

Comment 12

[Paul]

Attachment: mozregression-tests_FF119b.rtf

I have attached a mozregression log which seems to point towards BUG 1853627.

During the bisections I loaded our PKCS11 module manually via 'settings->Privacy & Security->Security Devices...->Device Manager->Load'.

Regards,
Paul.

Comment 13

[Dan Mehic]

Looks like this is set to appropriate component. If privacy/anti-tracking team can help let us know.

Comment 14

[Paul]

Dana, in reply to your comment No10:

We only load our PKCS11 module when testing. There are no other modules loaded or attempted to be loaded. We have performed a quick test using the Belgian eID module https://eid.belgium.be/fr (we can only attempt to load the module but not exploit it in any way) and that won't load either.

Some cryptographic operations can be performed in Firefox 119 by using the OS native libraries (so TokenDriver on macOS in this case via OSClientcerts) but a great many Healthcare professionals in France still need to use our PKCS11 module (as opposed to the TokenDriver) in their daily work.

Paul.

Comment 15

[Donal Meehan [:dmeehan]]

Setting Bug 1853627 as the regressor based on Comment 12

Comment 16

[:gerard-majax]

(In reply to Donal Meehan [:dmeehan] from comment #15)

Setting Bug 1853627 as the regressor based on Comment 12

I'm skeptical: bug 1853627 only acts on windows, comments on this bug are pointing to macOS

Comment 17

[:gerard-majax]

(In reply to Paul from comment #12)

Created attachment 9357448 [details]
mozregression-tests_FF119b.rtf

I have attached a mozregression log which seems to point towards BUG 1853627.

During the bisections I loaded our PKCS11 module manually via 'settings->Privacy & Security->Security Devices...->Device Manager->Load'.

Regards,
Paul.

The code from bug 1853627 does not exists on macOS, it's not really possible this is the cause of the regression

Comment 18

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to Paul from comment #14)

Some cryptographic operations can be performed in Firefox 119 by using the OS native libraries (so TokenDriver on macOS in this case via OSClientcerts) but a great many Healthcare professionals in France still need to use our PKCS11 module (as opposed to the TokenDriver) in their daily work.

What, specifically, do they use it for? If they used osclientcerts instead, would they no longer need to use your PKCS11 module?

Comment 19

[Paul]

Hi Dana,

90% of Healthcare professionals in France have to use our PKCS11 module (and cannot use osclientcerts) because other 3rd Party applications (such as patient record systems, French social security systems, etc) access the professional's Healthcare card via the PKCS11. These 3rd Party healthcare systems are written to interface with a PKCS11 module.

At the moment, as it stands, when FF119 is released all these healthcare professionals will not be able to access the systems they require in order to do their job.

Regards,

Paul.

Comment 20

[:gerard-majax]

Paul, I think there was some mistake during the mozregression session as I highlighted earlier ; can you re-do it ?

Comment 21

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to Paul from comment #19)

90% of Healthcare professionals in France have to use our PKCS11 module (and cannot use osclientcerts) because other 3rd Party applications (such as patient record systems, French social security systems, etc) access the professional's Healthcare card via the PKCS11. These 3rd Party healthcare systems are written to interface with a PKCS11 module.

Are these 3rd party applications websites? Or are they separate applications? (i.e. not Firefox, and if so, how does Firefox's behavior affect these separate applications?)

At the moment, as it stands, when FF119 is released all these healthcare professionals will not be able to access the systems they require in order to do their job.

Again, can you please be more specific? What do these healthcare professionals need Firefox to do?

Comment 22

[:gerard-majax]

I think it relates to https://www.sesam-vitale.fr/o/porte-document-web/atsam/ael/Connexion%20amelipro%20Web%20DMP.html?CryptolibCPSMacUtilisationTraces.html

Comment 23

[Haik Aftandilian [:haik]]

I'm just starting to look at this, but it is almost certainly caused by the fix for bug 1593072.

Comment 24

[Haik Aftandilian [:haik]]

Hi @BPER_ILEX, while we work on resolving this, could you provide the following information:

1. Please confirm the websites do not work without loading the module?
2. Which version of macOS are you using to test?
3. Is the module always installed in /usr/local/lib?
4. Can we download the module for our own testing? If so, please provide a link.
5. In addition to what's mentioned on comment 14, do you know of other pkcs11 modules commonly used?
6. Do the healthcare sites work with Chrome?

Sorry for so many questions and thank you!

macOS 10.15 and later has native support for smartcards, but it appears that loading pkcs11 libraries is still needed in some cases.

To address this, we may have to relax some of the restrictions (shipped in bug 1593072) for the parent process executable for now.

We'll work on getting an official build to test and verify the fix.

Comment 25

[Haik Aftandilian [:haik]]

Here's a potential fixed build for testing (not meant to be used long term - just for verification - it is Firefox.app, version 120, universal build).

This build removes one of the entitlement restrictions added in bug 1593072 that is likely to be causing the problem.

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/HCbBn5WZRh2JjXx0l4JBQw/runs/0/artifacts/public/build/target.tar.gz

Comment 26

[Alvaro]

We're going to test this version.

However, on my macOS (Ventura/ARM) i still not able to reproduce the problem with the Mozregression tool ???
By installing one by one each build versions, i finaly found the first build where the problem occurs :
https://ftp.mozilla.org/pub/firefox/nightly/2023/08/2023-08-29-21-12-24-mozilla-central/

Comment 27

[Paul]

Hi Haik,

We have tested the version that you provided and confirm that loading of our PKCS11 library works correctly. We can load our PKCS11 manually via the Settings pages and also via our extension that we load into FF.

We note that the Belgian eID module (mentioned in an earlier comment) can also be loaded correctly.

Thanks

Comment 28

[BPER_ILEX]

(In reply to Haik Aftandilian [:haik] from comment #24)

Hi @BPER_ILEX, while we work on resolving this, could you provide the following information:

1. Please confirm the websites do not work without loading the module?
2. Which version of macOS are you using to test?
3. Is the module always installed in /usr/local/lib?
4. Can we download the module for our own testing? If so, please provide a link.
5. In addition to what's mentioned on comment 14, do you know of other pkcs11 modules commonly used?
6. Do the healthcare sites work with Chrome?

Sorry for so many questions and thank you!

macOS 10.15 and later has native support for smartcards, but it appears that loading pkcs11 libraries is still needed in some cases.

To address this, we may have to relax some of the restrictions (shipped in bug 1593072) for the parent process executable for now.

We'll work on getting an official build to test and verify the fix.

1°) Yes we confirm that the sites that require SSL client authentication don't work without loading
our PKCS#11 module

2°) We are currently testing on :

• ARM & Intel Ventura
• ARM & Intel Sonoma
• Intel Big Sur

3°) Yes it's always installed in /usr/local/lib

4°) Yes, you can download the CPS middleware installer (.dmg) here :
https://diagcps.eservices.esante.gouv.fr/

Steps to obtain the DMG file from Diag CPS site:

• click outside the rectangle named "Assistance ..."
• Then click on the white smartcard image that appears
• Wait for the end of the countdown
• A dark blue rectangle appears with the installer link
• click the link

5°) the Belgian eID module https://eid.belgium.be/fr is used commonly in Belgium to access identity documents

Comment 29

[Haik Aftandilian [:haik]]

Attachment: Bug 1856972 - Unable to load PKCS#11 security modules in FF119b2-4 in macOS r?spohl

For now, add back the entitlement to allow loading of third party modules in the parent process executable to support pkcs11 modules.

Comment 30

[Donal Meehan [:dmeehan]]

Setting Bug 1593072 as the regressor based on Comment 23

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/da6072252134
Unable to load PKCS#11 security modules in FF119b2-4 in macOS r=spohl

Comment 32

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/da6072252134

Comment 33

[Haik Aftandilian [:haik]]

Regarding the mozregression results, unfortunately this problem is not detectable in builds using mozregression at this time. This is because mozregression re-signs downloaded builds with an ad-hoc certificate in a way that drops enforcement of Mozilla code signatures on loaded dylibs.

In more detail, it re-signs builds on the local machine as a workaround. mozregression has to modify policies.json to prevent tested builds from updating, but that breaks code signatures and prevents launch. We have some work ongoing that could address this, but not a bug for this specifically. I'll file a bug and link it up after doing some research. For posterity, this is explained on bug 1781111 comment 29.

Comment 34

[Haik Aftandilian [:haik]]

A fix (the same fix Paul tested) for this problem should be in the next available Firefox Nightly build. One can verify the presence of the fix by running $ codesign -d --entitlements - --xml /Applications/Firefox.app/ and checking the output includes disable-library-validation (replacing /Applications/Firefox.app with the bundle you want to check.)

I will request the fix be uplifted to 119.

Lastly, if you have a test environment and downloadable certs we could use to reproduce this problem ourselves, that would be helpful.

Thanks for reporting the problem and verifying the fix.

Comment 35

[Haik Aftandilian [:haik]]

Comment on attachment 9357873 [details]
Bug 1856972 - Unable to load PKCS#11 security modules in FF119b2-4 in macOS r?spohl

Beta/Release Uplift Approval Request

• User impact if declined:
  Users that depend on loading a pkcs11 module for accessing secure sites will not be able to do so. It has been reported that many healthcare sites in France depend on this functionality.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce:

1. Install a pkcs11 module using brew with $ brew install opensc and ensure /opt/homebrew/Cellar/opensc/0.23.0/lib/onepin-opensc-pkcs11.so has been installed.
2. Open Firefox
3. Open Firefox Settings...
4. Click the "Security Devices..." button under "Certificates" in the "Privacy and Security" section.
5. Click the "Load" button
6. Enter "/opt/homebrew/Cellar/opensc/0.23.0/lib/onepin-opensc-pkcs11.so" in the "Module filename" field and click OK
7. Ensure the module loads without error.

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
  Restores earlier behavior by allowing third party modules to be loaded in the parent process again. A change to macOS codesigning entitlements only.
• String changes made/needed:
• Is Android affected?: No

Comment 36

[Donal Meehan [:dmeehan]]

Comment on attachment 9357873 [details]
Bug 1856972 - Unable to load PKCS#11 security modules in FF119b2-4 in macOS r?spohl

Approved for 119.0b9

Comment 37

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/33cdae2bc20e

Comment 38

[Bogdan Maris, Desktop Test Engineering]

I reproduced the issue here using an old Firefox beta 119.0b2 and the steps from comment 35 and I verified that using the same steps in the latest Nightly 120.0a1 the module is successfully loaded. I tested on macOS 13 with both Intel and ARM.

Comment 39

[Bogdan Maris, Desktop Test Engineering]

Also verified as fixed using the same machines (Intel and ARM) using Firefox 119 beta from https://hg.mozilla.org/releases/mozilla-beta/rev/33cdae2bc20e.