1966632 - Certificate error on seagate.com, lego.com, etc. with SEC_ERROR_UNKNOWN_ISSUER (for Cloudflare TLS Issuing ECC CA 1)

Status: VERIFIED FIXED

(Core :: Security: PSM, defect)

Description

[Manuel Bucher [:manuel]]

Potentially correct, filing regression bug in case this is unexpected.

Root ca SSL.com TLS ECC Root CA 2022 is missing in chain.

regression bug found via mozregression: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f7793ca6811c7a77af4e6704d71a0902e2f437db&tochange=ac1b812c716e075be32f87b3ec78feec1cbd388b

Comment 1

[Manuel Bucher [:manuel]]

Attachment: about_certificate_bad.txt

Comment 2

[Manuel Bucher [:manuel]]

Attachment: about_certificate_good.txt

Comment 3

[Manuel Bucher [:manuel]]

Note that this is not failing in my own Nightly, only in clean profile moz-regression nightly for me.

Comment 4

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1957519

:beurdouche, since you are the author of the regressor, bug 1957519, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 5

[John Schanck [:jschanck]]

Here's the intermediate for the good chain: https://crt.sh/?id=8505503577, and here's the intermediate for the bad chain: https://crt.sh/?id=13467527106. Note that the bad chain was issued by AAA Certificate Services, and we removed the TLS trust bit for that CA in NSS 3.112 (Bug 1957685).

If the server presents the bad chain, then we rely on intermediate preloading to discover the good chain. I think this is a WORKSFORME but I'll let Dana make the call as to whether we need to do anything here.

Comment 6

[Dana Keeler (she/her) [:keeler]]

Firefox handles situations like this with intermediate preloading. However, since the set of intermediates is not bundled with the installer, new profiles may encounter issues like this until they download the set. We're looking into either bundling the intermediates with the installer (which would cause it to grow not insignificantly) or perhaps bundling "important" intermediates (where "important" may mean "we just removed a root and these intermediates are the ones that would paper over any servers that still assume that root is trusted").

In the meantime, Randell - I was told you may have cloudflare contacts we could reach out to and ask them to reconfigure their servers?

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

I don't, but perhaps Kershaw does

Comment 8

[Kershaw Chang [:kershaw]]

Hi Lars,

Please take a look at comment #6.
Do you know who we should contact at Cloudflare regarding this issue?

Thanks.

Comment 9

[Daniel Holbert [:dholbert]]

I'm seeing this same issue at https://www.lego.com

Loading that site in a fresh profile -- e.g. with mozregression like so...

mozregression --launch 2025-04-25 -a "https://www.lego.com"

...I get a cert error page with Error code: SEC_ERROR_UNKNOWN_ISSUER. It looks like it's the same Cloudflare issuing certificate that seagate uses, so it's the same issue discussed here, manifesting at another site.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

What's the NSS change that caused this regression?

Comment 11

[Daniel Holbert [:dholbert]]

[Tracking Requested - why for this release]: regression causing major sites to fail to load (at least, until some time has passed to let us lazily download intermediate certificates) --> bad first-run experience for new users, depending on what site(s) they try to visit after first install.

Comment 12

[John Schanck [:jschanck]]

This was caused by removal of the TLS trust bit for the "AAA Certificate Services" root certificate in Bug 1957685.

Comment 13

[John Schanck [:jschanck]]

I've gotten in touch with some people on the Cloudflare side. I think we'll need a workaround in Firefox in the short term. I'll work up a patch.

Comment 14

[John Schanck [:jschanck]]

Attachment: Bug 1966632 - bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler

Comment 15

[John Schanck [:jschanck]]

Attachment: Bug 1966632 - bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler

Original Revision: https://phabricator.services.mozilla.com/D250486

Comment 16

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: Users will see certificate errors on some popular domains until the intermediate certifcate preload list is retrieved from remote settings.
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: Create a new profile and visit any of the sites listed in Bug 1966632.
• Risk associated with taking this patch: low
• Explanation of risk level: The patch uses an existing codepath for side-loading an intermediate certificate.
• String changes made/needed: none
• Is Android affected?: yes

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f6164d98d8d
bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e7fdcbb9dc9
Revert "Bug 1966632 - bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler" for causing bc failures

Comment 19

[Sandor Molnar[:smolnar]]

Backed out for causing bc/xpc failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/5e7fdcbb9dc9bad7554b0d40be8a8ed14deed69c

Push with failures

Failure log bc

Failure log xpc

[Child 8028, Main Thread] WARNING: 'ps->NeedLayoutFlush()', file /builds/worker/checkouts/gecko/dom/base/Document.cpp:18327
[task 2025-05-22T09:37:51.012Z] 09:37:51     INFO - GECKO(5212) | ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to C:\Users\task_174790466603417\AppData\Local\Temp\tmp6fjt32tb.mozrunner\runtests_leaks_tab_pid7288.log
[task 2025-05-22T09:37:51.220Z] 09:37:51     INFO - GECKO(5212) | 8568> [Parent 8568, Main Thread] WARNING: NS_ENSURE_TRUE(inst) failed: file StaticComponents.cpp:12963
[task 2025-05-22T09:37:51.226Z] 09:37:51     INFO - GECKO(5212) | 8568> [8568] Hit MOZ_CRASH() at /builds/worker/checkouts/gecko/xpcom/build/LateWriteChecks.cpp:118
[task 2025-05-22T09:37:51.345Z] 09:37:51     INFO - GECKO(5212) | 8568> #01: LateWriteObserver::Observe(mozilla::IOInterposeObserver::Observation&) [xpcom/build/LateWriteChecks.cpp:118]
[task 2025-05-22T09:37:51.352Z] 09:37:51     INFO - GECKO(5212) | 8568> #02: mozilla::IOInterposer::Report(mozilla::IOInterposeObserver::Observation&) [xpcom/build/IOInterposer.cpp:479]
[task 2025-05-22T09:37:51.352Z] 09:37:51     INFO - GECKO(5212) | 8568> #03: mozilla::IOInterposeObserver::Observation::Report() [xpcom/build/IOInterposer.cpp:386]
[task 2025-05-22T09:37:51.353Z] 09:37:51     INFO - GECKO(5212) | 8568> #04: (anonymous namespace)::InterposedNtWriteFile(void*, void*, void (*)(void*, _IO_STATUS_BLOCK*, unsigned long), void*, _IO_STATUS_BLOCK*, void*, unsigned long, _LARGE_INTEGER*, unsigned long*) [xpcom/build/PoisonIOInterposerWin.cpp:358]
[task 2025-05-22T09:37:51.353Z] 09:37:51     INFO - GECKO(5212) | 8568> #05: std::sys::pal::windows::handle::Handle::synchronous_write() [git:github.com/rust-lang/rust:library/std/src/sys/pal/windows/handle.rs:05f9846f893b09a1be1fc8560e33fc3c815cfecb:313]
[task 2025-05-22T09:37:51.354Z] 09:37:51     INFO - GECKO(5212) | 8568> #06: std::fs::write::inner() [git:github.com/rust-lang/rust:library/std/src/fs.rs:05f9846f893b09a1be1fc8560e33fc3c815cfecb:349]
[task 2025-05-22T09:37:51.355Z] 09:37:51     INFO - GECKO(5212) | 8568> #07: rkv::backend::impl_safe::environment::EnvironmentImpl::write_to_disk() [third_party/rust/rkv/src/backend/impl_safe/environment.rs:241]
[task 2025-05-22T09:37:51.356Z] 09:37:51     INFO - GECKO(5212) | 8568> #08: rkv::backend::impl_safe::transaction::impl$4::commit(rkv::backend::impl_safe::transaction::RwTransactionImpl) [third_party/rust/rkv/src/backend/impl_safe/transaction.rs:194]
[task 2025-05-22T09:37:51.357Z] 09:37:51     INFO - GECKO(5212) | 8568> #09: cert_storage::SecurityState::add_certs_internal(ref$<slice2$<tuple$<nsstring::nsCString,nsstring::nsCString,i16> > >, bool) [security/manager/ssl/cert_storage/src/lib.rs:661]
[task 2025-05-22T09:37:51.358Z] 09:37:51     INFO - GECKO(5212) | 8568> #10: cert_storage::SecurityState::load_bundled_intermediates() [security/manager/ssl/cert_storage/src/lib.rs:703]
[task 2025-05-22T09:37:51.358Z] 09:37:51     INFO - GECKO(5212) | 8568> #11: cert_storage::SecurityState::open_db() [security/manager/ssl/cert_storage/src/lib.rs:251]
[task 2025-05-22T09:37:51.359Z] 09:37:51     INFO - GECKO(5212) | 8568> #12: cert_storage::impl$11::run(cert_storage::BackgroundReadDeltasTask*) [security/manager/ssl/cert_storage/src/lib.rs:1147]
[task 2025-05-22T09:37:51.361Z] 09:37:51     INFO - GECKO(5212) | 8568> #13: async_task::raw::RawTask<enum2$<moz_task::impl$2::dispatch_with_options::async_block_env$0>,alloc::boxed::Box<dyn$<moz_task::Task,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>,moz_task::executor::impl$5::spawn_onto::closure_env$0<enum2$<moz_task::impl$2::dispatch_with_options::async_block_env$0> > >::run<enum2$<moz_task::impl$2::dispatch_with_options::async_block_env$0>,alloc::boxed::Box<dyn$<moz_task::Task,core::marker::Send,core::marker::Sync>,alloc::alloc::Global>,moz_task::executor::impl$5::spawn_onto::closure_env$0<enum2$<moz_task::impl$2::dispatch_with_options::async_block_env$0> > >(tuple$<>*) [third_party/rust/async-task/src/raw.rs:511]
[task 2025-05-22T09:37:51.362Z] 09:37:51     INFO - GECKO(5212) | 8568> #14: moz_task::dispatcher::impl$4::allocate::Run<moz_task::executor::schedule::closure_env$1>(xpcom::interfaces::idl::nsIRunnable*) [xpcom/rust/moz_task/src/dispatcher.rs:16]
[task 2025-05-22T09:37:51.363Z] 09:37:51     INFO - GECKO(5212) | 8568> #15: mozilla::TaskQueue::Runner::Run() [xpcom/threads/TaskQueue.cpp:272]
[task 2025-05-22T09:37:51.363Z] 09:37:51     INFO - GECKO(5212) | 8568> #16: nsThreadPool::Run() [xpcom/threads/nsThreadPool.cpp:458]
[task 2025-05-22T09:37:51.364Z] 09:37:51     INFO - GECKO(5212) | 8568> #17: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1154]
[task 2025-05-22T09:37:51.364Z] 09:37:51     INFO - GECKO(5212) | 8568> #18: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:480]
[task 2025-05-22T09:37:51.367Z] 09:37:51     INFO - GECKO(5212) | 8568> #19: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:300]
[task 2025-05-22T09:37:51.367Z] 09:37:51     INFO - GECKO(5212) | 8568> #20: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:363]
[task 2025-05-22T09:37:51.367Z] 09:37:51     INFO - GECKO(5212) | 8568> #21: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:345]
[task 2025-05-22T09:37:51.367Z] 09:37:51     INFO - GECKO(5212) | 8568> #22: nsThread::ThreadFunc(void*) [xpcom/threads/nsThread.cpp:368]
[task 2025-05-22T09:37:51.638Z] 09:37:51     INFO - GECKO(5212) | 8568> #23: _PR_NativeRunThread(void*) [nsprpub/pr/src/threads/combined/pruthr.c:391]
[task 2025-05-22T09:37:51.638Z] 09:37:51     INFO - GECKO(5212) | 8568> #24: pr_root(void*) [nsprpub/pr/src/md/windows/w95thred.c:130]
[task 2025-05-22T09:37:51.642Z] 09:37:51     INFO -  fix-stacks: error: failed to read debug info file `ucrtbase.pdb` for `C:\Windows\System32\ucrtbase.dll`
[task 2025-05-22T09:37:51.642Z] 09:37:51     INFO -  fix-stacks: note: this is expected and harmless for all PDB files on opt automation runs
[task 2025-05-22T09:37:51.643Z] 09:37:51     INFO -  fix-stacks: The system cannot find the file specified. (os error 2)
[task 2025-05-22T09:37:51.643Z] 09:37:51     INFO - GECKO(5212) | 8568> #25: wcsrchr [C:\Windows\System32\ucrtbase.dll + 0x37b0]

Comment 20

[John Schanck [:jschanck]]

I heard from Cloudflare that they plan on switching over to the SSL.com TLS ECC Root CA 2022 chain shortly, so this issue should resolve itself w/o my patch. I'll still try to fix up the patch this morning.

Comment 21

[John Schanck [:jschanck]]

Attachment: Bug 1966632 - bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler

Original Revision: https://phabricator.services.mozilla.com/D250486

Comment 22

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined: Users will see certificate errors on some popular domains until the intermediate certifcate preload list is retrieved from remote settings.
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: Create a new profile and visit any of the sites listed in Bug 1966632.
• Risk associated with taking this patch: low
• Explanation of risk level: The patch modifies a build script but otherwise adds no new code.
• String changes made/needed: none
• Is Android affected?: yes

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad4cdf75dfc0
bundle cross-signed "SSL.com TLS Transit ECC CA R2" intermediate. r=keeler

Comment 24

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/268ce4ad24a2

Comment 26

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Reproduced the issue on Windows 10x64 with Firefox 140.0a1 (2025-05-15). Creating a new profile and visiting https://www.lego.com/ and https://www.seagate.com/ will show the Warning: Potential Security Risk Ahead page.
The issue is verified fixed with Firefox 139.0 RC2 (20250522210034) on Windows 10x64, macOS 12 and Ubuntu 24. After creating a new profile and visiting https://www.lego.com/ and https://www.seagate.com/ the Warning: Potential Security Risk Ahead page is no longer displayed.

Comment 27

[amarc]

https://hg.mozilla.org/mozilla-central/rev/ad4cdf75dfc0

Comment 28

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Verified fixed with Firefox 140.0a1 (2025-05-25) on Windows 10x64, macOS 12 and Ubuntu 24. After creating a new profile and visiting https://www.lego.com/ and https://www.seagate.com/ the Warning: Potential Security Risk Ahead page is no longer displayed.