Can no longer connect to various microsoft domains with SEC_ERROR_OCSP_INVALID_SIGNING_CERT

RESOLVED INCOMPLETE

Status

Tech Evangelism
Desktop
--
major
RESOLVED INCOMPLETE
6 months ago
a month ago

People

(Reporter: Harald der Zweite, Assigned: rhelmer)

Tracking

Firefox Tracking Flags

(firefox-esr52 ?, firefox53blocking fix-optional, firefox54+ fix-optional, firefox55+ fix-optional)

Details

(Whiteboard: [parity-Chrome][parity-Edge])

Attachments

(5 attachments)

(Reporter)

Description

6 months ago
Created attachment 8872281 [details]
1.JPG

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170518000419

Steps to reproduce:

Try to open site https://portal.azure.com/


Actual results:

A security page appears with error code SEC_ERROR_OCSP_INVALID_SIGNING_CERT
I cannot add an exception to allow visiting this site.


Expected results:

Either the page should open or the option to add an exception should be offered.

Comment 1

6 months ago
Having same problem with other Microsoft portals:
- https://*.visualstudio.com
- https://*.yammer.com
- https://*.sharepoint.com

See also https://blogs.msdn.microsoft.com/vsoservice/?p=14225
Other reports on Sumo the official support site. Some of these are tagged ms-stapling
> https://support.mozilla.org/questions/all?tagged=ms-stapling&show=all 
(That link is usable whilst in Sumo Kitsune rollback)
The workaround is at your own risk to disable stapling using either the about config or settings menu as explained in some of those posts.

I will make first guess at trying to classify this bug, (Websites ?? or evangelism?) and confirm it.

I will repeat the  workaround advice here  



Further information

To clarify use the threebar button top right.
Next click on options (cog icon).
Then use Advanced 
Look for the tab or panel Certificates
Remove the tick by clicking the tick in the box in front of  OCSP

It is probably best to make sure that is back on again if you need to do anything important like banking, and of course once MS fixes the issue turn it back on permanently.  

Alternative method

cor-el said: {/questions/1161934#answer-972795}

    There seems to be something wrong on some Microsoft servers. Hopefully they fix this quickly on affected servers. This looks like a problem with OCSP stapling on the server because it works when I disable OCSP Stapling in Firefox.

        https://en.wikipedia.org/wiki/OCSP_Stapling
        https://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox/ 

    You can temporarily toggle this pref to false on the about:config page to see if disabling OCSP Stapling works for you. It is best to reset this pref via the right-click context menu to true once you are done with the this website.

        security.ssl.enable_ocsp_stapling = false 

Please see also this article explaining how to use  about:config

    Configuration Editor for Firefox https://support.mozilla.org/kb/about-config-editor-firefox
Severity: normal → major
Status: UNCONFIRMED → NEW
Component: Untriaged → Other
Ever confirmed: true
Product: Firefox → Websites
Version: 53 Branch → unspecified

Comment 3

6 months ago
also affects hotmail/bing/skype.com and other popular domains judging by recent sumo user questions
Component: Other → Desktop
Product: Websites → Tech Evangelism

Updated

6 months ago
Duplicate of this bug: 1368432

Updated

6 months ago
Duplicate of this bug: 1368441

Updated

6 months ago
Duplicate of this bug: 1368450

Updated

6 months ago
Summary: Can no longer connect to https://portal.azure.com SEC_ERROR_OCSP_INVALID_SIGNING_CERT → Can no longer connect to various microsoft domains with SEC_ERROR_OCSP_INVALID_SIGNING_CERT

Comment 7

6 months ago
Microsoft might have fixed the issue because I can no longer reproduce it.

Comment 8

6 months ago
Nope I was wrong, https://web.skype.com is semi-functional (cannot load conversations history), while https://portal.azure.com cannot be accessed at all with the same error message.
I contacted Microsoft about this issue.

Updated

6 months ago
Duplicate of this bug: 1368475

Comment 11

6 months ago
From https://blogs.msdn.microsoft.com/vsoservice/?p=14225 

The issue has been fixed:

> Update: Monday, May 29th, 2017 12:21 UTC
> 
> Team has identified the root cause as an expired OCSP certificate got cached and blocking web role calls. We are in the process of identifying a fix and applying it accordingly.
> 
>     Work Around: Use different browser than Firefox.
>     Next Update: Before Monday, May 29th, 2017 16:30 UTC
> 
> Sincerely,
> Sraddhananda

This bug report might be safely closed.

Still, it would be better if Firefox's error were a little bit easier to understand and read, or like with self-signed certificates there were an option to access the website regardless (which is for some reasons not possible with this particular error).

Comment 12

6 months ago
Same problem on win10 Firefox. bing seacrh.
url: https://www.bing.com/search?q=%E3%82%BF%E3%83%99%E3%82%B7%E3%83%A3&pc=MOZI&form=MOZSBR

But not on Edge and Chrome with the above URL.

So definitely, this is Firefox problem.

Updated

6 months ago
Whiteboard: [parity-Chrome][parity-Edge]
I wonder if a separate bug would be required if we are to change Firefox's behaviour, rather than just track the problem with MS sites.

Updated

6 months ago
Duplicate of this bug: 1368565
(In reply to Artem S. Tashkinov from comment #11)
> This bug report might be safely closed.
> 

still happening for me, 7hours after the alleged "fix"

Comment 16

6 months ago
not a funny day on sumo today: https://support.mozilla.org/en-US/search/advanced?q=ocsp&num_voted=0&num_votes=&asked_by=&answered_by=&q_tags=&created=2&created_date=05%2F28%2F2017&updated=0&updated_date=&sortby=2&a=1&w=2

Comment 17

6 months ago
Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP stapled data if it's unverifiable? Something else?
Flags: needinfo?(dkeeler)

Updated

6 months ago
Duplicate of this bug: 1368538
Duplicate of this bug: 1368611
A side note about this is that the SSL errors could be more user friendly and maybe be more explicit about where the blame lies (i.e. rarely on Firefox, most likely on the server, or here, the OCSP server)
SUMO top-answer suggests disabling OCSP Stapling support. That's harmful. :(
(In reply to Frederik Braun [:freddyb] from comment #21)
> SUMO top-answer suggests disabling OCSP Stapling support. That's harmful. :(

Maybe this calls for warnings that can be overridden, rather than ones where the only recourses in case of misconfigured sites are switching browsers or disabling the feature entirely.

Comment 23

6 months ago
(In reply to Artem S. Tashkinov from comment #7)
> Microsoft might have fixed the issue because I can no longer reproduce it.

still there the problem

I tried to contact support@live.com and others but hard to keep in touch with webmaster or any tech support.

I tought was MS problem but even seen that was an old certificate so possible not MS (not able to make a debug myself)

Tried to see RSA certificate OCSP FQDN and seen was up and working (sending someting)

an override is important becouse by now or I use Internet Explorer or I'm out from microsoft sites (ones in SSL/TLS)

Comment 24

6 months ago
(In reply to Eric Rescorla (:ekr) from comment #17)
> Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP
> stapled data if it's unverifiable? Something else?

According to sleevi, Chrome does not hard fail: https://bugs.chromium.org/p/chromium/issues/detail?id=727255
(In reply to J.C. Jones [:jcj] from comment #24)
> (In reply to Eric Rescorla (:ekr) from comment #17)
> > Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP
> > stapled data if it's unverifiable? Something else?
> 
> According to sleevi, Chrome does not hard fail:
> https://bugs.chromium.org/p/chromium/issues/detail?id=727255

Thanks, J.C - I didn't actually know the details there.
Flags: needinfo?(dkeeler)

Comment 26

6 months ago
At this point, I think it's time to remotely flip OCSP stapling off and make it soft-fail the way Chrome does in a future release.

Tagging Selena for her opinion
Flags: needinfo?(sdeckelmann)

Comment 27

6 months ago
rhelmer, can you spin up a system add-on to flip this pref ASAP?
Flags: needinfo?(rhelmer)
(Assignee)

Comment 28

6 months ago
Created attachment 8872742 [details] [review]
one-off system add-on update to disable ocsp stapling

Created add-on, tracking down a reviewer.
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
Flags: needinfo?(rhelmer)
(Assignee)

Comment 29

6 months ago
Comment on attachment 8872742 [details] [review]
one-off system add-on update to disable ocsp stapling

r=felipe via IRC
Attachment #8872742 - Flags: review+
(Assignee)

Comment 30

6 months ago
Created attachment 8872750 [details]
disable-ocsp-stapling@mozilla.org-1.0-UNSIGNED.xpi

Jason, can you please sign this system add-on update? It's urgent. Thanks!
Flags: needinfo?(jthomas)
(Assignee)

Comment 31

6 months ago
Created attachment 8872752 [details]
disable-ocsp-stapling@mozilla.org-1.0-UNSIGNED.xpi

Re-attaching for signing - last one had an unnecessary file. Still urgent though :) Thanks!
Flags: needinfo?(jthomas)
(Assignee)

Comment 32

6 months ago
(In reply to Robert Helmer [:rhelmer] from comment #31)
> Created attachment 8872752 [details]
> disable-ocsp-stapling@mozilla.org-1.0-UNSIGNED.xpi
> 
> Re-attaching for signing - last one had an unnecessary file. Still urgent
> though :) Thanks!
Flags: needinfo?(jthomas)
Created attachment 8872757 [details]
disable-ocsp-stapling@mozilla.org-1.0-SIGNED.xpi

Please see attached.
Flags: needinfo?(jthomas)
Sounds like we should ship the system add-on for 53 as soon as possible and make sure we also make changes on nightly 55 and beta 54.    Is ESR also affected?
status-firefox53: --- → affected
status-firefox54: --- → affected
status-firefox55: --- → affected
status-firefox-esr52: --- → ?
tracking-firefox53: --- → blocking
tracking-firefox54: --- → blocking
tracking-firefox55: --- → +
do you want us to start rolling out to 100% on that ocsp patch, or should we get more QA and/or slow-roll it?

Comment 36

6 months ago
My vote would be to start at 5-10%, but that's a relman decision
(Assignee)

Comment 37

6 months ago
I've tested this out on 53.0.3 macOS on the release-sysaddon channel, and WFM:

* security.ssl.enable_ocsp_stapling is true initially
* after forcing background update check, OCSP add-on is present in about:support and security.ssl.enable_ocsp_stapling is false
* removing the add-on and restarting Firefox resets security.ssl.enable_ocsp_stapling to the default (true)
Noting I can't reproduce this from 52esr or 45esr (the pref is set to true, with a clean profile).

Comment 39

6 months ago
Keeler, JCJ, do we cache OCSP? I'm trying to figure out the perf impact.
Flags: needinfo?(jjones)
Flags: needinfo?(dkeeler)

Comment 40

6 months ago
https://origin.col430-sec.mail.live.com/ currently still reproduces the OCSP error from my network location but according to https://portal.office.com/servicestatus MS expects all servers to be fixed within the next 5 hours (not sure if this only applies to outlook or other web properties as well though).
OK, now I see the OCSP error for the url in comment 40. 

As I understand it we aren't sure if Microsoft's rollout will fix the remaining sites showing the OCSP error, or if their fix may still leave some issues. We aren't sure which, yet.  

The site at https://portal.office.com/servicestatus says that the rollout for outlook.com may not finish until this Thursday, June 1.  We could ship this system add-on tomorrow morning (or tonight if we really want to rush it)  with a slow rollout.  Turning off stapling may cause other issues. 

It sounds like we should aim to fix this to "soft fail" in 54, 55, and for the next esr (52.2.0) since turning off the pref is not the best way to deal with this (though it may be our best option right now for 53 release).
(In reply to Eric Rescorla (:ekr) from comment #26)
> At this point, I think it's time to remotely flip OCSP stapling off and make
> it soft-fail the way Chrome does in a future release.
> 
> Tagging Selena for her opinion

+1 to soft-fail
We cache OCSP for the duration of the running session of Firefox (or until the responses expire, obviously). So if a user restarts Firefox, they'll have to re-fetch responses again.
Flags: needinfo?(jjones)
Flags: needinfo?(dkeeler)
Flags: needinfo?(sdeckelmann)
Blocks: 1368868
Filed Bug 1368868 on "What should we do in Firefox 54?" so it doesn't complicate this "quickly-disable" emergency bug.
Reports of the issue have stopped coming into SUMO and I don't see any new reports on Twitter. The sites we know were failing yesterday now look OK. So, I don't think we need to push this system add-on after all. It's good that we were prepared to, though.
Should I remove it from the test channel as well then?
Flags: needinfo?(lhenry)
Yes please.
Flags: needinfo?(lhenry)
I also don't think this needs to block 54, at this point.
status-firefox53: affected → fix-optional
tracking-firefox54: blocking → +
(Assignee)

Updated

6 months ago
Status: ASSIGNED → RESOLVED
Last Resolved: 6 months ago
Resolution: --- → INCOMPLETE
Pretty sure MS has resolved it at this point.
status-firefox54: affected → fix-optional
status-firefox55: affected → fix-optional
See Also: → bug 1368868
You need to log in before you can comment on or make changes to this bug.