1368433 - Can no longer connect to various microsoft domains with SEC_ERROR_OCSP_INVALID_SIGNING_CERT

Status: RESOLVED INCOMPLETE

(Web Compatibility :: Site Reports, defect)

Description

[Harald der Zweite]

Attachment: 1.JPG

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170518000419

Steps to reproduce:

Try to open site https://portal.azure.com/


Actual results:

A security page appears with error code SEC_ERROR_OCSP_INVALID_SIGNING_CERT
I cannot add an exception to allow visiting this site.


Expected results:

Either the page should open or the option to add an exception should be offered.

Comment 1

[BelFox]

Having same problem with other Microsoft portals:
- https://*.visualstudio.com
- https://*.yammer.com
- https://*.sharepoint.com

See also https://blogs.msdn.microsoft.com/vsoservice/?p=14225

Comment 2

[John Hesling [:John99] (NeedInfo me)]

Other reports on Sumo the official support site. Some of these are tagged ms-stapling
> https://support.mozilla.org/questions/all?tagged=ms-stapling&show=all 
(That link is usable whilst in Sumo Kitsune rollback)
The workaround is at your own risk to disable stapling using either the about config or settings menu as explained in some of those posts.

I will make first guess at trying to classify this bug, (Websites ?? or evangelism?) and confirm it.

I will repeat the  workaround advice here  



Further information

To clarify use the threebar button top right.
Next click on options (cog icon).
Then use Advanced 
Look for the tab or panel Certificates
Remove the tick by clicking the tick in the box in front of  OCSP

It is probably best to make sure that is back on again if you need to do anything important like banking, and of course once MS fixes the issue turn it back on permanently.  

Alternative method

cor-el said: {/questions/1161934#answer-972795}

    There seems to be something wrong on some Microsoft servers. Hopefully they fix this quickly on affected servers. This looks like a problem with OCSP stapling on the server because it works when I disable OCSP Stapling in Firefox.

        https://en.wikipedia.org/wiki/OCSP_Stapling
        https://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox/ 

    You can temporarily toggle this pref to false on the about:config page to see if disabling OCSP Stapling works for you. It is best to reset this pref via the right-click context menu to true once you are done with the this website.

        security.ssl.enable_ocsp_stapling = false 

Please see also this article explaining how to use  about:config

    Configuration Editor for Firefox https://support.mozilla.org/kb/about-config-editor-firefox

Comment 3

[[:philipp]]

also affects hotmail/bing/skype.com and other popular domains judging by recent sumo user questions

Comment 7

[Artem S. Tashkinov]

Microsoft might have fixed the issue because I can no longer reproduce it.

Comment 8

[Artem S. Tashkinov]

Nope I was wrong, https://web.skype.com is semi-functional (cannot load conversations history), while https://portal.azure.com cannot be accessed at all with the same error message.

Comment 9

[Sylvestre Ledru [:Sylvestre]]

I contacted Microsoft about this issue.

Comment 11

[Artem S. Tashkinov]

From https://blogs.msdn.microsoft.com/vsoservice/?p=14225 

The issue has been fixed:

> Update: Monday, May 29th, 2017 12:21 UTC
> 
> Team has identified the root cause as an expired OCSP certificate got cached and blocking web role calls. We are in the process of identifying a fix and applying it accordingly.
> 
>     Work Around: Use different browser than Firefox.
>     Next Update: Before Monday, May 29th, 2017 16:30 UTC
> 
> Sincerely,
> Sraddhananda

This bug report might be safely closed.

Still, it would be better if Firefox's error were a little bit easier to understand and read, or like with self-signed certificates there were an option to access the website regardless (which is for some reasons not possible with this particular error).

Comment 12

[Alice0775 White]

Same problem on win10 Firefox. bing seacrh.
url: https://www.bing.com/search?q=%E3%82%BF%E3%83%99%E3%82%B7%E3%83%A3&pc=MOZI&form=MOZSBR

But not on Edge and Chrome with the above URL.

So definitely, this is Firefox problem.

Comment 13

[John Hesling [:John99] (NeedInfo me)]

I wonder if a separate bug would be required if we are to change Firefox's behaviour, rather than just track the problem with MS sites.

Comment 15

[Jean-Yves Avenard [:jya]]

(In reply to Artem S. Tashkinov from comment #11)
> This bug report might be safely closed.
> 

still happening for me, 7hours after the alleged "fix"

Comment 16

[[:philipp]]

not a funny day on sumo today: https://support.mozilla.org/en-US/search/advanced?q=ocsp&num_voted=0&num_votes=&asked_by=&answered_by=&q_tags=&created=2&created_date=05%2F28%2F2017&updated=0&updated_date=&sortby=2&a=1&w=2

Comment 17

[Eric Rescorla (:ekr)]

Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP stapled data if it's unverifiable? Something else?

Comment 20

[Mike Hommey [:glandium]]

A side note about this is that the SSL errors could be more user friendly and maybe be more explicit about where the blame lies (i.e. rarely on Firefox, most likely on the server, or here, the OCSP server)

Comment 21

[Frederik Braun [:freddy]]

SUMO top-answer suggests disabling OCSP Stapling support. That's harmful. :(

Comment 22

[Julien Cristau [:jcristau]]

(In reply to Frederik Braun [:freddyb] from comment #21)
> SUMO top-answer suggests disabling OCSP Stapling support. That's harmful. :(

Maybe this calls for warnings that can be overridden, rather than ones where the only recourses in case of misconfigured sites are switching browsers or disabling the feature entirely.

Comment 23

[alcol]

(In reply to Artem S. Tashkinov from comment #7)
> Microsoft might have fixed the issue because I can no longer reproduce it.

still there the problem

I tried to contact [e-mail] and others but hard to keep in touch with webmaster or any tech support.

I tought was MS problem but even seen that was an old certificate so possible not MS (not able to make a debug myself)

Tried to see RSA certificate OCSP FQDN and seen was up and working (sending someting)

an override is important becouse by now or I use Internet Explorer or I'm out from microsoft sites (ones in SSL/TLS)

Comment 24

[J.C. Jones [:jcj] (he/they)]

(In reply to Eric Rescorla (:ekr) from comment #17)
> Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP
> stapled data if it's unverifiable? Something else?

According to sleevi, Chrome does not hard fail: https://bugs.chromium.org/p/chromium/issues/detail?id=727255

Comment 25

[Dana Keeler (she/her) (use needinfo) [:keeler]]

(In reply to J.C. Jones [:jcj] from comment #24)
> (In reply to Eric Rescorla (:ekr) from comment #17)
> > Keeler, do you know offhand why Chrome works? Does it just ignore the OCSP
> > stapled data if it's unverifiable? Something else?
> 
> According to sleevi, Chrome does not hard fail:
> https://bugs.chromium.org/p/chromium/issues/detail?id=727255

Thanks, J.C - I didn't actually know the details there.

Comment 26

[Eric Rescorla (:ekr)]

At this point, I think it's time to remotely flip OCSP stapling off and make it soft-fail the way Chrome does in a future release.

Tagging Selena for her opinion

Comment 27

[Laura Thomson :laura]

rhelmer, can you spin up a system add-on to flip this pref ASAP?

Comment 28

[Robert Helmer [:rhelmer]]

Attachment: one-off system add-on update to disable ocsp stapling

Created add-on, tracking down a reviewer.

Comment 29

[Robert Helmer [:rhelmer]]

Comment on attachment 8872742 [details] [review]
one-off system add-on update to disable ocsp stapling

r=felipe via IRC

Comment 30

[Robert Helmer [:rhelmer]]

Attachment: [e-mail]

Jason, can you please sign this system add-on update? It's urgent. Thanks!

Comment 31

[Robert Helmer [:rhelmer]]

Attachment: [e-mail]

Re-attaching for signing - last one had an unnecessary file. Still urgent though :) Thanks!

Comment 32

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #31)
> Created attachment 8872752 [details]
> [e-mail]
> 
> Re-attaching for signing - last one had an unnecessary file. Still urgent
> though :) Thanks!

Comment 33

[Jason Thomas [:jason]]

Attachment: [e-mail]

Please see attached.

Comment 34

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds like we should ship the system add-on for 53 as soon as possible and make sure we also make changes on nightly 55 and beta 54.    Is ESR also affected?

Comment 35

[Rehan Dalal (Inactive)]

do you want us to start rolling out to 100% on that ocsp patch, or should we get more QA and/or slow-roll it?

Comment 36

[Eric Rescorla (:ekr)]

My vote would be to start at 5-10%, but that's a relman decision

Comment 37

[Robert Helmer [:rhelmer]]

I've tested this out on 53.0.3 macOS on the release-sysaddon channel, and WFM:

* security.ssl.enable_ocsp_stapling is true initially
* after forcing background update check, OCSP add-on is present in about:support and security.ssl.enable_ocsp_stapling is false
* removing the add-on and restarting Firefox resets security.ssl.enable_ocsp_stapling to the default (true)

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Noting I can't reproduce this from 52esr or 45esr (the pref is set to true, with a clean profile).

Comment 39

[Eric Rescorla (:ekr)]

Keeler, JCJ, do we cache OCSP? I'm trying to figure out the perf impact.

Comment 40

[[:philipp]]

https://origin.col430-sec.mail.live.com/ currently still reproduces the OCSP error from my network location but according to https://portal.office.com/servicestatus MS expects all servers to be fixed within the next 5 hours (not sure if this only applies to outlook or other web properties as well though).

Comment 41

[Liz Henry (:lizzard) (relman/hg->git project)]

OK, now I see the OCSP error for the url in comment 40. 

As I understand it we aren't sure if Microsoft's rollout will fix the remaining sites showing the OCSP error, or if their fix may still leave some issues. We aren't sure which, yet.  

The site at https://portal.office.com/servicestatus says that the rollout for outlook.com may not finish until this Thursday, June 1.  We could ship this system add-on tomorrow morning (or tonight if we really want to rush it)  with a slow rollout.  Turning off stapling may cause other issues. 

It sounds like we should aim to fix this to "soft fail" in 54, 55, and for the next esr (52.2.0) since turning off the pref is not the best way to deal with this (though it may be our best option right now for 53 release).

Comment 42

[Selena Deckelmann :selenamarie :selena]

(In reply to Eric Rescorla (:ekr) from comment #26)
> At this point, I think it's time to remotely flip OCSP stapling off and make
> it soft-fail the way Chrome does in a future release.
> 
> Tagging Selena for her opinion

+1 to soft-fail

Comment 43

[Dana Keeler (she/her) (use needinfo) [:keeler]]

We cache OCSP for the duration of the running session of Firefox (or until the responses expire, obviously). So if a user restarts Firefox, they'll have to re-fetch responses again.

Comment 44

[Daniel Veditz [:dveditz]]

Filed Bug 1368868 on "What should we do in Firefox 54?" so it doesn't complicate this "quickly-disable" emergency bug.

Comment 45

[Liz Henry (:lizzard) (relman/hg->git project)]

Reports of the issue have stopped coming into SUMO and I don't see any new reports on Twitter. The sites we know were failing yesterday now look OK. So, I don't think we need to push this system add-on after all. It's good that we were prepared to, though.

Comment 46

[Rehan Dalal (Inactive)]

Should I remove it from the test channel as well then?

Comment 47

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes please.

Comment 48

[Liz Henry (:lizzard) (relman/hg->git project)]

I also don't think this needs to block 54, at this point.

Comment 49

[Liz Henry (:lizzard) (relman/hg->git project)]

Pretty sure MS has resolved it at this point.