Closed Bug 508408 Opened 15 years ago Closed 15 years ago

addons.mozilla.org connection interrupted

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rdoherty, Assigned: mrz)

Details

AMO is having issues loading (connection interrupted in Firefox). Had someone in IRC say it wouldn't load for them too.

Works in Safari but not Firefox. 

Sometimes it loads phenomenally slowly.
Works in seamonkey for me, but just "connection was interrupted" in firefox.

Someone else reports it working in Fennec 1.0b1 but not the FF nightly.
And if someone can make this bug public, please do, thanks!
Group: infra
Updates and recommended API feed both appear to work fine.
Not sure what this means but in Live HTTP Headers, I'm seeing requests for 
#request# POST http://ocsp.globalsign.com/ExtendedSSLCA
#request# POST http://ocsp.globalsign.com/ExtendedSSLCACross

and ocscp.globalsign.com:80 isn't responding, so could this be the problem?
Someone also brought up some issues with globalsign and the EV certs here: http://forums.mozillazine.org/viewtopic.php?f=38&t=1399945
Seems like this is indeed the problem. When I disable OCSP in about:config, it loads fine.
about:config  --> change security.OCSP.enabled=>value from 1 to 0
GlobalSign's OSCP server is offline and Firefox is timing out and failing to
load the page.  I can verify this by disabling OSCP checks by using one of the
following methods:

1. Preferences | Advanced | Encyption | Validation
2. security.OCSP.enabled=>value = 0 FT

IT will revert to an OV cert.
Assignee: server-ops → mrz
Maybe time to choose a CA with better OCSP server uptime?
GlobalSign has no after hours support number I can find (!!).
I'm browsing on hard-fail for OCSP. It appears to time out the first time, succeeds on successive tries, but show DV UI only.
> IT will revert to an OV cert.

btw, reverted to the *.mozilla.org cert.
Ha, that's why :-)
These guys are driving me nuts.  I can't even call them right now.  I filed a support ticket through email, not sure how else to contact them.

Telephone Support:
US Call Toll Free: 1-877-GO-SSLHELP (877-467-7543)
Mon to Fri - 9am to 6pm EST.

Europe Call: +44 1622-766-766
UK Call: 01622-766-766
Mon to Fri - 9am to 6pm UK Time.
Perhaps a blog post will get their attention.
> We effectively had a DDOS through 1Million OCSP requests on one certificate
> so we are investigating the cause at this moment
Gee, I bet it was ours, too.
Resolved.  Replacement EV cert tracked in bug 503040.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
In reply to comment #15:
> We effectively had a DDOS through 1Million OCSP requests on one certificate
> so we are investigating the cause at this moment

Matthew, did GlobalSign indicate the timeframe over which those "1Million OCSP requests" were spread?
(I'd be very surprised if it was 1 million concurrent OCSP requests!  I'd believe you if you said it was over a 24hr period).

I hope you don't mind me asking.  The performance of Comodo's OCSP Responders falls largely within my remit, and I would like to reassure myself that our Responders would not struggle with this kind of load.

Thanks.
Rob, do you call one million OCSP requests PER day a DOS attack? :S
IIRC, they were only built out to handle 200k/day.
In reply to comment #19:
> Rob, do you call one million OCSP requests PER day a DOS attack? :S

Certainly not!  I wouldn't even call one *billion* OCSP requests PER day a DOS attack!

https://press.verisign.com/easyir/customrel.do?easyirid=AFC0FF0DB5C560D3&version=live&prid=518965&releasejsp=custom_97

1,000,000,000 OCSP requests per day = an average of ~11,500 per second.

In reply to comment #20:
> IIRC, they were only built out to handle 200k/day.

200,000 OCSP requests per day = an average of only ~2 per second!!

In reply to comment #8:
> Maybe time to choose a CA with better OCSP server uptime?

Perhaps it would be prudent for Mozilla to obtain EV certs for addons.mozilla.org from more than one EV CA?  Then you wouldn't have to fall back to using the *.mozilla.org DV cert if GlobalSign's OCSP Responder should ever melt again.

Comodo would be happy to oblige.

Further to comment #18:
> ...I would like to reassure myself that our Responders would not struggle
> with this kind of load.

I have reassured myself. :-)
(In reply to comment #21)
> In reply to comment #19:
> > Rob, do you call one million OCSP requests PER day a DOS attack? :S
> 
> Certainly not!  I wouldn't even call one *billion* OCSP requests PER day a DOS
> attack!
> 
> https://press.verisign.com/easyir/customrel.do?easyirid=AFC0FF0DB5C560D3&version=live&prid=518965&releasejsp=custom_97
> 
> 1,000,000,000 OCSP requests per day = an average of ~11,500 per second.

Kudos to Verisign, it's quite impressive and a big responsibility, isn't it :-)

> Comodo would be happy to oblige.

Dhuuu?!
In reply to comment #22:
> > 1,000,000,000 OCSP requests per day = an average of ~11,500 per second.
>
> Kudos to Verisign, it's quite impressive and a big responsibility, isn't it
> :-)

IMHO, every CA should treat the provisioning of revocation information as a big responsibility.  Uptime is important even for those CAs who usually experience relatively low OCSP and/or CRL traffic levels.

> > Comodo would be happy to oblige.
>
> Dhuuu?!

Eddy, if you re-read my preceding paragraph in comment #21 I think you'll then understand this statement.
> Perhaps it would be prudent for Mozilla to obtain EV certs for
> addons.mozilla.org from more than one EV CA?  Then you wouldn't have to fall
> back to using the *.mozilla.org DV cert if GlobalSign's OCSP Responder should
> ever melt again.

That's an interesting idea.  Feels like it has a high operational overhead

I'm working on getting another EV cert.
It would be interesting to know how many unique visits AMO has per day?
In reply to comment #24:
> That's an interesting idea.  Feels like it has a high operational overhead
>
> I'm working on getting another EV cert.

To echo comment #8, I hope you intend to choose a CA that you are confident are capable of operating a high-volume OCSP Responder!
Naturally.  I've looked at who provides EV for sites like paypal.com where downtime equals lots of money.
(In reply to comment #25)
According to this blog post [1] it can be assumed that there are at least 33 million OCSP requests per day originating from AMO. That's about ~ 400 requests per seconds on average. 

[1] http://blog.mozilla.com/addons/2009/08/11/how-many-firefox-users-use-add-ons/
Well, actually I think it's lot less then that, since I assumed (wrong) that AMO pings aren't made over HTTPS. Is this assumption correct?
(In reply to comments #28 and #29)
Eddy, thanks for finding that blog post.  Those stats are interesting, but...

A quick grep through mozilla-central finds...
browser/app/profile/firefox.js:pref("extensions.update.url", "https://versioncheck.addons.mozilla.org/update/VersionCheck.php?<snip>

I'm guessing, therefore, that "AMO pings" are made over HTTPS to versioncheck.addons.mozilla.org, which is using a DV cert from Equifax/GeoTrust/VeriSign that does not contain an OCSP Responder URL.
If I'm right, "AMO pings" would not have caused any OCSP lookups for GlobalSign's addons.mozilla.org EV cert.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.