924785 - Cannot reach MXR raw file link (effective_tld_names.dat)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Mihnea Dobrescu-Balaur (:mihneadb)]

http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat

I get "The connection to the server was reset while the page was loading."

Comment 3

[Kendall Libby [:fubar] (he/him)]

mxr was ddos'ed yesterday and part of the attack was aimed at that particular url. once I verify it's stopped, I'll remove the block and update this bug.

Comment 4

[Gervase Markham [:gerv]]

OK - thanks :-)

Gerv

Comment 5

[Kendall Libby [:fubar] (he/him)]

"attack" against that url continues. It appears to be an issue with opencart attempting to load that file; a support request to them has been submitted.

Comment 6

[Martijn Pieters]

Temporary work-around is to load from the GitHub mirror instead:

    https://raw.github.com/mozilla/mozilla-central/master/netwerk/dns/effective_tld_names.dat

Comment 7

[Gervase Markham [:gerv]]

Or even hg:

https://hg.mozilla.org/mozilla-central/raw-file/tip/netwerk/dns/effective_tld_names.dat

fubar: is there a URL where we can track the opencart support request?

Perhaps we need to make it clear that people shouldn't bake this MXR URL into their products... :-| What would be a better canonical URL? Perhaps we need a redirect on publicsuffix.org that people can use.

Gerv

Comment 8

[Martijn Pieters]

(In reply to Gervase Markham [:gerv] from comment #7)
> Or even hg:
> 
> https://hg.mozilla.org/mozilla-central/raw-file/tip/netwerk/dns/
> effective_tld_names.dat

I assumed the GitHub servers were better equipped for high load, but I'm perfectly prepared to believe I underestimated the Moz Mercurial infrastructure. :-P

I *do* think publicsuffix.org should use a more robust strategy for serving that file somewhere.

Comment 9

[Kendall Libby [:fubar] (he/him)]

(In reply to Gervase Markham [:gerv] from comment #7)
> Or even hg:
> 
> https://hg.mozilla.org/mozilla-central/raw-file/tip/netwerk/dns/
> effective_tld_names.dat
> 
> fubar: is there a URL where we can track the opencart support request?


There was, but I seem to have misplaced it. In any case, the issue isn't in opencart itself; I checked their github repo.

> Perhaps we need to make it clear that people shouldn't bake this MXR URL
> into their products... :-| What would be a better canonical URL? Perhaps we
> need a redirect on publicsuffix.org that people can use.

I had no idea that existed, but that certainly explains why someone would just link to it. :-( I would suggest either something that people can't link to directly, or something one CAN link to but has the infrastructure to handle a significant amount of load - we're currently seeing ~40k hits per second.

Comment 10

[Gervase Markham [:gerv]]

(In reply to Kendall Libby [:fubar] from comment #9)
> I had no idea that existed, but that certainly explains why someone would
> just link to it. :-( I would suggest either something that people can't link
> to directly, or something one CAN link to but has the infrastructure to
> handle a significant amount of load - we're currently seeing ~40k hits per
> second.

This data changes monthly; that level of load is utterly inappropriate, and would be whatever URL we used.

Gerv

Comment 11

[Kendall Libby [:fubar] (he/him)]

Also, I'm going to go back and try to find other possible culprits: while the issue has clearly shown up in opencart sites, it's possibly worth taking the mod developer at his word that he'd fixed a while ago. I also find it slightly hard to believe that the opencart+analytics mod is popular enough for the amount of traffic we're seeing.

Comment 12

[Kendall Libby [:fubar] (he/him)]

I've opened up https for that URL since it's much, much lower traffic. I enabled http very briefly, to see if perhaps the crazy numbers I'm currently seeing were simply because clients were constantly re-connecting but the load went immediately to 70+. 

Trying to determine from the logs if the vulnerablity scan was the original culprit (it hammering the machine was how we noticed all of the traffic to effective_tld_names.dat), and now clients are stuck in freak-out mode. If they are (e.g. clients retrying quickly) then I'll see about setting up something beefy to shunt the load onto until things even out.

Comment 13

[Kendall Libby [:fubar] (he/him)]

I've opened up the fire hose a little bit; incoming connections are throttled and I'm slowly increasing the limit, trying to find a comfortable balance between beating the server to death and clearing the backlog - lots of clients were being crazy and retrying every second.

Comment 14

[Gervase Markham [:gerv]]

That's some very broken software. If we can figure out what these clients are, that would be awesome.

Gerv

Comment 15

[Gervase Markham [:gerv]]

:fubar: any more recent news here?

Gerv

Comment 16

[Kendall Libby [:fubar] (he/him)]

Negative; I still have it on my near-term list, but it's a lower priority.

Comment 17

[Mihnea Dobrescu-Balaur (:mihneadb)]

All back to normal, I guess we can close this.

Comment 18

[Gervase Markham [:gerv]]

I was hoping :fubar: would be able to look and see if there are any clues to know what app or service is requesting the file so ridiculously often, so we can contact them and tell them to knock it off.

fubar: any chance of that?

Gerv

Comment 19

[Chris Turra [:cturra]]

i just put a traffic rule in place on our load balancer to redirect (http 302) all requests for effective_tld_names.dat that do *not* contain a referer header of mxr.mozilla.org over to the publicsuffix.org to serve these requests out of the static cluster.


 $host    = http.getHostHeader();
 $referer = http.getHeader("Referer");
 $path    = http.getPath();

 if ( string.containsI($host, "mxr.mozilla.org")) {
    if (
          (!string.containsI($referer, "mxr.mozilla.org" )) &&
          ( string.containsI($path,    "effective_tld_names.dat"))
       ) {
          http.redirect("https://publicsuffix.org/list/effective_tld_names.dat");
    }
 }