747747 - Test runs getting 503 Server Too Busy from ftp.mozilla.org

Status: RESOLVED DUPLICATE of bug 735870

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Bill Gianopoulos [:WG9s]]

We seem to be getting Red builds with the following errors:

Connecting to ftp.mozilla.org|63.245.209.137|:80... connected. HTTP request sent, awaiting response... 503 Server Too Busy

Should the failure of a single mirror server result in red tests?

Perhaps we should be using stage.mozilla.org as the site to load from so that at least if there is an issue it is related to a mozilla.org controlled website.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Is this possibly a misconfiguration on new FTP nodes?

Comment 2

[Bill Gianopoulos [:WG9s]]

That could be.  Another idea would be to take whatever is doing the load balancing between the ftp.mozilla.org sites to do a more complex check using a script rather than just verifying it is accepting connections on the port to ensure that requests are returning a 200 status rather than this 503 status error.

Comment 3

[Bill Gianopoulos [:WG9s]]

So the point is that even if this issue was caused by new servers not being configured to accept an adequate number of simultaneous connection, the test in the lad balancer for if a server is up or not instead of deciding if it accepts a connection to port 80 should rally be making sure that an attempt to actually load the page http://ftp.mozilla.org/pub/mozilla.org/ results in a 200 status.

And in case you have not figured it out, this is the kind of thing I do on my real job.

Comment 4

[Chris AtLee [:catlee]]

See also bug 735870

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Sounds like a dupe to me.

Bill, fyi, ftp.mozilla.org is not a mirror, and 503 isn't something that can be tested for in advance, since it only happens on some requests.  Which is not to say it shouldn't be fixed!