292942 - Need Sentry to test servers for byte-range capability

Status: RESOLVED FIXED

(Webtools :: Bouncer, defect)

Description

[Dave Miller [:justdave]]

The Firefox 1.1 application update system is intending to do incremental
background downloading of updates by requesting small chunks of the file
periodically, to avoid using all of the user's available bandwidth if they're on
a slow connection.  In order to effectively pull this off, we'll need Sentry to
be checking the servers to ensure they support byte range requests via HTTP for
the download channel used by the application update service.

Comment 1

[Scott Kveton]

We should probably check all of the mirrors today and see what percentage of
them actually provide this functionality already.  If not many mirrors provide
it, it would be rather painful to get it added in.  We want to leverage default
configurations as much as possible.

Justdave: if I give you a list of all of the mirrors, can you test them?

Comment 2

[Dave Miller [:justdave]]

(In reply to comment #1)

> Justdave: if I give you a list of all of the mirrors, can you test them?

Sure.  Anyone know how to test for such a thing?  Can I pass the headers
requesting it and see if I get an error?  Or should I expect a non-200 response
code if it works? (like 206 or something)

Comment 3

[James Ross]

Perform a HEAD HTTP operation on a file on the server, and check for the
"Accept-Ranges" header returned including "bytes". Not sure what the easiest way
to check it actually works, though.

Comment 4

[Fred Wenzel [:wenzel]]

Actually a HEAD request is not sufficient to be sure, because the Accept-Ranges header is optional (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.5).

But just requesting a range of, say, 2K unconditionally and checking for a 206 response code ("Partial content") works. I have just written a little script that does that with the current mirrors.

Comment 5

[Fred Wenzel [:wenzel]]

All right: In the current active mirrors list there is only one that does not support byte ranges:

"Wroclaw Centre Supercomputing sent wrong response header (405 Method Not Allowed)!
URL: http://ftp.pwr.wroc.pl/pub/mozilla/firefox....."

Maybe we can ask them to turn it on for us?

(I do not mention other mirrors here that didn't return the correct data due to unrelated problems, mainly error 404 or 500).

Comment 6

[Michael Morgan [:morgamic]]

Justin, mrz -- do you guys know who our mirror contact is there?

Comment 7

[Justin Fitzhugh]

Yea, I'll ping them and ask them to change it.

Comment 8

[Justin Fitzhugh]

He thinks this is fixed - could you re-test?

Comment 9

[Fred Wenzel [:wenzel]]

Sorry for the delay: Now the server does not answer at all anymore, quote:

"Wroclaw Centre Supercomputing sent wrong response header (500 read timeout)!"

Maybe they are doing maintenance or so -- I will check again in a few hours.

Comment 10

[Fred Wenzel [:wenzel]]

Justin: I just re-checked again and the  Wroclaw server still answers with an error 405 (Method Not Allowed) to my request.

However, this does not seem to be a Range problem anymore: Now the server runs Apache, but the error 405 is returned on *any* download request, not just Range ones:
http://ftp.pwr.wroc.pl/pub/mozilla/firefox/releases/2.0.0.7/win32/en-US/Firefox%20Setup%202.0.0.7.exe

Comment 11

[Fred Wenzel [:wenzel]]

Attachment: Sentry hash checking

This patch teaches sentry to hash-check files. It takes a random chunk of the specified size and compares the local hash with the remote hash.

You need a local tree to do that, obviously.

Also, if I committed any major Perl crimes, let me know. ;)

Comment 12

[Fred Wenzel [:wenzel]]

Comment on attachment 284335 [details] [diff] [review]
Sentry hash checking

Oops, other than this being the wrong bug for the patch, nothing went wrong.

I uploaded the patch to bug 371237 again.

Comment 13

[Fred Wenzel [:wenzel]]

The patch from bug 371237 was committed to SVN, r7784. This includes checking mirrors for byte-range capabilities and using the byte ranges to verify randomly chosen pieces of the mirrored files.