378634 - Need a script to verify bouncer1 links against bouncer2

Status: RESOLVED INVALID

(Webtools :: Bouncer, defect)

Description

[Michael Morgan [:morgamic]]

We need a script to generate all possible links for bouncer1 and then verify their existence in bouncer2.

I have something that uses sentry.pl to do this.  Will post it here after I've cleaned it up a bit.

Comment 1

[Michael Morgan [:morgamic]]

Attachment: v1, quick and dirty verify script without file sizes

This is a quickly made script to check bouncer1 links against bouncer2.

Comment 2

[Michael Morgan [:morgamic]]

Attachment: v2, adding content-type check

Comment 3

[Michael Morgan [:morgamic]]

For a dump database to use, here is the bouncer1 dump from this morning:
http://people.mozilla.org/~morgamic/mozilla-mirror.2007.04.24.tar.gz

If you want to play with things we know exist, you can modify the products_sql to only retrieve Tb 2.0.

Comment 5

[Robert Helmer [:rhelmer]]

Comment on attachment 262670 [details] [diff] [review]
v2, adding content-type check

Looks good! Here are some suggestions:

Let's separate the part of this script that calls into the DB from the part that queries the web service.. I'd like to be able to e.g. query the DB and save the result to a file, then query the webservice based on the contents of that file.

># Locales
>my @locales = qw(ar be bg ca cs da de el en-GB en-US es-AR es-ES eu  fi fr fy-NL ga-IE ga-IN gu-IN he hu it ja ja-JP-mac ka ko ku lt mk mn nb-NO nl nn-NO pa-IN pl pt-BR pt-PT ro ru sk sl sv-SE tr zh-CN zh-TW);


We could get this from CVS:mozilla/%product%/locales/shipped-locales if we know the release tag (%PRODUCT%_%VERSION%_RELEASE).


># Base url for bouncer
>my $bouncer_url = 'http://download2.mozilla.org/';


bouncer_url should be configurable via command-line switch (and have download.mozilla.org as a default).


># Generate all our URLs and do a HEAD request on them to test the response.
># Right now we're doing a "does it exist and give us a 200?" test.


Right now bouncer redirects to mozilla.org if sentry hasn't found anything (yet).. also some mirrors might do this wrong and serve up a 200 with a "file not found" page.

Should check the "Content-Type:" and ensure that it does not contain '"text/html".

Comment 6

[Michael Morgan [:morgamic]]

(In reply to comment #5)
> Right now bouncer redirects to mozilla.org if sentry hasn't found anything
> (yet).. also some mirrors might do this wrong and serve up a 200 with a "file
> not found" page.
Filed bug 380074 about this -- will fix it asap.

> Should check the "Content-Type:" and ensure that it does not contain
> '"text/html".
This check already exists in v2 of the patch. 

Comment 7

[Robert Helmer [:rhelmer]]

No longer needed; went with a modified version of cesar's AUS script instead.