Closed Bug 326485 Opened 14 years ago Closed 13 years ago
Sentry should ignore disabled products and also disable mirrors with domains that can't resolve
The list of active products is becoming so long that sentry runs are taking too long, so the addition of a new product is delayed unnecessarily. Using the admin interface, users are able to mark a product as "disabled" meaning that it should no longer be checked. Tasks that need to be completed: * add index for product_active in prod * add WHERE clause to initial locations query in sentry.pl
Additional TODO: * Create a default page to catch previously working requests. When bouncer flips the product to disabled status, people should be directed to a page that explains how to go to a mirror and retrieve previous releases manually.
Status: NEW → ASSIGNED
After further investigation: * Sentry does not handle failed DNS lookups correctly * Sentry was hanging on a particular mirror (http://mozilla.cnnet.upr.edu) because of this * The mirror has been deleted so Sentry will no longer be affected by this * A manual rerun of Sentry was forced Solution: * Add a Net::DNS lookup before Sentry uses LWP::UserAgent->request() to pull a HTTP HEAD from any mirror location. * If the lookup fails, disable the mirror and continue to the next iteration in the main mirror loop. This will probably require two additional Perl modules to be installed on gecko: * URI * Net::DNS
Summary: Sentry needs to ignore disabled products to shorten runtime → Sentry should ignore disabled products and also disable mirrors with domains that can't resolve
Here is the new behavior for failed DNS resolution. Posting patch shortly. -- Testing: Firefox-1.5 on osx content-type: application/octet-stream ftp.chg.ru for Firefox-1.5 on osx FAILED due to content-type mis-match. ftp.chg.ru for Firefox-1.5 on win is okay. DNS resolution for ftp.cnnet.upr.edu (Puerto Rico) FAILED! Moving on to next mirror. ftp.rz.tu-bs.de for Firefox-1.5 on linux is okay. ftp.rz.tu-bs.de for Firefox-1.5 on osx is okay. --
Attachment #211395 - Flags: review? → review?(justdave)
Attachment #211395 - Flags: review?(kveton)
Comment on attachment 211395 [details] [diff] [review] Modification to Sentry to properly handle DNS resolution. r=kveton this afternoon -- talked with him.
Attachment #211395 - Flags: review?(kveton) → review+
Comment on attachment 211395 [details] [diff] [review] Modification to Sentry to properly handle DNS resolution. this ever get implemented? Looks good to me.
Attachment #211395 - Flags: review?(justdave) → review+
No -- requested a review a long, long time ago but it never got looked at. We should pick this up and deploy it so we can reduce Sentry's delay. Some updates the patch needs: * This block needs to be taken out of the Locations while() loop: + # first, we should test the DNS for the mirror + my $netres = Net::DNS::Resolver->new(); + $netres->tcp_timeout('5'); * Need to set $ua->redirect_ok=false before the $ua->request call to disallow 200 OK HTTP responses, which sometimes pop up for certain mirrors instead of a 404 when a file is not found.
This patch should remove mirrors that fail DNS, and should also ignore disabled products. I also added $ua->request_ok=false; to make sure that a redirect 200 OK response isn't mistaken for a valid file.
Comment on attachment 238508 [details] [diff] [review] Patch, v2 Looks good - assume you have tested...?
Attachment #238508 - Flags: review?(justin) → review+
Here is Sentry's output, using 220.127.116.11 as a test case.
We should get this up soon -- speeding up Sentry is something that would help QA and Build quite a bit.
Assignee: morgamic → server-ops
Status: ASSIGNED → NEW
Component: Bouncer → Server Operations
Product: Webtools → mozilla.org
Version: Trunk → other
Comment on attachment 238508 [details] [diff] [review] Patch, v2 Yes, there is a pw accidentally diffed in the patch. No, it no longer works with anything.
Patched version QA'd by Mike and me and is in production.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.