Closed Bug 326485 Opened 14 years ago Closed 13 years ago

Sentry should ignore disabled products and also disable mirrors with domains that can't resolve

Categories

(mozilla.org Graveyard :: Server Operations, task, major)

x86
Linux
task
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: morgamic, Assigned: justin)

References

Details

Attachments

(2 files, 1 obsolete file)

The list of active products is becoming so long that sentry runs are taking too long, so the addition of a new product is delayed unnecessarily.

Using the admin interface, users are able to mark a product as "disabled" meaning that it should no longer be checked.

Tasks that need to be completed:
* add index for product_active in prod
* add WHERE clause to initial locations query in sentry.pl
Additional TODO:
    * Create a default page to catch previously working requests.  When bouncer flips the product to disabled status, people should be directed to a page that explains how to go to a mirror and retrieve previous releases manually.
Status: NEW → ASSIGNED
After further investigation:
* Sentry does not handle failed DNS lookups correctly
* Sentry was hanging on a particular mirror (http://mozilla.cnnet.upr.edu) because of this
* The mirror has been deleted so Sentry will no longer be affected by this
* A manual rerun of Sentry was forced

Solution:
* Add a Net::DNS lookup before Sentry uses LWP::UserAgent->request() to pull a HTTP HEAD from any mirror location.
* If the lookup fails, disable the mirror and continue to the next iteration in the main mirror loop.

This will probably require two additional Perl modules to be installed on gecko:
* URI
* Net::DNS
Summary: Sentry needs to ignore disabled products to shorten runtime → Sentry should ignore disabled products and also disable mirrors with domains that can't resolve
Depends on: 326664
Here is the new behavior for failed DNS resolution.  Posting patch shortly.
--
Testing: Firefox-1.5 on osx content-type: application/octet-stream
ftp.chg.ru for Firefox-1.5 on osx FAILED due to content-type mis-match.
ftp.chg.ru for Firefox-1.5 on win is okay.
DNS resolution for ftp.cnnet.upr.edu (Puerto Rico) FAILED!  Moving on to next mirror.
ftp.rz.tu-bs.de for Firefox-1.5 on linux is okay.
ftp.rz.tu-bs.de for Firefox-1.5 on osx is okay.
--
Attachment #211395 - Flags: review? → review?(justdave)
Attachment #211395 - Flags: review?(kveton)
Comment on attachment 211395 [details] [diff] [review]
Modification to Sentry to properly handle DNS resolution.

r=kveton this afternoon -- talked with him.
Attachment #211395 - Flags: review?(kveton) → review+
QA Contact: kveton → bouncer
Comment on attachment 211395 [details] [diff] [review]
Modification to Sentry to properly handle DNS resolution.

this ever get implemented?  Looks good to me.
Attachment #211395 - Flags: review?(justdave) → review+
No -- requested a review a long, long time ago but it never got looked at.  We should pick this up and deploy it so we can reduce Sentry's delay.  Some updates the patch needs:

* This block needs to be taken out of the Locations while() loop:
+    # first, we should test the DNS for the mirror
+    my $netres = Net::DNS::Resolver->new();
+    $netres->tcp_timeout('5');

* Need to set $ua->redirect_ok=false before the $ua->request call to disallow 200 OK HTTP responses, which sometimes pop up for certain mirrors instead of a 404 when a file is not found.
Attached patch Patch, v2Splinter Review
This patch should remove mirrors that fail DNS, and should also ignore disabled products.  I also added $ua->request_ok=false; to make sure that a redirect 200 OK response isn't mistaken for a valid file.
Attachment #211395 - Attachment is obsolete: true
Attachment #238508 - Flags: review?(justin)
Comment on attachment 238508 [details] [diff] [review]
Patch, v2

Looks good - assume you have tested...?
Attachment #238508 - Flags: review?(justin) → review+
Attached file v2 patch output
Here is Sentry's output, using 1.5.0.3 as a test case.
We should get this up soon -- speeding up Sentry is something that would help QA and Build quite a bit.
Assignee: morgamic → server-ops
Status: ASSIGNED → NEW
Component: Bouncer → Server Operations
Flags: review+
Product: Webtools → mozilla.org
Version: Trunk → other
Comment on attachment 238508 [details] [diff] [review]
Patch, v2

Yes, there is a pw accidentally diffed in the patch.  No, it no longer works with anything.
Severity: blocker → major
Assignee: server-ops → oremj
Assignee: oremj → justin
Patched version QA'd by Mike and me and is in production.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.