Closed Bug 408673 Opened 12 years ago Closed 10 years ago

Need a command line script to ping mirror uptake

Categories

(Webtools :: Bouncer, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: morgamic, Assigned: justdave)

References

Details

Attachments

(2 files)

Need a script to echo out mirror uptake values.

The mirror uptake values would mimic the behavior of this script:
http://download.mozilla.org/admin/monitor.php

Basically for a given product, it should echo out the % of bandwidth it is currently hosted on according to the DB.

Platform could be an optional parameter.  Output should be simple plaintext, like a MySQL query result.  Should optionally be able to get it back in XML format.  Text format is default.
I am not a bouncer admin: can I get a screenshot?
Here's a Python script for the mirror uptake. I added a new "python" folder to Bouncer because it didn't have Python scripts before.

As asked for above above, you can define a product you want to query and you can get XML output (including a little DTD) or a plain text table (the default). The output can be restricted by OS too.

Just enter ./uptake.py --help to see the available options. Each of the options can also be defined in the config file.

Please review and let me know if anything is missing.
Assignee: morgamic → fwenzel
Status: NEW → ASSIGNED
Attachment #294421 - Flags: review?(morgamic)
I forgot to mention an external requirement: You need the CSE Python library (written by Lars Lohn) which has been used in a few of our products before (http://wiki.osuosl.org/display/Bouncer/cse+utils).
Rob/John -- is a python command-line script something that you can use from the current build system?
Comment on attachment 294421 [details] [diff] [review]
Python CLI script for mirror uptake

This works for me.  Can you check it in?
Attachment #294421 - Flags: review?(morgamic) → review+
Oops, I missed this at first, sorry for the delay. I checked this into the SVN tree, r10249.

Marking FIXED.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
ummm... having these files in svn makes me wonder. How do build automation machines get these scripts? We already pull from cvs - do we now have to pull from svn also? Or should this script live with rest of automation in cvs?
(In reply to comment #8)
> ummm... having these files in svn makes me wonder. How do build automation
> machines get these scripts? We already pull from cvs - do we now have to pull
> from svn also? Or should this script live with rest of automation in cvs?

Whats the best way to get these svn-based scripts picked up and used by our hg-based release-automation?
This is an implementation detail on our side really, but assuming the script would run on one platform like final_verification then I'd suggest we pick Mac. We have svn v1.3.2 on bm-xserve05 (Tb2) & fx-mac-1.9-slave2 (Fx3.0), and v1.4.6 on bm-xserveN/moz2-darwin9-slaveN (Fx3.5). Linux is the other option, if we install svn on production-prometheus-vm (Tb2.0); v1.4.2 is present on fx-linux-1.9-slave2 and moz2-linux-slaveN.
Fred - does this script work with the current Bouncer DB?  Jeremy, what would it take to get this up and running and get build access to it?
(btw - build is now ready to use it, so if we could help them out that'd be awesome)
Will you include the cse library, so that it does not need to be installed globally? None of our apps use it and it's kind of a pain to maintain since not (m)any projects use it.
Dave it appears build will need direct access to the bouncer master for this script to work. How do you suggest we do that?
Mike: Yes, I think it should still work, I don't think we removed any columns or so.

(In reply to comment #13)
> Will you include the cse library

Yes, I can do that. I'll drop it into the bouncer/python dir. For reference, it's Lars' library that I snagged from here: http://wiki.osuosl.org/display/Bouncer/cse+utils
(In reply to comment #15)
> Yes, I can do that. I'll drop it into the bouncer/python dir.

r49075.
(In reply to comment #14)
> Dave it appears build will need direct access to the bouncer master for this
> script to work. How do you suggest we do that?

Unless I am mistaken, they only need read access to that database. So if you can give them that, they could run the script locally.
Giving access to that database to the entire build network seems a bit excessive.

Why don't we make this a CGI, install it on the download server, and have the command line script hit that URL and parse the output?
(In reply to comment #18)
> Why don't we make this a CGI

Yea, I can do that.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
This patch adds some CGI magic to the script. You want something like this as the .htaccess file in bouncer's "python" dir:

$ cat .htaccess 
AddHandler cgi-script py
Options +ExecCGI

... and then you can call http://.../uptake.py

with the possible GET variables "format" (text or xml, defaulting to text), "product" or "os".

Sound good?
Attachment #394377 - Flags: review?(morgamic)
Status: REOPENED → ASSIGNED
Attachment #394377 - Flags: review?(morgamic) → review?(dd)
Comment on attachment 394377 [details] [diff] [review]
Adding CGI handling to the script

Hey Dave, can you check out if this code makes sense :)

FWIW, you don't need to set up a bouncer dev instance if you don't want to, you can look at the script's output at http://fwenzel.khan.mozilla.org/bouncer-cgi/uptake.py?format=xml .
Attachment #394377 - Flags: review?(jbalogh)
Attachment #394377 - Flags: review?(dd)
Attachment #394377 - Flags: review+
Comment on attachment 394377 [details] [diff] [review]
Adding CGI handling to the script

Unfortunately I don't understand what this script is supposed to do, other than produce the output you mentioned.  So I'm going to have jbalogh give a look at it.

I didn't see anything overly gross from a python sense, except for one thing:

import cgi should be at the top of the file not burried in the handle_cgi method per PEP-8.

Fix that and I'll be happy.

Jeff- please weigh in if you can.
Oops, yea, I duplicated that line. I fixed that locally.
Attachment #394377 - Flags: review?(jbalogh) → review-
Comment on attachment 394377 [details] [diff] [review]
Adding CGI handling to the script

It seems to work, but doesn't really look like the Python I know.
(In reply to comment #24)
> It seems to work, but doesn't really look like the Python I know.

wtf
I made some style modifications after chatting with jbalogh and committed this to SVN in r49206.

Passing this on to justdave to set it up.
Assignee: fwenzel → justdave
Justdave: Is this deployed?
Blocks: 539522
Note that this is becoming a part of the tuxedo branch, and will have to be called just a tad bit differently. See bug 539522.
(In reply to comment #27)
> Justdave: Is this deployed?

Yes, it's sentryviewer/csv.cgi on the production branch, has this change from svn in production though:

 print "Content-type: text/csv; charset=utf-8"
+print "Content-disposition: attachment; name=mozillamirrors.csv"
 print
 print "# mirror_id, mirror_name, mirror_baseurl, mirror_rating, mirror_active"
I'll close this for now, though this API is now part of the tuxedo app, so we can deprecate this part of sentryviewer when the new version of Bouncer lands.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.