Closed
Bug 909902
Opened 11 years ago
Closed 10 years ago
[releases-api] deploy releases-api.mozilla.org on generic cluster
Categories
(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)
Infrastructure & Operations Graveyard
WebOps: Socorro
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: rhelmer, Assigned: bburton)
Details
As an experiment I tried splitting out the ftpscraper code from Socorro to a standalone project, it went ok: http://releases.paas.allizom.org https://github.com/rhelmer/releases Since we don't know exactly when a production PaaS will be available, could we run this on the generic cluster? The idea is that instead of this being internal to Socorro, it could be a public service that Socorro (and others) can consume. It has no important dependencies on Socorro code or services. Right now "scraper.py" runs from cron every 5 minutes and scrapes http://ftp.m.o for release metadata, and dumps to a JSON file. There is a very simple flask app which serves the JSON file with the right content-type as an index (anything else 404s) - we may want to refine this later, but does the job for now.
Assignee | ||
Comment 1•11 years ago
|
||
Per irc, naming releases-api Will document setup in https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org :rhelmer is working on a vendor submodule for dependencies and a .wsgi file to use it, then I'll deploy things We'll use this cluster's NFS share for storing the JSON file
Assignee: server-ops-webops → bburton
Status: NEW → ASSIGNED
Summary: deploy releases app on generic cluster → [releases-api] deploy releases-api.mozilla.org on generic cluster
Reporter | ||
Comment 2•11 years ago
|
||
vendor lib in place, and renamed the repo to "releases-api" for consistency: https://github.com/rhelmer/releases-api
Assignee | ||
Comment 3•11 years ago
|
||
(In reply to Robert Helmer [:rhelmer] from comment #2) > vendor lib in place, and renamed the repo to "releases-api" for consistency: > > https://github.com/rhelmer/releases-api To have it work with mod_wsgi we need some kind of .wsgi file that fixes up the path for the vendor submodule, the other flask app we have deployed looks like https://github.com/mozilla/treestatus/blob/master/treestatus.wsgi-dist But in looking at the content, do we need to use the Flask app? It seems like we could just serve index.html and the .json files directly from Apache?
Reporter | ||
Comment 4•11 years ago
|
||
(In reply to Brandon Burton [:solarce] from comment #3) > (In reply to Robert Helmer [:rhelmer] from comment #2) > > vendor lib in place, and renamed the repo to "releases-api" for consistency: > > > > https://github.com/rhelmer/releases-api > > To have it work with mod_wsgi we need some kind of .wsgi file that fixes up > the path for the vendor submodule, the other flask app we have deployed > looks like > https://github.com/mozilla/treestatus/blob/master/treestatus.wsgi-dist > > But in looking at the content, do we need to use the Flask app? > > It seems like we could just serve index.html and the .json files directly > from Apache? Oh right, sorry forgot about the .wsgi file - I think we will want it, it now has an index page and even if we use JSON files on-disk we'll want to have different endpoints for different uses in the very near future.
Reporter | ||
Comment 5•11 years ago
|
||
How does this look? https://github.com/rhelmer/releases-api/blob/master/releases-api.wsgi I think it should work, it's about as simple as I can get it.
Flags: needinfo?(bburton)
Assignee | ||
Comment 6•11 years ago
|
||
(In reply to Robert Helmer [:rhelmer] from comment #5) > How does this look? > https://github.com/rhelmer/releases-api/blob/master/releases-api.wsgi > > I think it should work, it's about as simple as I can get it. We'll find out shortly bburton@althalus [08:38:59] [~/code/mozilla/sysadmins/puppet/trunk] -> % vim modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org bburton@althalus [08:39:19] [~/code/mozilla/sysadmins/puppet/trunk] -> % svn add $_ A modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org bburton@althalus [08:39:21] [~/code/mozilla/sysadmins/puppet/trunk] -> % svn ci -m "initial apache config for releases-api.mozilla.org, bug 909902" Adding trunk/modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org Transmitting file data . Committed revision 74062.
Flags: needinfo?(bburton)
Assignee | ||
Comment 7•11 years ago
|
||
Got us almost all the way there, https://github.com/rhelmer/releases-api/pull/4 contains the fix needed to work in prod
Assignee | ||
Comment 8•11 years ago
|
||
Fix merged and deployed, onto setting up the cron job [root@genericadm.private.phx1 releases-api-code]# git pull remote: Counting objects: 6, done. remote: Compressing objects: 100% (2/2), done. remote: Total 4 (delta 2), reused 4 (delta 2) Unpacking objects: 100% (4/4), done. From https://github.com/rhelmer/releases-api 1ae44d3..9234cbf master -> origin/master Updating 1ae44d3..9234cbf Fast-forward releases-api.wsgi | 1 + 1 file changed, 1 insertion(+) [root@genericadm.private.phx1 releases-api-code]# /data/genericrhel6/deploy releases-api.mozilla.org [2013-08-28 10:19:17] Running rsync_project [2013-08-28 10:19:17] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/releases-api.mozilla.org/ /data/genericrhel6/www/releases-api.mozilla.org/ [2013-08-28 10:19:17] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/releases-api.mozilla.org/ /data/genericrhel6/www/releases-api.mozilla.org/ (0.064s) [2013-08-28 10:19:17] Finished rsync_project (0.064s) [2013-08-28 10:19:17] Running commit_www [2013-08-28 10:19:17] [localhost] running: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['releases-api.mozilla.org']' [2013-08-28 10:19:22] [localhost] finished: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['releases-api.mozilla.org']' (5.226s) [localhost] out: [master 24832f1] deploy [releases-api.mozilla.org] [localhost] out: 1 file changed, 1 insertion(+) [2013-08-28 10:19:22] Finished commit_www (5.227s) [2013-08-28 10:19:22] Running push_www [2013-08-28 10:19:22] [generic1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:22] [generic2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:22] [generic3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:22] [generic4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:22] [generic5.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:22] [generic102.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org [2013-08-28 10:19:29] [generic2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (6.900s) [generic2.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:29] [generic3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.137s) [generic3.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:30] [generic5.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.749s) [generic5.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:30] [generic102.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.945s) [generic102.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:30] [generic4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.968s) [generic4.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:31] [generic1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (8.376s) [generic1.webapp.phx1.mozilla.com] out: Not removing fhr.cdn.mozilla.net/web-output.orig/ [generic1.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-08-28 10:19:31] Finished push_www (8.378s)
Comment 9•11 years ago
|
||
What's the point of ftpscraper? Trying to wrap my head around what this does and why we need it, how it compares to product-details, etc. Kinda wondering if ftp.mozilla.org scraping is the right/best way to do this or if we can/should do better. Thanks!
Reporter | ||
Comment 10•11 years ago
|
||
(In reply to Jake Maul [:jakem] from comment #9) > What's the point of ftpscraper? Trying to wrap my head around what this does > and why we need it, how it compares to product-details, etc. Kinda wondering > if ftp.mozilla.org scraping is the right/best way to do this or if we > can/should do better. releng told us to use ftp.mozilla.org for this data - I'd love to do better! The point of this bug is to get this code out of Socorro and into a standalone service, if we come up with a better way to get this data then we can and should do it.
Comment 11•11 years ago
|
||
(In reply to Jake Maul [:jakem] from comment #9) > What's the point of ftpscraper? Trying to wrap my head around what this does > and why we need it, how it compares to product-details, etc. Kinda wondering > if ftp.mozilla.org scraping is the right/best way to do this or if we > can/should do better. Rob already addressed it -- but in addition, we've had a bit of a chicken-and-egg problem: without a sample service to have people try out, it's difficult to move the conversation about having a service forward. I worked on getting JSON available from Firefox and B2G that describes releases, and that has made the scraping better for B2G... shortly will be better for Firefox. (less parsing of paths, more using the JSON keys!) There's still quite a ways to go, but IMO, this work Rob is doing *is* making the situation much better.
Assignee | ||
Comment 12•11 years ago
|
||
https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org is mostly fleshed out, :rhelmer can you fill out the about the service, developer docs, and if you want add a diagram, otherwise remove those sections? Thanks!
Flags: needinfo?(rhelmer)
Reporter | ||
Updated•10 years ago
|
Flags: needinfo?(rhelmer)
Reporter | ||
Comment 13•10 years ago
|
||
(In reply to Brandon Burton [:solarce] from comment #12) > https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org is > mostly fleshed out, :rhelmer can you fill out the about the service, > developer docs, and if you want add a diagram, otherwise remove those > sections? > > Thanks! I filed bug 953235 as a followup so this one can be closed out. Sorry for taking so long to respond!
Assignee | ||
Updated•10 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Updated•8 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•