Closed Bug 909902 Opened 11 years ago Closed 10 years ago

[releases-api] deploy releases-api.mozilla.org on generic cluster

Categories

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rhelmer, Assigned: bburton)

Details

As an experiment I tried splitting out the ftpscraper code from Socorro to a standalone project, it went ok:

http://releases.paas.allizom.org
https://github.com/rhelmer/releases

Since we don't know exactly when a production PaaS will be available, could we run this on the generic cluster? The idea is that instead of this being internal to Socorro, it could be a public service that Socorro (and others) can consume. It has no important dependencies on Socorro code or services.

Right now "scraper.py" runs from cron every 5 minutes and scrapes http://ftp.m.o for release metadata, and dumps to a JSON file. There is a very simple flask app which serves the JSON file with the right content-type as an index (anything else 404s) - we may want to refine this later, but does the job for now.
Per irc, naming releases-api

Will document setup in https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org

:rhelmer is working on a vendor submodule for dependencies and a .wsgi file to use it, then I'll deploy things

We'll use this cluster's NFS share for storing the JSON file
Assignee: server-ops-webops → bburton
Status: NEW → ASSIGNED
Summary: deploy releases app on generic cluster → [releases-api] deploy releases-api.mozilla.org on generic cluster
vendor lib in place, and renamed the repo to "releases-api" for consistency:

https://github.com/rhelmer/releases-api
(In reply to Robert Helmer [:rhelmer] from comment #2)
> vendor lib in place, and renamed the repo to "releases-api" for consistency:
> 
> https://github.com/rhelmer/releases-api

To have it work with mod_wsgi we need some kind of .wsgi file that fixes up the path for the vendor submodule, the other flask app we have deployed looks like https://github.com/mozilla/treestatus/blob/master/treestatus.wsgi-dist

But in looking at the content, do we need to use the Flask app? 

It seems like we could just serve index.html and the .json files directly from Apache?
(In reply to Brandon Burton [:solarce] from comment #3)
> (In reply to Robert Helmer [:rhelmer] from comment #2)
> > vendor lib in place, and renamed the repo to "releases-api" for consistency:
> > 
> > https://github.com/rhelmer/releases-api
> 
> To have it work with mod_wsgi we need some kind of .wsgi file that fixes up
> the path for the vendor submodule, the other flask app we have deployed
> looks like
> https://github.com/mozilla/treestatus/blob/master/treestatus.wsgi-dist
> 
> But in looking at the content, do we need to use the Flask app? 
> 
> It seems like we could just serve index.html and the .json files directly
> from Apache?

Oh right, sorry forgot about the .wsgi file - I think we will want it, it now has an index page and even if we use JSON files on-disk we'll want to have different endpoints for different uses in the very near future.
How does this look? https://github.com/rhelmer/releases-api/blob/master/releases-api.wsgi

I think it should work, it's about as simple as I can get it.
Flags: needinfo?(bburton)
(In reply to Robert Helmer [:rhelmer] from comment #5)
> How does this look?
> https://github.com/rhelmer/releases-api/blob/master/releases-api.wsgi
> 
> I think it should work, it's about as simple as I can get it.

We'll find out shortly

bburton@althalus [08:38:59] [~/code/mozilla/sysadmins/puppet/trunk]
-> % vim modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org
bburton@althalus [08:39:19] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn add $_
A         modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org
bburton@althalus [08:39:21] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "initial apache config for releases-api.mozilla.org, bug 909902"
Adding         trunk/modules/webapp/files/genericrhel6/etc-httpd/domains/releases-api.mozilla.org
Transmitting file data .
Committed revision 74062.
Flags: needinfo?(bburton)
Got us almost all the way there, https://github.com/rhelmer/releases-api/pull/4 contains the fix needed to work in prod
Fix merged and deployed, onto setting up the cron job

[root@genericadm.private.phx1 releases-api-code]# git pull
remote: Counting objects: 6, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 2), reused 4 (delta 2)
Unpacking objects: 100% (4/4), done.
From https://github.com/rhelmer/releases-api
   1ae44d3..9234cbf  master     -> origin/master
Updating 1ae44d3..9234cbf
Fast-forward
 releases-api.wsgi | 1 +
 1 file changed, 1 insertion(+)
[root@genericadm.private.phx1 releases-api-code]# /data/genericrhel6/deploy releases-api.mozilla.org
[2013-08-28 10:19:17] Running rsync_project
[2013-08-28 10:19:17] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/releases-api.mozilla.org/ /data/genericrhel6/www/releases-api.mozilla.org/
[2013-08-28 10:19:17] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/releases-api.mozilla.org/ /data/genericrhel6/www/releases-api.mozilla.org/ (0.064s)
[2013-08-28 10:19:17] Finished rsync_project (0.064s)
[2013-08-28 10:19:17] Running commit_www
[2013-08-28 10:19:17] [localhost] running: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['releases-api.mozilla.org']'
[2013-08-28 10:19:22] [localhost] finished: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['releases-api.mozilla.org']' (5.226s)
[localhost] out: [master 24832f1] deploy [releases-api.mozilla.org]
[localhost] out: 1 file changed, 1 insertion(+)
[2013-08-28 10:19:22] Finished commit_www (5.227s)
[2013-08-28 10:19:22] Running push_www
[2013-08-28 10:19:22] [generic1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:22] [generic2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:22] [generic3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:22] [generic4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:22] [generic5.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:22] [generic102.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh releases-api.mozilla.org
[2013-08-28 10:19:29] [generic2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (6.900s)
[generic2.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:29] [generic3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.137s)
[generic3.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:30] [generic5.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.749s)
[generic5.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:30] [generic102.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.945s)
[generic102.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:30] [generic4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (7.968s)
[generic4.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:31] [generic1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh releases-api.mozilla.org (8.376s)
[generic1.webapp.phx1.mozilla.com] out: Not removing fhr.cdn.mozilla.net/web-output.orig/
[generic1.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/
[2013-08-28 10:19:31] Finished push_www (8.378s)
What's the point of ftpscraper? Trying to wrap my head around what this does and why we need it, how it compares to product-details, etc. Kinda wondering if ftp.mozilla.org scraping is the right/best way to do this or if we can/should do better.

Thanks!
(In reply to Jake Maul [:jakem] from comment #9)
> What's the point of ftpscraper? Trying to wrap my head around what this does
> and why we need it, how it compares to product-details, etc. Kinda wondering
> if ftp.mozilla.org scraping is the right/best way to do this or if we
> can/should do better.

releng told us to use ftp.mozilla.org for this data - I'd love to do better! The point of this bug is to get this code out of Socorro and into a standalone service, if we come up with a better way to get this data then we can and should do it.
(In reply to Jake Maul [:jakem] from comment #9)
> What's the point of ftpscraper? Trying to wrap my head around what this does
> and why we need it, how it compares to product-details, etc. Kinda wondering
> if ftp.mozilla.org scraping is the right/best way to do this or if we
> can/should do better.

Rob already addressed it -- but in addition, we've had a bit of a chicken-and-egg problem: without a sample service to have people try out, it's difficult to move the conversation about having a service forward. 

I worked on getting JSON available from Firefox and B2G that describes releases, and that has made the scraping better for B2G... shortly will be better for Firefox. (less parsing of paths, more using the JSON keys!)

There's still quite a ways to go, but IMO, this work Rob is doing *is* making the situation much better.
https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org is mostly fleshed out, :rhelmer can you fill out the about the service, developer docs, and if you want add a diagram, otherwise remove those sections?

Thanks!
Flags: needinfo?(rhelmer)
Flags: needinfo?(rhelmer)
(In reply to Brandon Burton [:solarce] from comment #12)
> https://mana.mozilla.org/wiki/display/websites/releases-api.mozilla.org is
> mostly fleshed out, :rhelmer can you fill out the about the service,
> developer docs, and if you want add a diagram, otherwise remove those
> sections?
> 
> Thanks!

I filed bug 953235 as a followup so this one can be closed out. Sorry for taking so long to respond!
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.