Closed Bug 516371 Opened 15 years ago Closed 15 years ago

Machine/VM for Bugzilla HTTP API proxy

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gerv, Assigned: fox2mike)

Details

Dear Server Ops,

I am in the process of writing an HTTP RESTful API for Bugzilla. The initial implementation is a proxy, to decouple the development from the b.m.o. update schedule.

Once I've got bits of it working, I want to put it somewhere where people can try it out. My conception of that would be a Linux machine or VM somewhere physically close to the Bugzilla server, managed by IT but to which I had upload and admin privileges. But perhaps you can advise on the best way to go?

The machine would need to have the Catalyst Perl web framework installed. I'm most familiar with Ubuntu, but if you guys have standardised on something else, I can try and adapt.

The machine doesn't need any special access to the Bugzilla server or database or any other internal systems - it interacts with it only via its public HTTP interface.

Ideally, I'd be able to get access to such a machine by the end of this week.
Do let me know if all this would be possible, or whether I need to modify my ideas :-) 

Gerv
Redirecting.  

Seth - there's VM capacity in Amsterdam, none in San Jose.
Assignee: server-ops → sethb
Component: Server Operations → Community Giving
QA Contact: mrz → community-giving
mrz: I'm not quite sure why this has been sent over to Community Giving. This is an official, part-of-my-MoFo-job, shaver-wants-this-yesterday, important IT project for helping bugzilla.mozilla.org users do their jobs better. Who do I need to convince of this in order to get a machine or VM, officially supported by IT, in the rack next to the Bugzilla machines? :-)

Gerv
Matter of resource allocations.  There's a delineation of assets between Mozilla Corp, Mozilla Foundation, Mozilla (the project) and Mozilla Messaging.  

Second paragraph suggests this is for Bugzilla, not the Foundation.  As a community resource/project, I punted to Community Giving for resources.  CG can donate resources or we can get some server and bill back to the Foundation (but still have Mozilla IT support).
(In reply to comment #3)
> Second paragraph suggests this is for Bugzilla, not the Foundation.  As a
> community resource/project, I punted to Community Giving for resources.  CG can
> donate resources or we can get some server and bill back to the Foundation (but
> still have Mozilla IT support).

Not sure punting this to CG seems like the best or most efficient process.  We examine each and every request for community giving requests.  This seems to be a bigger infrastructure need that might have to happen on a more immediate basis.  I'd be happy to add it to a queue where Gerv, Shaver and I debate and document it, but I wonder if some of the IT requests that get sent over to CG should just be responded to if they are strategic and immediate.  

In this case, should IT implement the request and figure out if MoFo or MoCo funds the idea?  Based on Gerv's comments above, it seems to be something we should just do without having to go through the CG process.  Thoughts?
seth - this is for the bugzilla community or under mofo, but out of scope for moco, hence why we kicked it over.  what time frame could we expect a response from CG?
Apologies: the confusion here is being caused by me using the word "Bugzilla" in multiple senses without distinguishing what I mean.

I am in the process of writing an HTTP RESTful API for Bugzilla, implemented as a proxy. The installation of this software for which I require this machine will be "pointed at" bugzilla.mozilla.org - in other words, it's an alternative access mechanism for that particular server. This will enable bugzilla.mozilla.org data to be more easily used in tools, both web-based and desktop.

This project is happening because Mozilla project community members find the current HTML user interface of bugzilla.mozilla.org deficient in a wide variety of different ways, and we want to enable people to innovate on top of bugzilla.mozilla.org by writing their own UIs for it and tools which integrate with it. To do that, we need an easy-to-use programmatic interface. People who have already expressed an interest in doing so include murali, fligtar, Rob Arnold (qimportbz) and harthur.

I'm doing this work as part of the Mozilla Foundation's role in supporting the community, but there are Mozilla Corporation people (e.g. shaver) who are very keen to see it happen ASAP. My assertion is that this API is/will be a core Mozilla project service. 

If the question is about who pays for the server, then that decision is above my pay grade; CCing Mark :-)

Gerv
As Gerv says, this is something MoFo is doing as a part of our broader Mozilla community support efforts. 

He should be ready with something experimental for people to bang on next week. However, he needs a sandbox to host on in order for this to happen. 

Is a VM on the Labs sandbox an option to get this quickly? (copied Zandr here)
this is hard as moco (labs included) is pressed for capacity currently.  I think the best option is for mofo to buy a machine and vmware license - then you guys can add/delete vm's as you choose, we'll still host the machine and maintain the base vmware os for you.  matthew already has some hardware on order which should be up in a short time frame (i.e. a week or less).  assuming this is ok from a cost perspective for mofo, we'll go that route...ok by everyone?
How much would this cost? Can MoCo IT do the basic setup and admin?
Gerv - can you give me an idea of disk/memory requirements?
Assignee: sethb → mrz
Component: Community Giving → Server Operations
QA Contact: community-giving → mrz
I'm not sure how to best estimate this, so guidance would be appreciated.

Disk: OS install with webserver and various Perl modules; it's a proxy, so it's not storing any data. Base OS plus 500MB should be plenty, unless there are performance issues with having the virtual disk nearly full.

Memory: Apache + Catalyst. No database. The Apache instance on my box is currently using 233m of virtual memory, but I'm not doing anything with it. 

Gerv
Gerv, sorry - I thought I had punted this over to Shyam but I didn't!
Assignee: mrz → shyam
Would I look stupid if I whispered "EC2" into the wind, so that we can scale the node easily, or dup it, and not use up our scarce production capacity?  This isn't a production service, just a place to host some work in progress so that people can bang on them.
Not sure if this was discussed elsewhere, but the XMLRPC API for Bugzilla is going to take a huge jump in capability when we upgrade to Bugzilla 3.4 (which is happening in the next couple weeks).  That should enable people to do lots of innovating on top of Bugzilla, shouldn't it?  Or is this just a REST<->XMLRPC converter to lessen the number of libraries you have to run to make it work? :)
@shaver we have so much capacity in Amsterdam.
Dave: the XML-RPC API doesn't have key features like being able to alter bugs or add attachments, it doesn't provide all the data for each bug, the search is not nearly as powerful and (as you say) is harder to use from web mashups. The HTTP API is built in part on top of the XML-RPC one, but falls back to other methods such as our old XML interface, CSV buglists or HTTP page scraping when absolutely necessary.
See
https://wiki.mozilla.org/User:Gerv/BugzillaAPIDesign
for a full list of capabilities.

The plan is that eventually the HTTP API will be reimplemented on top of Bugzilla itself, but doing things this way allows for a rapid development and fix cycle, but still using data that people are actually interested in.

Gerv
Alright, dn-bugzilla-api01.nl.mozilla.com is up with 2 GB of RAM and has Gerv's keys on it. The flip side though is that right now it's only reachable through the admin box in nl, I'm probably going to need mrz's help in fixing that and putting it out on the internet and you should be good to go.
Oh and it's running Ubuntu 9.04. Had to do a 8.04 -> 8.10 -> 9.04 though.
Thanks, Shyam :-)

Any chance of a slightly nicer DNS alias - say bugzilla-api.mozilla.org? (I don't want to share bugzilla.mozilla.org cookies so api.bugzilla.mozilla.org would not be right).

Once we've got the alias, it would be good to get an HTTPS cert for it. At the monent, the API requires usernames and passwords to be passed on URLs, so HTTPS is important. Could IT please arrange the purchase for me? I would hope the CAs wouldn't let me get one without proof of domain control! ;-) A cheap DV cert is fine.

Gerv
(In reply to comment #19)
> Any chance of a slightly nicer DNS alias - say bugzilla-api.mozilla.org? (I
> don't want to share bugzilla.mozilla.org cookies so api.bugzilla.mozilla.org
> would not be right).

You wouldn't share cookies by using api.bugzilla.mozilla.org. We currently use *.bugzilla.mozilla.org to handle attachments, which aren't allowed access to Bugzilla cookies (because bugzilla.mozilla.org only sets a host-based [and not domain-based] cookie). So, api.bugzilla.mozilla.org would actually work fine as a hostname.
given this is dev, I really want this to not have a hostname that implies anything production.  if anything - api-dev.mozilla.org or something to that would be better.
And, can you deal with a Mozilla-signed SSL cert until there's a finalized hostname?
dev-api.bugzilla.mozilla.org or bugzilla-api-dev.mozilla.org would be fine by me.

A Mozilla-signed (as opposed to Firefox-default-trusted) SSL cert is an additional barrier for clients. Say someone makes a test mashup of Tinderbox and Bugzilla data, anyone in the community who wants to use it has to import the Mozilla root cert. Which isn't ideal. So I'd prefer a real cert. It won't even cost you any money:
https://www.godaddy.com/gdshop/ssl/ssl_opensource.asp
:-)

Gerv
api-dev.bugzilla.mozilla.org / 63.245.212.105

Gerv, you should have your keys on this box already and I'll let you handle dealing with GoDaddy's chained certificate.  Let me know if you need help here.

For production, we use GeoTrust and those aren't free.  For dev/staging we use Mozilla signed certs.
Solve the "cert installation is a barrier for mashup" problem when it's a problem.  The real barrier right now is that *nobody* can use it, and you should focus on getting it up at all before worrying that it's "not ideal".  People are literally ready day-by-day to start using the API proxy to give you feedback and improve our tools.  They will *cheerfully* install any Mozilla CA root required, I assure you.
mrz: thank you :-)

shaver: OK. It's deploying next week. See 
https://wiki.mozilla.org/User:Gerv/Status:2009-09-25
for current status.

Gerv
You have the VM, calling this closed.  We'll use a new bug when you're done with dev and we want to roll this to prod.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
I think the DNS may be typoed.

gerv@kitten:/usr/src/bugzilla-3.4$ dig +short api-dev.bugzilla.mozilla.org
63.245.121.105
gerv@kitten:/usr/src/bugzilla-3.4$ 

Compare comment 24, which says 63.245.212.105. 

There's no reply from 121.105. 212.105 is accepting ssh connections, although I get asked for a password (which I'd expect not to if my keys were on there). I'm trying to log in as "gerv"; is that my username?

My public key fingerprint is:
DSA 1024 ad:41:89:7a:3b:46:e4:19:6c:9d:bc:d1:96:67:d4:47 /home/gerv/.ssh/id_dsa.pub

Gerv
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Try logging in as root and I've fixed DNS.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.