545642 - Test Bouncer/Tuxedo REST API

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Fred Wenzel [:wenzel]]

Over in bug 543452, IT set up a staging instance of Bouncer/Tuxedo at: https://tuxedo.stage.mozilla.com/ .

It runs on a copy of production data, so you can log in with your known credentials.

When logged in, you can find the API docs in the menu, which should give you an overview of the available commands. Most commands need HTTP basic auth.

RelEng: Please try out these commands and let me know if they do what you want them to, or if anything needs changed or is missing.

Comment 1

[Rail Aliiev [:rail]]

Fred,

First of all, thanks for the new Bouncer admin interface and API. 

My first 2 notes after 5 minutes playing with Tuxedo API.

1. Uptake API. The response XML uses capitalized product naming and case insensitive. 
https://tuxedo.stage.mozilla.com/api/uptake/?product=firefox-3.6rc1-complete returns an XML with <product>Firefox-3.6rc1-Complete</product>
I suspect legacy stuff, but it looks inconsistent a bit and will require extra string conversion steps for API clients.

2. It would be great to have an error code besides an error text, especially for cases when you may get more than one type of error (db error, format error, already exists error, etc).

To be continued... :)

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to comment #1)
> 1. Uptake API. The response XML uses capitalized product naming and case
> insensitive. 
> https://tuxedo.stage.mozilla.com/api/uptake/?product=firefox-3.6rc1-complete
> returns an XML with <product>Firefox-3.6rc1-Complete</product>
> I suspect legacy stuff, but it looks inconsistent a bit and will require extra
> string conversion steps for API clients.

FWIW, 'Firefox-3.6rc1-Complete' is that way the product strings are formatted in bouncer. The naming is standardized for "normal" releases (i.e. releases involving a product with a complete and partial update), so a conversion routine will likely suffice. 

More exotic bouncer entries can always be made by hand with the understanding that they are infrequent.

Comment 3

[Fred Wenzel [:wenzel]]

(In reply to comment #1)
> 1. Uptake API. The response XML uses capitalized product naming and case
> insensitive. 

I am not sure what you're asking for? Do you want the product name to be an exact match, and only if you specify "fuzzy=True" it's case-insensitive? That can be arranged, so it'd be consistent with the other API commands.

(In reply to comment #2)
> FWIW, 'Firefox-3.6rc1-Complete' is that way the product strings are formatted
> in bouncer. The naming is standardized for "normal" releases (i.e. releases
> involving a product with a complete and partial update), so a conversion
> routine will likely suffice. 

Coop is right. I am not altering the product name in any way in the XML return, and I am not planning on it.

(In reply to comment #1)
> 2. It would be great to have an error code besides an error text, especially
> for cases when you may get more than one type of error (db error, format error,
> already exists error, etc).

Okay I can do that. I'd have to make up some error numbers because Python doesn't have any, but that shouldn't be too hard for the errors that I distinguish.

Comment 4

[Rail Aliiev [:rail]]

(In reply to comment #3)
> I am not sure what you're asking for?

Just thinking out loud. :)

> Coop is right. I am not altering the product name in any way in the XML return,
> and I am not planning on it.

Ah, in this case this there is nothing to do. Adding "fuzzy=True" is optional, but may be useful for other projects.

> Okay I can do that. I'd have to make up some error numbers because Python
> doesn't have any, but that shouldn't be too hard for the errors that I
> distinguish.

Thanks.

Comment 5

[Rail Aliiev [:rail]]

Fred,

I've just started testing this API, but has stuck with POST requests (adding/deleting products/locations)...

Here is a simple test script where I'm trying to add a product and it fails with 400 BAD REQUEST (product_show works fine). 
http://python.pastebin.mozilla.org/705017

I also tried to use a simple HTML page with a POST form with the same result...

Do you have any idea regarding this?

Comment 6

[Fred Wenzel [:wenzel]]

I could reproduce the issue with your script and it took me a long time to figure out how simple it is, sigh...

If you use /api/product_add without a trailing slash, Django will send you a 301 redirect pointing to /api/product_add/ instead. According to the RFC, 301 redirects must not automatically be followed with POST requests, so Django makes a GET request from it and tries again. That fails because the API expects POST.

Short-term fix: Use trailing slashes.

Long-term fix: I'll make sure the API works both ways.

Thanks for finding this!

Comment 7

[Rail Aliiev [:rail]]

(In reply to comment #6)
> Long-term fix: I'll make sure the API works both ways.

Ah... Stupid Django. :)

Anyways, here are some notes.

1. If you try to add a product twice you don't get an error, while adding a location twice raises an error (the same 400, Bad Request).

2. When you try to add a location with an unknown os you get 400 too.

3. I couldn't add a location for lang=ja-JP-mac

Comment 8

[Fred Wenzel [:wenzel]]

(In reply to comment #7)
> (In reply to comment #6)
> > Long-term fix: I'll make sure the API works both ways.
> 
> Ah... Stupid Django. :)

FWIW, turning a POST into a GET when a 301 redirect is encountered is apparently deliberate violation of the standard, taken by most modern web browsers, and also taken by urllib2. Still, not helpful in this case! :)

> Anyways, here are some notes.
> 
> 1. If you try to add a product twice you don't get an error, while adding a
> location twice raises an error (the same 400, Bad Request).

Ah, that shouldn't be inconsistent. Which do you prefer, silently ignoring the duplicate (with the reasoning, you got what you wanted, it was just already there), or throwing an error every time (with an error number indicating "record already exists")?

> 2. When you try to add a location with an unknown os you get 400 too.

Does it give you a useful error message? You need to wrap your urllib2.open() command into a try/except block and read the exception that is thrown with the error 400.

Note that error 400 makes sense for a failed API request. We shouldn't serve code 200 if the command didn't succeed.

> 3. I couldn't add a location for lang=ja-JP-mac

That doesn't seem to be part of our locale database:
http://viewvc.svn.mozilla.org/vc/libs/product-details/localeDetails.class.php?view=markup

Is there anything exceptional about this locale, any reason it's not in the list?

Comment 9

[Wil Clouser [:clouserw]]

(In reply to comment #8)
> > 3. I couldn't add a location for lang=ja-JP-mac
> 
> That doesn't seem to be part of our locale database:
> http://viewvc.svn.mozilla.org/vc/libs/product-details/localeDetails.class.php?view=markup
> 
> Is there anything exceptional about this locale, any reason it's not in the
> list?

It's something we made up because macs didn't have the fonts they needed.  It's not a real locale, so it's not in the list.  We redirect it to ja on each site.  ja-jp-mac is only important as far as the builds go, not the webdev stuff.

Comment 10

[Rail Aliiev [:rail]]

(In reply to comment #8)
> > 1. If you try to add a product twice you don't get an error, while adding a
> > location twice raises an error (the same 400, Bad Request).
> 
> Ah, that shouldn't be inconsistent. Which do you prefer, silently ignoring the
> duplicate (with the reasoning, you got what you wanted, it was just already
> there), or throwing an error every time (with an error number indicating
> "record already exists")?

Throwing an error would be fine. Try/except works as usual, no need to parse the resulting XML.
 
> > 2. When you try to add a location with an unknown os you get 400 too.
> 
> Does it give you a useful error message? You need to wrap your urllib2.open()
> command into a try/except block and read the exception that is thrown with the
> error 400.
> 
> Note that error 400 makes sense for a failed API request. We shouldn't serve
> code 200 if the command didn't succeed.

Sounds good.

> > 3. I couldn't add a location for lang=ja-JP-mac
> 
> That doesn't seem to be part of our locale database:
> http://viewvc.svn.mozilla.org/vc/libs/product-details/localeDetails.class.php?view=markup
> 
> Is there anything exceptional about this locale, any reason it's not in the
> list?

See http://hg.mozilla.org/releases/mozilla-1.9.2/raw-file/tip/browser/locales/shipped-locales for locale/platform mappings. ja-JP-mac is also used in the current Bouncer setup.

Comment 11

[Fred Wenzel [:wenzel]]

Quick heads-up, IT just updated tuxedo staging, so the bugs known so far should be gone. I've yet to add ja-JP-mac, though.

Comment 12

[Rail Aliiev [:rail]]

Great to hear this.

I run the following scenario to test the API.

1. Delete old products (product_delete) added by previous tests.
2. Add product for full version, complete MAR and partial MAR. (product_add)
3. Add locations for all languages and platforms from shipped-locales for full, complete and partial (location_add)
4. Get locations (location_show) for the added products, check if they exist on mirrors (HTTP 200)
5. Manually test *_show API URLs

These tests should be enough to see if Tuxedo is ready for releng automation.

The only stopper is ja-JP-mac locale.

Comment 13

[Fred Wenzel [:wenzel]]

Attachment: Patch, rev. 1

Wil, how does that look?

Comment 14

[Wil Clouser [:clouserw]]

Comment on attachment 429789 [details] [diff] [review]
Patch, rev. 1

if you say so

Comment 15

[Fred Wenzel [:wenzel]]

Landed on r63574. Fixed on tuxedo, and pushed. Once IT updates staging again (bug 549677), you can test with ja-JP-mac.

Comment 16

[Rail Aliiev [:rail]]

ja-JP-mac worked fine.

Fred,
Thanks a lot for your prompt changes. :)

Comment 17

[Kohei Yoshino]

Just FYI: Japanese version of Mac OS X uses various different terminologies from Windows, like Applications vs Programs. So we have the ja-JP-mac locale aside from the ja locale which combines ja-JP-winxp and ja-JP-unix style wordings.