Closed Bug 684363 Opened 13 years ago Closed 13 years ago

Push out updates to basket web service

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jlong, Assigned: nmaul)

References

Details

Basket is a web service running at http://basket.mozilla.org. It provides email functionality for Firefox Home. We just extended it to handle newsletter subscriptions from mozilla.org. Internally it works with Responsys to handle all the data, so it's a middle-man service.

We need our code to be pushed live. Let's schedule this for 9/7 if possible.
Assignee: nobody → server-ops
Component: www.mozilla.org/firefox → Server Operations
Product: Websites → mozilla.org
QA Contact: www-mozilla-com → mrz
Version: unspecified → other
The code lives at https://github.com/mozilla/basket. We might create a production branch; any input from IT here would be good for one-off services like this. It's a Python web service.

The code is almost ready and will be done early next week. Is there any staging environment available?
Assignee: server-ops → nmaul
Comment 0 sounds like this already exists and you're just extending the code base... but basket.mozilla.org doesn't resolve to anything. Is this a new service or is that URL wrong? I know I've heard of "basket" before...

I found a basket.mozilla.com python app on the generic cluster. It appears to be in git, but not at that location. It lives at: git://github.com/abuchanan/basket.git

It looks like the same code base to me, though, at a glance.

As far as I know, there is no staging site for this as of yet. However, we may be able to migrate this site to the new generic cluster in Phoenix, which has dev/stage/prod environments.

How quickly would you need a stage environment? I see the original intent was to push to prod on 9/07, but if it's not too urgent I'd like a bit more time to get a dev/stage environment set up for this.
To clarify: if you want to push straight to prod and forgo stage this time, we can push tomorrow with no problems. :)
(In reply to Jake Maul [:jakem] from comment #2)
> I found a basket.mozilla.com python app on the generic cluster. It appears
> to be in git, but not at that location. It lives at:
> git://github.com/abuchanan/basket.git

Oh, you're right, it's com not org. Also, we forked that repo into our own in the Mozilla group, so it needs to be based off that now.

> How quickly would you need a stage environment? I see the original intent
> was to push to prod on 9/07, but if it's not too urgent I'd like a bit more
> time to get a dev/stage environment set up for this.

How long will that take? I'm not sure even how we'll work with everyone to test this, but we can wait a little bit because the PHP frontend is still going through security review and other stuff.

(In reply to Jake Maul [:jakem] from comment #3)
> To clarify: if you want to push straight to prod and forgo stage this time,
> we can push tomorrow with no problems. :)

Thanks! It's not urgent, so let's not do it yet because I'm scared of breaking Firefox Home. I'll ping someone to see if they can help us test it when we get it on a staging env.
Blocks: 663528
Hey Jake, I know things are crazy, but how long should it take to move it over?

The impact of our changes is minimal, so I don't think it's a blocker to get it moved to the dev/staging setup.
We should be able to do this on Monday. I don't want to do it right now because it's late in the day on a Friday. Apart from that I'm also mildly concerned about it because I don't know anything about this service, and our documentation is basically non-existent.

Doing some research, it should be possible to edit .git/config to point to the new origin location. It looks like this would have to be done for the base dir and the vendor/ dir, which currently uses "git://github.com/abuchanan/basket-lib.git" (presumably that's been ported to github.com/mozilla as well?). It looks like vendor/ has some submodules... probably a sync/update cycle would be good for them.

The usual update procedure for most of our django sites looks something like this:

git pull origin <branch>
git submodule sync
git submodule update --init
cd vendor
git pull origin <branch>
git submodule sync
git submodule update --init
cd ..

Some teams prefer to set up separate branches for each push, so instead of a pull it's "git fetch" and "git checkout".

Let me know what seems reasonable and we can do a trial run of the update somewhere, and I can send you the output to make sure it seems sane.
(In reply to Jake Maul [:jakem] from comment #6)
> 
> Doing some research, it should be possible to edit .git/config to point to
> the new origin location. It looks like this would have to be done for the
> base dir and the vendor/ dir, which currently uses
> "git://github.com/abuchanan/basket-lib.git" (presumably that's been ported
> to github.com/mozilla as well?). It looks like vendor/ has some
> submodules... probably a sync/update cycle would be good for them.

I haven't even been using a vendor library. But we should make one since that's standard. I just forked it into:

https://github.com/mozilla/basket-lib

And I will update it with our dependencies.

Once I do that, do I just need to ping you to start the move?
Yeah, that'll work.
Ok, I just updated the vendor library, and added it as a submodule to the main project.

Let me know how else I can help. It'd be great to have this moved over. I can get the Firefox Home people to test the staging instance to make sure everything works before you move the live site.
For Firefox Home I get the following error when I switch to the basket test server:

POST /subscriptions/subscribe/ HTTP/1.1
Host: jlongster.com:7001
User-Agent: FirefoxHome/1.1.1 CFNetwork/485.13.9 Darwin/11.0.1
Content-Type: multipart/form-data; boundary=FFH2011-09-19 22:31:30 +0000HFF
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Length: 349
Connection: keep-alive

--FFH2011-09-19 22:31:30 +0000HFF
Content-Disposition: form-data; name="email"

sarentz@mozilla.com
--FFH2011-09-19 22:31:30 +0000HFF
Content-Disposition: form-data; name="locale"

en_US
--FFH2011-09-19 22:31:30 +0000HFF
Content-Disposition: form-data; name="campaign"

firefox-home-instructions
--FFH2011-09-19 22:31:30 +0000HFF--




HTTP/1.0 500 INTERNAL SERVER ERROR
Date: Mon, 19 Sep 2011 22:31:05 GMT
Server: WSGIServer/0.1 Python/2.7.1
Vary: Authorization
Content-Type: text/html; charset=utf-8

Piston/0.2.2 (Django 1.4 pre-alpha) crash report:

Traceback (most recent call last):

  File "/sites/basket/apps/subscriptions/handlers.py", line 24, in create
    email = request.data.get('email', '')

AttributeError: 'WSGIRequest' object has no attribute 'data'
(In reply to Stefan Arentz [:st3fan] from comment #10)
> Traceback (most recent call last):
> 
>   File "/sites/basket/apps/subscriptions/handlers.py", line 24, in create
>     email = request.data.get('email', '')
> 
> AttributeError: 'WSGIRequest' object has no attribute 'data'

eh, shouldn't that be request.POST.get?
It looks like it uses Piston, so maybe the library got upgraded or something.
Depends on: 687890
From a quick look at the piston docs, it might be that request.data is not set from raw POST data, however it will be set if you submit JSON data to it:

<https://bitbucket.org/jespern/django-piston/wiki/Documentation#!receiving-data>
No longer depends on: 687890
(In reply to Fred Wenzel [:wenzel] from comment #13)
> From a quick look at the piston docs, it might be that request.data is not
> set from raw POST data, however it will be set if you submit JSON data to it:
> 
> <https://bitbucket.org/jespern/django-piston/wiki/Documentation#!receiving-
> data>

Good call! Indeed, that looks like what it is.

Stefan, are you testing the actual app? This works for me:

curl -vi -H'Content-Type: application/json' -d'{"email": "test@example.com", "campaign": "firefox-home-instructions"}' http://jlongster.com:7001/subscriptions/subscribe/
Depends on: 687890
No longer depends on: 687890
Note for Jake:

I upgraded a bunch of packages in the vendor library and removed the prod requirements file. The vendor library should be stable now.
I've updated the vendor library on my test deployment: http://etherpad.mozilla.com:9000/KufR5eh4S1

It seems that the generic procedure in comment 6 works properly. I'm going to try quickly creating a "basket-dev.allizom.org" dev site, which will give me some more confidence in actually deploying this app successfully, as well as being a proper dev environment. If that works, it should be very straightforward to make "basket.allizom.org" as a staging environment... but I don't intend to block this code deployment that long.

Stefan, let us know if comment 14 helps you any, and if the app looks good at the jlongster.com:7001 address. I'll update this bug if/when basket-dev.allizom.org is ready to be used.
Looks right to me. Thanks Jake! We'll want to get a dev/stage env at some point, so thanks for working on that.

If we're posed to push this out this week, we'll probably go ahead and do it even if the staging env isn't up yet. Not ideal, but the email pref project is stagnating. I'll let you know what the schedule looks like.
Stefan, I hear you're on vacation. When do you get back? :)
Still getting a 500 Internal Server Error:




POST /subscriptions/subscribe/ HTTP/1.1
Host: jlongster.com:7001
User-Agent: FirefoxHome/1.1.1 CFNetwork/485.13.9 Darwin/11.0.1
Content-Type: multipart/form-data; boundary=FFH2011-09-27 14:20:43 +0000HFF
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Length: 346
Connection: keep-alive

--FFH2011-09-27 14:20:43 +0000HFF
Content-Disposition: form-data; name="email"

stefan@arentz.ca
--FFH2011-09-27 14:20:43 +0000HFF
Content-Disposition: form-data; name="locale"

en_US
--FFH2011-09-27 14:20:43 +0000HFF
Content-Disposition: form-data; name="campaign"

firefox-home-instructions
--FFH2011-09-27 14:20:43 +0000HFF--





HTTP/1.0 500 INTERNAL SERVER ERROR
Date: Tue, 27 Sep 2011 14:20:08 GMT
Server: WSGIServer/0.1 Python/2.7.1
Vary: Authorization
Content-Type: text/html; charset=utf-8

Piston/0.2.2 (Django 1.4 pre-alpha) crash report:

Traceback (most recent call last):

  File "/sites/basket/apps/subscriptions/handlers.py", line 24, in create
    email = request.data.get('email', '')

AttributeError: 'WSGIRequest' object has no attribute 'data'
See comment 14, you need the Content-Type to be 'application/json' and to post a json string instead of form encoded data.
Just talked to Stefan, ignore comment 14, the app is posting form encoded data so I'm looking into why it's breaking.
Jake, it looks like we need a staging server for security to review it (https://bugzilla.mozilla.org/show_bug.cgi?id=654777#c12). Is that possible?
Just tested with Firefox Home and I got a 200 OK back. I do not know if my info was correctly put in the basket database of course. I signed up with my stefan at arentz.ca address.
Jake, how's the staging site coming along?
http://basket-dev.allizom.org/ is up... it's using the generic VIP so there's no SSL... don't know offhand if basket uses that, but if so we can fix it later.

I did a somewhat quick job on the Apache config as I'm a bit backed up right now, but let me know how it goes. The index page throws a 500 for me right now, and I'm trying to track that down... not sure if that's expected behavior given that this isn't a normal "site".
Ah, there appears to be a database connectivity issue... haven't ported that over yet. I will open a separate bug to get that set up. Can dev and staging share a database or should they be separate?
(In reply to Jake Maul [:jakem] from comment #26)
> Ah, there appears to be a database connectivity issue... haven't ported that
> over yet. I will open a separate bug to get that set up. Can dev and staging
> share a database or should they be separate?

If it makes it easier, yes they can. This is a tiny site that I don't think even needs two separate dev sites (dev/stage) and the separation won't be used much.

Thanks for the work, I know your busy!
I believe the dev site should be working now, at least more than it was yesterday. The database stuff is done.

Please let me know how this site works for you. If someone could post a suitable curl or netcat test, I could test it myself as well. Thanks!

http://basket-dev.allizom.org/

Note this is technically the 'dev' site, not stage. Stage will be up soon as well... The difference being that dev will auto-update every 15min or so, while stage will function as a "test deployment" for prod.
Awesome! It does indeed work.

I need to add some debugging code for the security review. Can you update master, or turn on 15-min updating? It might be easier to do the former for now, but I hate to bug you with these kinds of requests.

We should be good for a while once that's done. Thanks a lot Jake!
Auto-updating is configured. Kinda surprised it works at all. :)

Let me know what branch/tag to use for staging, and I'll get that set up too. If nothing like that is configured yet we can just make it the same as dev, but manually-updating.
Thanks for setting up auto-updating!

I get a 500 server error when POSTing to the service:

curl -v -demail=test@example.com -dnewsletters=mozilla-and-you http://basket-dev.allizom.org/news/subscribe/
               
* About to connect() to basket-dev.allizom.org port 80 (#0)                    
*   Trying 63.245.217.82... connected
* Connected to basket-dev.allizom.org (63.245.217.82) port 80 (#0)
> POST /news/subscribe/ HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: basket-dev.allizom.org
> Accept: */*
> Content-Length: 50
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 500 Internal Server Error
< Date: Mon, 03 Oct 2011 19:24:52 GMT
< Server: Apache
< X-Backend-Server: node203
< Content-Length: 541
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 webmaster@mozilla.com and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
* Closing connection #0
Jake: Do you want to add jlongster and me to the ADMINS = () tuple in settings_local? This way we'll get error emails when they occur.
This is completed... you should have a 500 error in your email now, I think.
Got it, thanks! suds was not added to the vendor pth file, and I had it installed locally in my venv. Should be fixed. The auto-update should get this and I'll test it again soon.
http://basket-dev.allizom.org/ is now giving "Service Unavailable".
Ok, site is back up, but with a new 500 error about a database table not existing.

DatabaseError: (1146, "Table 'basket_dev_allizom_org.news_subscriber' doesn't exist")

Can you make sure the "./manage.py syncdb" was run?
I don't know how to verify if that was or was not run... it's not part of the update script, and hasn't been previously mentioned to me. When should that be run?


By hand just now:
[root@genericadm.private.phx1 basket]# python manage.py syncdb
Creating tables ...
Creating table news_subscriber
Installing custom SQL ...
Installing indexes ...
No fixtures found.
[root@genericadm.private.phx1 basket]#
Thanks for your help with this so far.  Would it be possible to get an estimate on when we expect these issues might be resolved?  We're trying to determine a potential go live date for the pref center.  Thanks again for your work.
Hey there -- realize everyone's working hard to get things live. Please do give us an ETA that we can work from. Thx.
I'm sorry, I won't be able to provide an accurate ETA. For my part, I believe this is not blocking on IT at the moment. I believe the dev site (basket-dev.allizom.org) to be working, but James would have to confirm that.

The main sticking point that I'm aware of at this point in time is Infrasec's approval... mcoates would be able to give a better time estimate on that. This was (at least in part) blocked by getting the dev site up.

I can say that once I get the green light to deploy (from infrasec and James, and QA if they're involved), it should only be a matter of a few minutes of work.

I do intend to get a separate 'staging' environment up, but we're not considering that a blocker for release... just a "nice to have". The goal with this is to make sure we can do the rollout without significant breakage.


The security review is Bug 654777. I've linked it here as a dependent bug.
Jane and Winston, we are fixing up the dev site and will move onto the sec review very soon. From there, we can being to roll everything out like Jake said.

It is difficult to give an ETA for this kind of project that is dependent on a lot of things. I would estimate a week or two as that's what I've been told for a security review.

Jake, most things are done on your end, thanks for the work. I'm still looking into an issue on the basket dev site (this one is really vague).
Thanks James and Jake. If you have a closer idea of ETA that would be super. We'll make out plans based on 2 weeks out from now.
Jake, the current error it's throwing is vague (but it's not an IT issue I don't think). I can't reproduce it locally, so it's annoying. I hate to continue bugging you about this.

There's one last thing I forgot, but I don't think it will fix the problem. In the local settings, can you set the responsys variables as follows:

RESPONSYS_USER = (see mana page)
RESPONSYS_PASS = (see mana page)
RESPONSYS_FOLDER = '!MasterData'
RESPONSYS_LIST = 'CONTACTS_LIST'

The mana page for credentials is here: https://mana.mozilla.org/wiki/display/websites/Responsys+configuration
(In reply to Jake Maul [:jakem] from comment #37)
> I don't know how to verify if that was or was not run... it's not part of
> the update script, and hasn't been previously mentioned to me. When should
> that be run?

I haven't been involved in rolling out Python sites before, but syncdb needs to be run to create the table for my new app. What's the standard process for that? It's not a migration. So run syncdb once when we roll this to the production server (at least just for my app).
Hey Jake, hate to keep bugging you about this, but if you could apply the tweak in comment 43 I will test the service to see if it's working then.
Comment 43 completed.
(In reply to Jake Maul [:jakem] from comment #46)
> Comment 43 completed.

Thanks! Unfortunately, it's still throwing on an error. I think at this point it would be good for me just to get SSH access (with root?). Is that possible? I'd be able to debug it much easier that way.
Jake, what do you think is the timeframe to get SSH access?
Depends on: 694094
Depends on: 694118
We fixed the dev server running at basket-dev.allizom.org and have tested everything. It's good to go.

Jake, when can we push this live? The sooner the better. Just need to update http://basket.mozilla.org. vendor should have a few updates but not much.
This is pushed out. For the record, this was a coordinated effort, in #www.

https://etherpad.mozilla.org/CjeV0IyxuQ
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Stefan tested that the Firefox Home portion still works and I tested the Responsys part and it works too. Woot!
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.