635246 - Configure __heartbeat__ in all python products

Status: VERIFIED FIXED

(Cloud Services :: Server: Core, defect)

Description

[Tarek Ziadé (:tarek)]

Now that we have __heartbeat__ and __debug__ available (in apps that use the base app), we should activate them in the configuration of the sync servers, and use them.

See: http://sync.ziade.org/doc/configuration.html#global

Apps that still need to include them:

- keyexchange
- identity
- account portal

This change is driven by bug 635245

Comment 1

[:Atoll]

we won't be enabling __debug__.

the configuration indicates that heartbeat defaults to "__heartbeat__", so does that mean it should be enabled by default evyerwhere already?

Comment 2

[Toby Elliott [:telliott]]

My reading of the code suggests that __heartbeat__ should simply be on and not require any configuration to exist

Comment 3

[:Atoll]

I checked production and got an entertaining variety of responses:

$ { for i in web ap reg sreg jpake; do echo -ne "$i\t"; curl -I http://wp-${i}01/__heartbeat__ 2>/dev/null | head -n1; done; echo; }
web   HTTP/1.1 400 Bad Request
ap    HTTP/1.1 405 Method Not Allowed
reg   HTTP/1.1 400 Bad Request
sreg  HTTP/1.1 400 Bad Request
jpake HTTP/1.1 404 Not Found

So it looks like there's some work left to do.

Comment 4

[:Atoll]

It's working in one server in staging, will model a set of changes after that:

$ { for i in sync1.web web1.reg web1.sreg ap1.reg setup1.web; do curl -I http://${i}/__heartbeat__ 2> /dev/null | head -n1; done; echo; }
HTTP/1.1 200 OK
HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
HTTP/1.1 405 Method Not Allowed
HTTP/1.1 404 Not Found

Comment 5

[:Atoll]

The only one returning OK was running PHP sync, rather than Python sync.

Pushed [global] heartbeat_page = __heartbeat__ to all staging python servers, to no avail.

Comment 6

[:Atoll]

Filed bug 670630 to implement support for HEAD /__heartbeat__, which is why I've been seeing errors.  Fixed curl test command line to use GET instead of HEAD:

{ for i in sync1.web web1.reg web1.sreg ap1.reg setup1.web; do curl -D - http://${i}.mtv1.stage.svc.mozilla.com/__heartbeat__ 2> /dev/null | head -n1; done; echo; }
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found

It appears that ap/setup did not honor the config option, probably due to the age of their python26-services dependency:

{ for i in sync1.web web1.reg web1.sreg ap1.reg setup1.web; do ssh -A ${i}.mtv1.stage.svc.mozilla.com rpm -qa \| grep python26-services; done; echo; }
python26-services-2.3-1
python26-services-2.4-3
python26-services-2.4-3
python26-services-1.7-2
python26-services-1.0-1

I checked and the code for heartbeat has been present since 0.2-11, so that's not it.  I tried upgrading setup1.web from SERVER_CORE rpm-1.0-1 to rpm-2.4-3, but that didn't work.

Comment 7

[:Atoll]

Tarek suggests that a code change is required for ap/setup and is looking into it further, bugs to follow.

Comment 9

[Toby Elliott [:telliott]]

(In reply to comment #7)
> Tarek suggests that a code change is required for ap/setup and is looking
> into it further, bugs to follow.

Yes, AP is not a service and thus isn't using our services baseapp.

Comment 10

[Tarek Ziadé (:tarek)]

baseapp is not meant only for services. 

It's for any Python application we run in a server, and provides basic features like __heartbeat__, Ops require to operate an application.

If I look at AP's wsgiapp at http://hg.mozilla.org/services/account-portal/file/ee552c3ba4aa/accountportal/wsgiapp.py, there are in fact several things that are duplicates of baseapp (that's historic I guess, like for keyex)

Comment 11

[:Atoll]

No action from dev since July, and no action that Ops is capable of taking to deploy this.  In review, this is actually a dev metabug, not an actionable Ops item.  Reassigning this dev metabug to Services: General and removing myself from CC:.

Ops is happy to accept individual deployment request bugs for { ap, setup } __heartbeat__ someday, if and when they are revised to support such.

Comment 12

[Toby Elliott [:telliott]]

This is part of cornice, so we should be getting it for free in future products.

Comment 13

[James Bonacci [:jbonacci]]

OK. Marking this as Verified.