807378 - snippets.mozilla.com uses shared memcache without a KEY_PREFIX

Status: VERIFIED FIXED

(Snippets :: Service, defect)

Description

[Osmose [:osmose, :mkelly]]

The following URL should should be returning HTML for a few snippets, but isn't:

https://snippets.mozilla.com/3/Firefox/18.0/20120713134347/Darwin_Universal-gcc3/en-US/release/Darwin%2010.8.0/mozilla14/1.0/

The admin interface and all other parts of the site seem to be functioning as expected. We recently had some work done setting up a dev and staging site; perhaps something was accidentally changed on production?

For starters, I'd like to request that we clear the Zeus and memcache caches to see if they've just cached invalid data. I'd also like to confirm that the site is still getting traffic in the apache logs and if there's anything showing up in the error logs.

Comment 1

[Chris Turra [:cturra]]

:mkelly - per our discussion on irc, i am seeing errors like the following on the web heads:

[Wed Oct 31 17:33:59 2012] [error] [client 10.8.81.215] mod_wsgi (pid=8147): Exception occurred processing WSGI script '/data/www/snippets.mozilla.com/home-snippets-server/wsgi/index.wsgi'.
[Wed Oct 31 17:33:59 2012] [error] [client 10.8.81.215] IOError: failed to write data

Comment 2

[Osmose [:osmose, :mkelly]]

An IOError usually only occurs if the client disconnects before the server can finish writing a response.

How far back do these go? We get these from time to time on most of our sites and it usually isn't an issue.

Comment 3

[Jake Maul [:jakem]]

Corey and I have discussed that, and we believe that error is an artifact of the Zeus "Simple HTTP" health check, and not an actual problem... probably not relevant here.

Comment 4

[Chris Turra [:cturra]]

i have cleared the cache in zeus.

Comment 5

[Osmose [:osmose, :mkelly]]

Still getting the issue. Per discussion on IRC, we also checked to ensure the DB is populated with data.

Is there any way to get a shell to one of the webheads? I could do some more testing to see if it's an app issue with that.

Comment 6

[Jake Maul [:jakem]]

This is fixed, for now. The temp fix was to completely clear memcache.


The problem seems to be that this is a Django 1.2 app, and as such does not support a KEY_PREFIX setting. It's currently using a shared memcache cluster. Most likely some other app overwrote one or more of its keys.

The long-term solution is to move this to a newer Django version which supports KEY_PREFIX.

A medium-term solution would be to move this to a dedicated memcache cluster, or skip memcache altogether. I don't think the load would be an issue.

Comment 7

[Jake Maul [:jakem]]

Morphing this bug and dropping severity.

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified both manually (comment 0), and via our automation: http://qa-selenium.mv.mozilla.com:8080/view/Snippets/job/snippets.prod/2987/testReport/

Comment 9

[Osmose [:osmose, :mkelly]]

Seeing the issue again. Code was updated to Django 1.3.4 in d and b. Push bug to fix issue is bug 808689.

Comment 10

[Osmose [:osmose, :mkelly]]

I totally meant to replace d and b with these links: https://github.com/mozilla/home-snippets-server-lib/commit/93bff4b4aeb452106f24fe4c6d0cb7cdd683d3f1 and https://github.com/mozilla/home-snippets-server/commit/16104da50c319de69ce8226b2a489082caa26e83

Comment 11

[Osmose [:osmose, :mkelly]]

Fixed and deployed, the snippets servers should stay working for the foreseeable future.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified; looked at this with Jake and mkelly, and it looks good on prod.