941747 - [mozillians-prod] site in broken state due to failed database migration

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Comment 1

[Andrei Hajdukewycz [:sancus]]

http://genericadm.private.phx1.mozilla.com/chief/mozillians.prod/logs/2013-11-21.1385057036 is the log.

I'm not exactly sure what's happened here: users:0042_auto__chg_field_externalaccount_type should have changed the type of the 'type' column in users_externalaccount to a varchar, but it apparently didn't work, despite no errors being returned. This caused the next migration to fail, and now the site is failing when trying to deal with these values.

Comment 2

[Chris Turra [:cturra]]

while we investigate this, i have hard hatted the site.

 $ curl -I https://mozillians.org
 HTTP/1.1 302 Moved Temporarily
 Content-Type: text/html
 Date: Thu, 21 Nov 2013 18:30:53 GMT
 Location: http://hardhat.mozilla.net/
 Connection: Keep-Alive
 Content-Length: 0

Comment 3

[Chris Turra [:cturra]]

we ran the following to get some debug info:

 ./manage.py migrate --db-dry-run --verbosity=2 > /home/cturra/migratelog.txt


once that was reviewed and we didn't find anything obvious, we ran the following:

 ./manage.py migrate users 0045
 ./manage.py migrate users 0046
 ./manage.py migrate

... and finally, a new chief deployment.


after that, we cleared the hardhat and everything looked good again :)

$ curl -IL https://mozillians.org/en-US/
HTTP/1.1 200 OK
Server: Apache
Vary: X-Mobile,User-Agent,Cookie, Accept-Encoding
X-Backend-Server: generic102.webapp.phx1.mozilla.com
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Content-Security-Policy: script-src 'self' http://www.mozilla.org https://www.mozilla.org http://*.mozilla.net https://*.mozilla.net https://*.google-analytics.com https://login.persona.org; default-src 'self'; img-src 'self' data: http://*.mozilla.net https://*.mozilla.net *.google-analytics.com *.gravatar.com *.wp.com; style-src 'self' 'unsafe-inline' http://www.mozilla.org https://www.mozilla.org http://*.mozilla.net https://*.mozilla.net; frame-src 'self' https://login.persona.org; font-src 'self' http://*.mozilla.net https://*.mozilla.net
Strict-Transport-Security: max-age=15768000 ; includeSubDomains
Date: Thu, 21 Nov 2013 19:06:53 GMT
Expires: Thu, 21 Nov 2013 19:06:53 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Set-Cookie: anoncsrf=LJVxzonoCknFgNnOFcNgdno8gZI8EERU; expires=Thu, 21-Nov-2013 21:06:53 GMT; httponly; Max-Age=7200; Path=/; secure
X-Frame-Options: DENY
Last-Modified: Thu, 21 Nov 2013 19:06:53 GMT
X-Cache-Info: not cacheable; response specified max-age <= 0

Comment 4

[Andrei Hajdukewycz [:sancus]]

Thanks for your help!