[mozillians-prod] site in broken state due to failed database migration

VERIFIED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Engagement
--
blocker
VERIFIED FIXED
4 years ago
a year ago

People

(Reporter: sancus, Assigned: cturra)

Tracking

Details

Comment hidden (empty)
(Reporter)

Updated

4 years ago
Severity: normal → blocker
(Reporter)

Comment 1

4 years ago
http://genericadm.private.phx1.mozilla.com/chief/mozillians.prod/logs/2013-11-21.1385057036 is the log.

I'm not exactly sure what's happened here: users:0042_auto__chg_field_externalaccount_type should have changed the type of the 'type' column in users_externalaccount to a varchar, but it apparently didn't work, despite no errors being returned. This caused the next migration to fail, and now the site is failing when trying to deal with these values.
(Assignee)

Updated

4 years ago
Assignee: server-ops-webops → cturra
(Assignee)

Comment 2

4 years ago
while we investigate this, i have hard hatted the site.

 $ curl -I https://mozillians.org
 HTTP/1.1 302 Moved Temporarily
 Content-Type: text/html
 Date: Thu, 21 Nov 2013 18:30:53 GMT
 Location: http://hardhat.mozilla.net/
 Connection: Keep-Alive
 Content-Length: 0
(Assignee)

Comment 3

4 years ago
we ran the following to get some debug info:

 ./manage.py migrate --db-dry-run --verbosity=2 > /home/cturra/migratelog.txt


once that was reviewed and we didn't find anything obvious, we ran the following:

 ./manage.py migrate users 0045
 ./manage.py migrate users 0046
 ./manage.py migrate

... and finally, a new chief deployment.


after that, we cleared the hardhat and everything looked good again :)

$ curl -IL https://mozillians.org/en-US/
HTTP/1.1 200 OK
Server: Apache
Vary: X-Mobile,User-Agent,Cookie, Accept-Encoding
X-Backend-Server: generic102.webapp.phx1.mozilla.com
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Content-Security-Policy: script-src 'self' http://www.mozilla.org https://www.mozilla.org http://*.mozilla.net https://*.mozilla.net https://*.google-analytics.com https://login.persona.org; default-src 'self'; img-src 'self' data: http://*.mozilla.net https://*.mozilla.net *.google-analytics.com *.gravatar.com *.wp.com; style-src 'self' 'unsafe-inline' http://www.mozilla.org https://www.mozilla.org http://*.mozilla.net https://*.mozilla.net; frame-src 'self' https://login.persona.org; font-src 'self' http://*.mozilla.net https://*.mozilla.net
Strict-Transport-Security: max-age=15768000 ; includeSubDomains
Date: Thu, 21 Nov 2013 19:06:53 GMT
Expires: Thu, 21 Nov 2013 19:06:53 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Set-Cookie: anoncsrf=LJVxzonoCknFgNnOFcNgdno8gZI8EERU; expires=Thu, 21-Nov-2013 21:06:53 GMT; httponly; Max-Age=7200; Path=/; secure
X-Frame-Options: DENY
Last-Modified: Thu, 21 Nov 2013 19:06:53 GMT
X-Cache-Info: not cacheable; response specified max-age <= 0
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

4 years ago
Thanks for your help!
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.