Last Comment Bug 523428 - [AMO] Update on 10/20
: [AMO] Update on 10/20
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
-- normal (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Depends on: 523539
  Show dependency treegraph
Reported: 2009-10-20 12:22 PDT by Wil Clouser [:clouserw]
Modified: 2013-10-09 10:29 PDT (History)
8 users (show)
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)

5.2.sql (8.99 KB, text/plain)
2009-10-20 12:40 PDT, Wil Clouser [:clouserw]
no flags Details
First set of queries. (8.28 KB, text/plain)
2009-10-20 21:30 PDT, Jeremy Orem [:oremj]
no flags Details
Second set of queries. (5.11 KB, text/plain)
2009-10-20 21:55 PDT, Jeremy Orem [:oremj]
no flags Details

Description User image Wil Clouser [:clouserw] 2009-10-20 12:22:05 PDT
This is a production AMO push for 5.2.  Steps are below, please paste the
output of all commands into this bug.  I expect there to be downtime so please
use a maintenance page.  Steps to follow:

1) There are 4 new defines in config.php.default starting with PERSONAS_.  They need to move to config.php and have values for the production personas database.  Please ensure the credentials you use have read access to the database and there are no other problems (eg. firewall) accessing the database from the AMO boxes.

2) Run the attached 5.2 SQL.  There is a query in here that might fail, so we need the output.  If it does fail, we'll write some SQL to fix it. (I was waiting to attach it, but it's too slow and I want to file this bug, so I'll attach it when it's done)

3) Sphinx - Do what was done on preview: disable the script, its no longer needed.  Use the new sphinx.conf that requires a, see for an example.  Run the indexer --all --rotate manually to verify things work and time it.  If it runs in under a few minutes change the cron from 1  hour to 5 minutes.

4) `svn up`  There are a couple local changes which need to be reverted.

5) Update with `svn up`

6) Restart apache to pick up gettext changes

7) Clear all caches (Zeus, memcache, local cake cache) on both AMO and SAMO.

8) There are a few new cron jobs as well as changes to a lot of existing ones.  Please make the production crontab look like preview's.  A bunch of jobs switched from maintenance.php to

I still need to tag the site so this bug is serving as advanced notice for now.
Comment 1 User image Wil Clouser [:clouserw] 2009-10-20 12:40:30 PDT
Created attachment 407360 [details]
Comment 2 User image Jeremy Orem [:oremj] 2009-10-20 21:30:52 PDT
Created attachment 407458 [details]
First set of queries.

The next set after the failed query is coming soon.
Comment 3 User image chizu 2009-10-20 21:34:09 PDT
New time for sphinx indexing:
real	2m27.385s
user	0m14.050s
sys	0m0.990s

I lowered it to 15 minutes for now.
Comment 4 User image Dave Dash [:davedash, :dd] (assign all bugs to mbrandt) 2009-10-20 21:34:55 PDT
Comment 5 User image Jeremy Orem [:oremj] 2009-10-20 21:52:21 PDT
steps 1, 3, 4, 5, 6, 7, 8 done.
Comment 6 User image Jeremy Orem [:oremj] 2009-10-20 21:55:09 PDT
Created attachment 407461 [details]
Second set of queries.

mysql> delete from test_results where test_case_id=122;
Query OK, 4270 rows affected (1.09 sec)

mysql> delete from test_cases where id=122;
Query OK, 1 row affected (0.04 sec)
Comment 7 User image Jeremy Orem [:oremj] 2009-10-20 21:55:36 PDT
I think that's everything.
Comment 8 User image Stephen Donner [:stephend] 2009-10-20 23:55:51 PDT
Verified FIXED; we ran Selenium, verified top fixes on production, and looks great now (we were waiting on script crons to finish running).

Note You need to log in before you can comment on or make changes to this bug.