[AMO] Update addons.mozilla.org on 10/20



Infrastructure & Operations
WebOps: Other
8 years ago
4 years ago


(Reporter: clouserw, Assigned: oremj)




(3 attachments)



8 years ago
This is a production AMO push for 5.2.  Steps are below, please paste the
output of all commands into this bug.  I expect there to be downtime so please
use a maintenance page.  Steps to follow:

1) There are 4 new defines in config.php.default starting with PERSONAS_.  They need to move to config.php and have values for the production personas database.  Please ensure the credentials you use have read access to the database and there are no other problems (eg. firewall) accessing the database from the AMO boxes.

2) Run the attached 5.2 SQL.  There is a query in here that might fail, so we need the output.  If it does fail, we'll write some SQL to fix it. (I was waiting to attach it, but it's too slow and I want to file this bug, so I'll attach it when it's done)

3) Sphinx - Do what was done on preview: disable the prime_sphinx_index.py script, its no longer needed.  Use the new sphinx.conf that requires a localsettings.py, see localsettings.py-dist for an example.  Run the indexer --all --rotate manually to verify things work and time it.  If it runs in under a few minutes change the cron from 1  hour to 5 minutes.

4) `svn up` addons.mozilla.org.  There are a couple local changes which need to be reverted.

5) Update services.addons.mozilla.org with `svn up`

6) Restart apache to pick up gettext changes

7) Clear all caches (Zeus, memcache, local cake cache) on both AMO and SAMO.

8) There are a few new cron jobs as well as changes to a lot of existing ones.  Please make the production crontab look like preview's.  A bunch of jobs switched from maintenance.php to maintenance.py.

I still need to tag the site so this bug is serving as advanced notice for now.

Comment 1

8 years ago
Created attachment 407360 [details]


8 years ago
Assignee: server-ops → jeremy.orem+bugs

Comment 2

8 years ago
Created attachment 407458 [details]
First set of queries.

The next set after the failed query is coming soon.

Comment 3

8 years ago
New time for sphinx indexing:
real	2m27.385s
user	0m14.050s
sys	0m0.990s

I lowered it to 15 minutes for now.

Comment 5

8 years ago
steps 1, 3, 4, 5, 6, 7, 8 done.

Comment 6

8 years ago
Created attachment 407461 [details]
Second set of queries.

mysql> delete from test_results where test_case_id=122;
Query OK, 4270 rows affected (1.09 sec)

mysql> delete from test_cases where id=122;
Query OK, 1 row affected (0.04 sec)

Comment 7

8 years ago
I think that's everything.
Last Resolved: 8 years ago
Resolution: --- → FIXED
Verified FIXED; we ran Selenium, verified top fixes on production, and https://addons.mozilla.org/en-US/firefox/personas/ looks great now (we were waiting on script crons to finish running).
Depends on: 523539
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.