573522 - Update addons.mozilla.org on 6/22 @ 1600

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Wil Clouser [:clouserw]]

This is an advanced notice bug for an AMO update on 6/22 at 1600 PDT.  I'll fill in details soon; don't forget to hang out in #remora and x301 for the release.

There is a pre-release meeting at 1630 today if you have questions/concerns.

Bonus note:  I landed the patch that will let us warm up memcache ahead of time - it would be great if we could do that.  Basically we update the main box, but don't push out to the webheads, and then run log_replay on that box to dump a bunch of stuff in memcache.  Adding this note here so IT can be prepared with a log file.

Comment 1

[Jeff Balogh (:jbalogh)]

The weirdness in this release is that we're upgrading Django.  zamboni 5.11.1 is incompatible with current Django trunk and won't serve any requests.  As long as we upgrade zamboni and then upgrade /vendor there won't be a problem.

Comment 2

[Wil Clouser [:clouserw]]

1) git fetch && git co -f origin/5.11.2

2) git submodule update --init

3) cd vendor; git co -f origin/5.11.2; cd ..

4) ./manage.py compress_assets

5) schematic migrations

6) Start sending traffic to the box to warm up memcache

7) Add NETAPP_STORAGE, UPLOADS_PATH, and USERPICS_PATH to settings_local.py.  Look at PAMO for an example.

8) Fix up USER_PIC_URL in settings_local.py.  See bug 573543 for what we did on PAMO

9) python scripts/runonce/568892-associate-mimetypes-with-user-pics.py

10) Repeat steps 1-4 for SAMO

11) Restart celeryd

12) Push the new sites to the webheads

13) Restart apache for l10n and the like

14) Wait a while for db/memcache to calm down (tell us a funny story!)

15) Start sending new pages to zamboni.  Rewrites:

>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/about/?)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/contribute.*)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/developers/?)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/developers/roadblock.*)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/versions.*)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/eula.*)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addon/\d+/privacy.*)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addons/versions/\d+/?)$ /z$1 [PT]
>    RewriteRule ^((?:(?:/\w{2,3}(?:-\w{2})?)(?:/(?:firefox|seamonkey|thunderbird|mobile|sunbird))?)?/addons/policy.*)$ /z$1 [PT]

Comment 3

[Reed Loden [:reed]]

Why are we doing an AMO upgrade on a release day? Sounds like a very bad idea.

Comment 4

[Jeff Balogh (:jbalogh)]

3.5) git submodule update (in /vendor)

(In reply to comment #3)
> Why are we doing an AMO upgrade on a release day? Sounds like a very bad idea.

Thank you for your valued input!

Comment 5

[LegNeato]

Is there going to be downtime? I didn't see a scheduled downtime email go out, but perhaps I missed it.

I'm just concerned about not getting ADU pings or skewed data for the just released 3.6.4. If that's not affected I don't really care then :-)

Comment 6

[Wil Clouser [:clouserw]]

(In reply to comment #5)
> Is there going to be downtime? I didn't see a scheduled downtime email go out,
> but perhaps I missed it.

no downtime expected

Comment 7

[LegNeato]

I still would feel more comfortable this waiting a day. Is there a pressing need to get this out today? I'd hate for something to go wrong and introduce downtime. As reed mentioned in IRC we direct people to AMO with our whatsnew page, the addons manager checks for updates, we get ADU pings, etc.

Again, this is your release, I would just prefer this to wait a day or so.

Thanks!

Comment 8

[Jeremy Orem [:oremj]]

[root@mradm02 ~]# cd /data/amo_python/src/prod/zamboni/
[root@mradm02 zamboni]# git fetch
remote: Counting objects: 633, done.
remote: Compressing objects: 100% (312/312), done.
remote: Total 520 (delta 351), reused 273 (delta 165)
Receiving objects: 100% (520/520), 69.91 KiB, done.
Resolving deltas: 100% (351/351), completed with 77 local objects.
From git://github.com/jbalogh/zamboni
   97d3602..9e97284  master     -> origin/master
 + f7ad5cf...478b1fe next       -> origin/next  (forced update)
 + 3ca7ec4...bfeb27f reviews    -> origin/reviews  (forced update)
 * [new tag]         5.11.2     -> 5.11.2


git submodule update --init
remote: Counting objects: 493, done.
remote: Compressing objects: 100% (168/168), done.
remote: Total 415 (delta 230), reused 414 (delta 229)
Receiving objects: 100% (415/415), 9.03 MiB | 4.98 MiB/s, done.
Resolving deltas: 100% (230/230), completed with 36 local objects.
From git://github.com/clouserw/amo-locales
   95922d4..3288043  master     -> origin/master
Submodule path 'locale': checked out '32880432a293a140a7730ab155d0d464d558447d'
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From git://github.com/jbalogh/amo2009-img
   f60bb5f..d8326ee  master     -> origin/master
Submodule path 'media/img/amo2009': checked out 'd8326eedccb806fd2c49523195837980ba7d0483'

git checkout 5.11.2
M	src/django
M	src/django-cache-machine
Note: moving to '5.11.2' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b <new_branch_name>
HEAD is now at d4b89ef... upgrading Django and cache-machine
[root@mradm02 vendor]# git submodule update --init
remote: Counting objects: 711, done.
remote: Compressing objects: 100% (200/200), done.
remote: Total 590 (delta 492), reused 481 (delta 388)
Receiving objects: 100% (590/590), 83.37 KiB, done.
Resolving deltas: 100% (492/492), completed with 89 local objects.
From git://github.com/django/django
 * [new branch]      1.2.X      -> origin/1.2.X
   8f943d1..7731985  master     -> origin/master
   d475449..61a43e9  soc2010/app-loading -> origin/soc2010/app-loading
   41e690d..876b022  soc2010/query-refactor -> origin/soc2010/query-refactor
   5dfc150..a4e2f9b  soc2010/test-refactor -> origin/soc2010/test-refactor
Submodule path 'src/django': checked out 'ef6f6a636ee0ad1affa1f6d55317c5d4b2107e6d'
remote: Counting objects: 18, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 13 (delta 9), reused 0 (delta 0)
Unpacking objects: 100% (13/13), done.
From git://github.com/jbalogh/django-cache-machine
 + 01965dd...cd862c9 byid       -> origin/byid  (forced update)
   e608717..5cb7dc4  master     -> origin/master
Submodule path 'src/django-cache-machine': checked out 'cd862c9b49d2665427e82c99c25b8db3c50aadc1'

Script end:
Total: 15814
Done.

Comment 9

[Jeremy Orem [:oremj]]

(In reply to comment #7)
> I still would feel more comfortable this waiting a day. Is there a pressing
> need to get this out today? I'd hate for something to go wrong and introduce
> downtime. As reed mentioned in IRC we direct people to AMO with our whatsnew
> page, the addons manager checks for updates, we get ADU pings, etc.
> 
> Again, this is your release, I would just prefer this to wait a day or so.
> 
> Thanks!

This came in right after we started.

Comment 10

[LegNeato]

Ok, then nevermind ha

Comment 11

[Jeremy Orem [:oremj]]

Rewrites pushed.

Comment 12

[krupa raj[:krupa]]

5.11.2 makes me super happy!