640291 - production deploy demos.mozilla.org 1.3 release aka Phase III

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Mesa [:lmesa] [:lvmesa]]

Hi, we have a web code + db migration push for Web O Wonder as part of the Fx 4 launch 

We need to do this Tuesday the 15th if Fx 4 is released the 16th or 17th. We need this pushed Thursday the 17th if Fx 4 is released the week of the 21st. We'll have a final confirmed push date on Friday evening.


Austin will comment on the bug with further instructions.

Comment 1

[Austin King [:ozten]]

What is the estimated time for this push?

Last I heard was Sunday night.

Comment 2

[Austin King [:ozten]]

Deployment instructions:
1) git pull origin master in webowonder-demos
This is the repo that serves up https://mozillademos.org

2) svn update in the locale directory
This is under the Django app that serves up https://demos.mozilla.org

3) Run './bin/update_site.py -e prod -v' in webowonder
This is the repro that is the Django app for https://demos.mozilla.org
This command migrates the db, compresses assets, etc.

4) Execute this SQL statement:
UPDATE demos_submission SET hidden = b'0' WHERE slug IN ('runfield', 'planetarium');
This undoes a manual change from the 1.2 release.

5) Clear zeus caches.

Rollback plan:
Return DJango app to git tag 1.2.2

Hide new demos via:
UPDATE demos_submission SET hidden = b'1' WHERE slug IN ('runfield', 'planetarium');

Comment 3

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to comment #1)
> What is the estimated time for this push?
> 
> Last I heard was Sunday night.

I thought the earliest we would push is Tuesday because Raymond needed time to qa. See comment 0.

Comment 4

[Corey Shields [:cshields]]

(In reply to comment #1)
> What is the estimated time for this push?
> 
> Last I heard was Sunday night.

This was not done last night per Laura's QA mention in comment 3

Comment 5

[Laura Mesa [:lmesa] [:lvmesa]]

Still haven't heard any firm update on Fx 4 launch timeline--I still think we should be aiming for a Tuesday push.  Raymond, can you confirm if this works for you?

Comment 6

[raymond [:retornam] (inactive)]

(In reply to comment #5)
> Still haven't heard any firm update on Fx 4 launch timeline--I still think we
> should be aiming for a Tuesday push.  Raymond, can you confirm if this works
> for you?

Tuesday works

Comment 7

[Laura Mesa [:lmesa] [:lvmesa]]

Corey--we can push on Tuesday--when can

Comment 8

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to comment #7)
> Corey--we can push on Tuesday--when can

Oops--meant to ask "when can we be slotted in?"

Comment 9

[Corey Shields [:cshields]]

Going to assign this to Jeremy to take care of during our 4:00 window.

Keep in mind, there is a discrepancy that was never resolved after the last push, in that demos.mozilla.org has a prod branch (and is on that branch) but mozillademos.org does not (and is currently on master branch where prod was instructed and expected).

Comment 10

[Paul Rouget [:paul]]

During launch of Phase I and Phase II, we ran into a lot of 403 errors, making the demos unavailable for a lot of people (afaik, only in Europe) for several days. Before launching Phase III, we need to make sure this is not going to happen again.

Also, launching in the afternoon is a problem because here, in Europe, it's the middle of the night, and most of the promotion of this demos is made from Europe.

Is it possible to postpone the demo to Thursday, at 9am or 10am?

Comment 11

[Corey Shields [:cshields]]

(In reply to comment #10)
> During launch of Phase I and Phase II, we ran into a lot of 403 errors, making
> the demos unavailable for a lot of people (afaik, only in Europe) for several
> days. Before launching Phase III, we need to make sure this is not going to
> happen again.

We are going to need more info than this..  Just avoiding 403 errors doesn't help us know what the problem was.

> Also, launching in the afternoon is a problem because here, in Europe, it's the
> middle of the night, and most of the promotion of this demos is made from
> Europe.
> 
> Is it possible to postpone the demo to Thursday, at 9am or 10am?

We do our web pushes during windows that happen tue/thur at 16:00 MV time.  

What is wrong with promoting the videos when you wake up the next day?

Comment 12

[Paul Rouget [:paul]]

(In reply to comment #11)
> (In reply to comment #10)
> > During launch of Phase I and Phase II, we ran into a lot of 403 errors, making
> > the demos unavailable for a lot of people (afaik, only in Europe) for several
> > days. Before launching Phase III, we need to make sure this is not going to
> > happen again.
> 
> We are going to need more info than this..  Just avoiding 403 errors doesn't
> help us know what the problem was.

About 403:
Phase I: https://bugzilla.mozilla.org/show_bug.cgi?id=638544
Phase II: afaik, we didn't open a bug, but this bug 641345 fixed it

> > Also, launching in the afternoon is a problem because here, in Europe, it's the
> > middle of the night, and most of the promotion of this demos is made from
> > Europe.
> > 
> > Is it possible to postpone the demo to Thursday, at 9am or 10am?
> 
> We do our web pushes during windows that happen tue/thur at 16:00 MV time.  
> 
> What is wrong with promoting the videos when you wake up the next day?

This can't be a silent launch since people are visiting WoW today.
But I can work around that if I really have to.

My main concern is about these 403 errors.

Comment 13

[Austin King [:ozten]]

(In reply to comment #12)
The root cause of the 403 errors was an attempted fix for Bug#635398. This has been rolled back. We are not pushing it again with this release.

Comment 14

[Paul Rouget [:paul]]

I talked to Austin about the 403 errors. I'm ok for the launch at 4pm (if there's no other way to do it in the morning another day).

Comment 15

[Austin King [:ozten]]

(In reply to comment #11)
(In reply to comment #14)

If we can do the release when Paul is available, it would be good. He gets a lot of feedback via Twitter and  email right after the push to production.

Launching and then promoting the next day can work, but he would be good for deployment verification too.

Comment 16

[Laura Mesa [:lmesa] [:lvmesa]]

Is it possible to do the push tomorrow or Thursday morning?

Comment 17

[Corey Shields [:cshields]]

(In reply to comment #16)
> Is it possible to do the push tomorrow or Thursday morning?

Ok, tomorrow at 9:00am

Comment 18

[Jeremy Orem [:oremj]]

[root@mradm02 mozillademos.org]# git pull
From https://github.com/mozilla/webowonder-demos
   c287742..c819fc1  master     -> origin/master
Updating c287742..c819fc1
Fast-forward
 demos/planetarium/css/style.css      |   65 +++++++++++++++++++++++++++++++++-
 demos/planetarium/demo.html          |    9 +++--
 demos/planetarium/img/tweet-this.png |  Bin 0 -> 1129 bytes
 demos/planetarium/img/tweet.png      |  Bin 0 -> 945 bytes
 4 files changed, 70 insertions(+), 4 deletions(-)
 create mode 100644 demos/planetarium/img/tweet-this.png
 create mode 100644 demos/planetarium/img/tweet.png
[root@mradm02 webowonder]# cd locale/
[root@mradm02 locale]# svn up
A    uk
A    uk/LC_MESSAGES
A    uk/LC_MESSAGES/messages.po
A    eu
A    eu/LC_MESSAGES
A    eu/LC_MESSAGES/messages.po
U    ko/LC_MESSAGES/messages.po
U    compile-mo.sh
U    it/LC_MESSAGES/messages.po
Updated to revision 85276.
[root@mradm02 webowonder]# git pull origin prod
From https://github.com/mozilla/webowonder
 * branch            prod       -> FETCH_HEAD
Already up-to-date.
[root@mradm02 webowonder]# git submodule update --init
[root@mradm02 vendor]# git pull 
From https://github.com/mozilla/webowonder-lib
   5a8df38..8313f61  master     -> origin/master
Already up-to-date.
[root@mradm02 vendor]# git submodule update --init
[root@mradm02 webowonder]# python2.6 manage.py compress_assets
Running migration 26:
UPDATE demos_submission SET source_code_url = 'http://github.com/littleworkshop/planetarium' WHERE slug = 'planetarium';
That took 0.04 seconds
################################################## 
mysql> UPDATE demos_submission SET hidden = b'0' WHERE slug IN ('runfield',
    -> 'planetarium');
Query OK, 2 rows affected (0.02 sec)
Rows matched: 2  Changed: 2  Warnings: 0

Comment 19

[raymond [:retornam] (inactive)]

verified fixed https://demos.mozilla.org/en-US/