production deploy demos.mozilla.org 1.3 release aka Phase III

VERIFIED FIXED

Status

VERIFIED FIXED
8 years ago
5 years ago

People

(Reporter: lmesa, Assigned: oremj)

Tracking

Details

(Whiteboard: [3/16 9:00am], URL)

(Reporter)

Description

8 years ago
Hi, we have a web code + db migration push for Web O Wonder as part of the Fx 4 launch 

We need to do this Tuesday the 15th if Fx 4 is released the 16th or 17th. We need this pushed Thursday the 17th if Fx 4 is released the week of the 21st. We'll have a final confirmed push date on Friday evening.


Austin will comment on the bug with further instructions.

Comment 1

8 years ago
What is the estimated time for this push?

Last I heard was Sunday night.

Comment 2

8 years ago
Deployment instructions:
1) git pull origin master in webowonder-demos
This is the repo that serves up https://mozillademos.org

2) svn update in the locale directory
This is under the Django app that serves up https://demos.mozilla.org

3) Run './bin/update_site.py -e prod -v' in webowonder
This is the repro that is the Django app for https://demos.mozilla.org
This command migrates the db, compresses assets, etc.

4) Execute this SQL statement:
UPDATE demos_submission SET hidden = b'0' WHERE slug IN ('runfield', 'planetarium');
This undoes a manual change from the 1.2 release.

5) Clear zeus caches.

Rollback plan:
Return DJango app to git tag 1.2.2

Hide new demos via:
UPDATE demos_submission SET hidden = b'1' WHERE slug IN ('runfield', 'planetarium');
(Reporter)

Comment 3

8 years ago
(In reply to comment #1)
> What is the estimated time for this push?
> 
> Last I heard was Sunday night.

I thought the earliest we would push is Tuesday because Raymond needed time to qa. See comment 0.
(In reply to comment #1)
> What is the estimated time for this push?
> 
> Last I heard was Sunday night.

This was not done last night per Laura's QA mention in comment 3
Assignee: server-ops → cshields
(Reporter)

Comment 5

8 years ago
Still haven't heard any firm update on Fx 4 launch timeline--I still think we should be aiming for a Tuesday push.  Raymond, can you confirm if this works for you?
(In reply to comment #5)
> Still haven't heard any firm update on Fx 4 launch timeline--I still think we
> should be aiming for a Tuesday push.  Raymond, can you confirm if this works
> for you?

Tuesday works
(Reporter)

Comment 7

8 years ago
Corey--we can push on Tuesday--when can
Whiteboard: [3/15]
(Reporter)

Comment 8

8 years ago
(In reply to comment #7)
> Corey--we can push on Tuesday--when can

Oops--meant to ask "when can we be slotted in?"
Going to assign this to Jeremy to take care of during our 4:00 window.

Keep in mind, there is a discrepancy that was never resolved after the last push, in that demos.mozilla.org has a prod branch (and is on that branch) but mozillademos.org does not (and is currently on master branch where prod was instructed and expected).
Assignee: cshields → jeremy.orem+bugs
Whiteboard: [3/15] → [3/15 4:00pm]

Comment 10

8 years ago
During launch of Phase I and Phase II, we ran into a lot of 403 errors, making the demos unavailable for a lot of people (afaik, only in Europe) for several days. Before launching Phase III, we need to make sure this is not going to happen again.

Also, launching in the afternoon is a problem because here, in Europe, it's the middle of the night, and most of the promotion of this demos is made from Europe.

Is it possible to postpone the demo to Thursday, at 9am or 10am?
(In reply to comment #10)
> During launch of Phase I and Phase II, we ran into a lot of 403 errors, making
> the demos unavailable for a lot of people (afaik, only in Europe) for several
> days. Before launching Phase III, we need to make sure this is not going to
> happen again.

We are going to need more info than this..  Just avoiding 403 errors doesn't help us know what the problem was.

> Also, launching in the afternoon is a problem because here, in Europe, it's the
> middle of the night, and most of the promotion of this demos is made from
> Europe.
> 
> Is it possible to postpone the demo to Thursday, at 9am or 10am?

We do our web pushes during windows that happen tue/thur at 16:00 MV time.  

What is wrong with promoting the videos when you wake up the next day?

Comment 12

8 years ago
(In reply to comment #11)
> (In reply to comment #10)
> > During launch of Phase I and Phase II, we ran into a lot of 403 errors, making
> > the demos unavailable for a lot of people (afaik, only in Europe) for several
> > days. Before launching Phase III, we need to make sure this is not going to
> > happen again.
> 
> We are going to need more info than this..  Just avoiding 403 errors doesn't
> help us know what the problem was.

About 403:
Phase I: https://bugzilla.mozilla.org/show_bug.cgi?id=638544
Phase II: afaik, we didn't open a bug, but this bug 641345 fixed it

> > Also, launching in the afternoon is a problem because here, in Europe, it's the
> > middle of the night, and most of the promotion of this demos is made from
> > Europe.
> > 
> > Is it possible to postpone the demo to Thursday, at 9am or 10am?
> 
> We do our web pushes during windows that happen tue/thur at 16:00 MV time.  
> 
> What is wrong with promoting the videos when you wake up the next day?

This can't be a silent launch since people are visiting WoW today.
But I can work around that if I really have to.

My main concern is about these 403 errors.
(In reply to comment #12)
The root cause of the 403 errors was an attempted fix for Bug#635398. This has been rolled back. We are not pushing it again with this release.

Comment 14

8 years ago
I talked to Austin about the 403 errors. I'm ok for the launch at 4pm (if there's no other way to do it in the morning another day).
(In reply to comment #11)
(In reply to comment #14)

If we can do the release when Paul is available, it would be good. He gets a lot of feedback via Twitter and  email right after the push to production.

Launching and then promoting the next day can work, but he would be good for deployment verification too.
(Reporter)

Comment 16

8 years ago
Is it possible to do the push tomorrow or Thursday morning?
(In reply to comment #16)
> Is it possible to do the push tomorrow or Thursday morning?

Ok, tomorrow at 9:00am
Whiteboard: [3/15 4:00pm] → [3/16 9:00am]
(Assignee)

Comment 18

8 years ago
[root@mradm02 mozillademos.org]# git pull
From https://github.com/mozilla/webowonder-demos
   c287742..c819fc1  master     -> origin/master
Updating c287742..c819fc1
Fast-forward
 demos/planetarium/css/style.css      |   65 +++++++++++++++++++++++++++++++++-
 demos/planetarium/demo.html          |    9 +++--
 demos/planetarium/img/tweet-this.png |  Bin 0 -> 1129 bytes
 demos/planetarium/img/tweet.png      |  Bin 0 -> 945 bytes
 4 files changed, 70 insertions(+), 4 deletions(-)
 create mode 100644 demos/planetarium/img/tweet-this.png
 create mode 100644 demos/planetarium/img/tweet.png
[root@mradm02 webowonder]# cd locale/
[root@mradm02 locale]# svn up
A    uk
A    uk/LC_MESSAGES
A    uk/LC_MESSAGES/messages.po
A    eu
A    eu/LC_MESSAGES
A    eu/LC_MESSAGES/messages.po
U    ko/LC_MESSAGES/messages.po
U    compile-mo.sh
U    it/LC_MESSAGES/messages.po
Updated to revision 85276.
[root@mradm02 webowonder]# git pull origin prod
From https://github.com/mozilla/webowonder
 * branch            prod       -> FETCH_HEAD
Already up-to-date.
[root@mradm02 webowonder]# git submodule update --init
[root@mradm02 vendor]# git pull 
From https://github.com/mozilla/webowonder-lib
   5a8df38..8313f61  master     -> origin/master
Already up-to-date.
[root@mradm02 vendor]# git submodule update --init
[root@mradm02 webowonder]# python2.6 manage.py compress_assets
Running migration 26:
UPDATE demos_submission SET source_code_url = 'http://github.com/littleworkshop/planetarium' WHERE slug = 'planetarium';
That took 0.04 seconds
################################################## 
mysql> UPDATE demos_submission SET hidden = b'0' WHERE slug IN ('runfield',
    -> 'planetarium');
Query OK, 2 rows affected (0.02 sec)
Rows matched: 2  Changed: 2  Warnings: 0
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
verified fixed https://demos.mozilla.org/en-US/
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.