1191420 - Need assistance for next Mozillians.org release

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[John Giannelos [:nemo-yiannis]]

Hi, we will need some assistance for our next code release for Mozillians.org (all environments: dev/stage/prod). The reason is that we are going to upgrade django to version 1.7 and we will need:

* Some additional steps (manual and ones-off, not included in our commander script), in order to use django's new migration system.

* For prod we will also need the ability to rollback to the DB state right before the code release, in case something goes wrong.

Thank you!

Comment 1

[Pierros Papadeas [:pierros]]

Need ETA for this as we are currently blocked.

Comment 2

[C. Liang [:cyliang]]

Nemo: Do you have a time / day in mind for doing these upgrades?  That would help us figure out who is in the best position to assist.  

In particular, if you need database work, we'll also need to loop in one of the database folks.  (CCing Sheeri.)

Comment 3

[John Giannelos [:nemo-yiannis]]

It would be great if we could arrange mozillians-dev.allizom.org (and if things work fine also mozillians.allizom.org) to be pushed tomorrow. After that we need a testing period and maybe next week we can push to prod as well (i will be offline for the rest of the week).

Comment 4

[John Giannelos [:nemo-yiannis]]

For prod we might need database help but for dev/stage its not so important since we are not using production data (we might need to rollback to a previous state buts its not critical for Mozillians.org operation).

Comment 5

[Sheeri Cabral [:sheeri]]

DB help is around in #data if you need it. If you let us know when you migrate we can watch things progress.

Comment 6

[John Giannelos [:nemo-yiannis]]

Sounds good :sheeri. I'll ping you when we are ready to push to production.

Thanks!

Comment 7

[C. Liang [:cyliang]]

Nemo: It looks like we've missed each other for the day as I had other commitments this morning.  Is it possible to do the push to mozillians -dev tomorrow morning (August 12th) some time after 2:30 PM UTC (5:30 EEST)?  If it needs to be earlier than that, it will need to be Thursday (August 13th), when I can be available after 1:00 PM UTC (4:00 PM EEST).

Comment 8

[John Giannelos [:nemo-yiannis]]

Hey :cyliang, sorry for the late response but last week I was on leave.

Could you please let me know which is the next (UTC friendly if possible) timeslot you could be available for that?

Thank you!

Comment 9

[C. Liang [:cyliang]]

Nemo: Would Monday, August 24th, at around 1:00 PM UTC (4:00 PM EEST) work for you?  If we can wait until then, the new ops guy (w0ts0n) can observe the push.  He is based out of the UK and is usually available earlier in the day.  That should make scheduling a future push somewhat easier. =)

Comment 10

[John Giannelos [:nemo-yiannis]]

Monday 24th works for us.

At the moment we are trying to orchestrate what we wanted you to do manually by changing the chief update script and push multiple releases. If everything goes as expected in dev/stage we might need you to help us only for prod.

Thank you!

Comment 11

[John Giannelos [:nemo-yiannis]]

After bug 1197288, that was opened because of selenium test failures, we are blocked for the stage/prod push. As I mention in the referenced bug I can reproduce the performance issues locally and we need some time to fix this.

Can we postpone this for a day later this week so we can tackle bug 1197288 first?

Thank you!

Comment 12

[C. Liang [:cyliang]]

No worries.  Let me know when you're ready to reschedule. =)

Comment 13

[John Giannelos [:nemo-yiannis]]

We fixed the issues that we had with Django 1.7 upgrade and now code is on stage. Can we reschedule to push the next release on prod this Thursday 27th?

FYI we most probably wont need much of a help since stage push went fine, but we are doing major changes in the DB so we need someone to rollback DB/code fast in case something goes wrong.

Comment 14

[Sheeri Cabral [:sheeri]]

The Mozillians db is only 251M large, so we can do a mysqldump before the changes, and restore if needed.

What time on Thu 27th? I'm personally not available after 9 am Pacific time...

Comment 15

[John Giannelos [:nemo-yiannis]]

:sheeri
Before 9am PST works for me (I am on UTC+1).

Comment 16

[Sheeri Cabral [:sheeri]]

OK, any time between 6-9 am Pacific work for me. Maybe let's shoot for 7 am Pacific, 10 am Eastern, 2 pm UTC, 3 pm UTC+1? Tomorrow (Thu) 27 Aug?

Comment 17

[John Giannelos [:nemo-yiannis]]

Tomorrow (Thu) 27 Aug, 7am PST is OK for me.

Comment 18

[Sheeri Cabral [:sheeri]]

OK, put it in my cal and invited :nemo and C.

Comment 19

[Sheeri Cabral [:sheeri]]

FWIW the release looks good.

I'm on until 9 am Pacific, noon Eastern. If for some reason there's a problem and we need to revert, the mozillians_org database export is at /root/mozillians_org.sql and Matt Pressman :mpressman can import it back into the generic phx1 production database.

Comment 20

[John Giannelos [:nemo-yiannis]]

Things look OK (no tracebacks/selenium tests are green).
I will give it some time to see if something goes wrong in the next couple of hours and will close the bug after that.

Thank you :sheeri :cyliang :)