bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Need assistance for next Mozillians.org release

VERIFIED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Engagement
--
critical
VERIFIED FIXED
3 years ago
2 years ago

People

(Reporter: nemo, Unassigned)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1517] )

(Reporter)

Description

3 years ago
Hi, we will need some assistance for our next code release for Mozillians.org (all environments: dev/stage/prod). The reason is that we are going to upgrade django to version 1.7 and we will need:

* Some additional steps (manual and ones-off, not included in our commander script), in order to use django's new migration system.

* For prod we will also need the ability to rollback to the DB state right before the code release, in case something goes wrong.

Thank you!

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1517]
(Reporter)

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1517]
(Reporter)

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1517]
Severity: normal → critical
Need ETA for this as we are currently blocked.

Comment 2

3 years ago
Nemo: Do you have a time / day in mind for doing these upgrades?  That would help us figure out who is in the best position to assist.  

In particular, if you need database work, we'll also need to loop in one of the database folks.  (CCing Sheeri.)
(Reporter)

Comment 3

3 years ago
It would be great if we could arrange mozillians-dev.allizom.org (and if things work fine also mozillians.allizom.org) to be pushed tomorrow. After that we need a testing period and maybe next week we can push to prod as well (i will be offline for the rest of the week).
(Reporter)

Comment 4

3 years ago
For prod we might need database help but for dev/stage its not so important since we are not using production data (we might need to rollback to a previous state buts its not critical for Mozillians.org operation).
DB help is around in #data if you need it. If you let us know when you migrate we can watch things progress.
(Reporter)

Comment 6

3 years ago
Sounds good :sheeri. I'll ping you when we are ready to push to production.

Thanks!

Comment 7

3 years ago
Nemo: It looks like we've missed each other for the day as I had other commitments this morning.  Is it possible to do the push to mozillians -dev tomorrow morning (August 12th) some time after 2:30 PM UTC (5:30 EEST)?  If it needs to be earlier than that, it will need to be Thursday (August 13th), when I can be available after 1:00 PM UTC (4:00 PM EEST).
(Reporter)

Comment 8

3 years ago
Hey :cyliang, sorry for the late response but last week I was on leave.

Could you please let me know which is the next (UTC friendly if possible) timeslot you could be available for that?

Thank you!

Comment 9

3 years ago
Nemo: Would Monday, August 24th, at around 1:00 PM UTC (4:00 PM EEST) work for you?  If we can wait until then, the new ops guy (w0ts0n) can observe the push.  He is based out of the UK and is usually available earlier in the day.  That should make scheduling a future push somewhat easier. =)
(Reporter)

Comment 10

3 years ago
Monday 24th works for us.

At the moment we are trying to orchestrate what we wanted you to do manually by changing the chief update script and push multiple releases. If everything goes as expected in dev/stage we might need you to help us only for prod.

Thank you!
(Reporter)

Comment 11

3 years ago
After bug 1197288, that was opened because of selenium test failures, we are blocked for the stage/prod push. As I mention in the referenced bug I can reproduce the performance issues locally and we need some time to fix this.

Can we postpone this for a day later this week so we can tackle bug 1197288 first?

Thank you!

Comment 12

3 years ago
No worries.  Let me know when you're ready to reschedule. =)
(Reporter)

Comment 13

3 years ago
We fixed the issues that we had with Django 1.7 upgrade and now code is on stage. Can we reschedule to push the next release on prod this Thursday 27th?

FYI we most probably wont need much of a help since stage push went fine, but we are doing major changes in the DB so we need someone to rollback DB/code fast in case something goes wrong.
The Mozillians db is only 251M large, so we can do a mysqldump before the changes, and restore if needed.

What time on Thu 27th? I'm personally not available after 9 am Pacific time...
Depends on: 1197288
(Reporter)

Comment 15

3 years ago
:sheeri
Before 9am PST works for me (I am on UTC+1).
OK, any time between 6-9 am Pacific work for me. Maybe let's shoot for 7 am Pacific, 10 am Eastern, 2 pm UTC, 3 pm UTC+1? Tomorrow (Thu) 27 Aug?
(Reporter)

Comment 17

3 years ago
Tomorrow (Thu) 27 Aug, 7am PST is OK for me.
OK, put it in my cal and invited :nemo and C.
FWIW the release looks good.

I'm on until 9 am Pacific, noon Eastern. If for some reason there's a problem and we need to revert, the mozillians_org database export is at /root/mozillians_org.sql and Matt Pressman :mpressman can import it back into the generic phx1 production database.
(Reporter)

Comment 20

3 years ago
Things look OK (no tracebacks/selenium tests are green).
I will give it some time to see if something goes wrong in the next couple of hours and will close the bug after that.

Thank you :sheeri :cyliang :)
(Reporter)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.