bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.
Please report any other irregularities here.
Hi, we will need some assistance for our next code release for Mozillians.org (all environments: dev/stage/prod). The reason is that we are going to upgrade django to version 1.7 and we will need: * Some additional steps (manual and ones-off, not included in our commander script), in order to use django's new migration system. * For prod we will also need the ability to rollback to the DB state right before the code release, in case something goes wrong. Thank you!
Need ETA for this as we are currently blocked.
Nemo: Do you have a time / day in mind for doing these upgrades? That would help us figure out who is in the best position to assist. In particular, if you need database work, we'll also need to loop in one of the database folks. (CCing Sheeri.)
It would be great if we could arrange mozillians-dev.allizom.org (and if things work fine also mozillians.allizom.org) to be pushed tomorrow. After that we need a testing period and maybe next week we can push to prod as well (i will be offline for the rest of the week).
For prod we might need database help but for dev/stage its not so important since we are not using production data (we might need to rollback to a previous state buts its not critical for Mozillians.org operation).
DB help is around in #data if you need it. If you let us know when you migrate we can watch things progress.
Sounds good :sheeri. I'll ping you when we are ready to push to production. Thanks!
Nemo: It looks like we've missed each other for the day as I had other commitments this morning. Is it possible to do the push to mozillians -dev tomorrow morning (August 12th) some time after 2:30 PM UTC (5:30 EEST)? If it needs to be earlier than that, it will need to be Thursday (August 13th), when I can be available after 1:00 PM UTC (4:00 PM EEST).
Hey :cyliang, sorry for the late response but last week I was on leave. Could you please let me know which is the next (UTC friendly if possible) timeslot you could be available for that? Thank you!
Nemo: Would Monday, August 24th, at around 1:00 PM UTC (4:00 PM EEST) work for you? If we can wait until then, the new ops guy (w0ts0n) can observe the push. He is based out of the UK and is usually available earlier in the day. That should make scheduling a future push somewhat easier. =)
Monday 24th works for us. At the moment we are trying to orchestrate what we wanted you to do manually by changing the chief update script and push multiple releases. If everything goes as expected in dev/stage we might need you to help us only for prod. Thank you!
After bug 1197288, that was opened because of selenium test failures, we are blocked for the stage/prod push. As I mention in the referenced bug I can reproduce the performance issues locally and we need some time to fix this. Can we postpone this for a day later this week so we can tackle bug 1197288 first? Thank you!
No worries. Let me know when you're ready to reschedule. =)
We fixed the issues that we had with Django 1.7 upgrade and now code is on stage. Can we reschedule to push the next release on prod this Thursday 27th? FYI we most probably wont need much of a help since stage push went fine, but we are doing major changes in the DB so we need someone to rollback DB/code fast in case something goes wrong.
The Mozillians db is only 251M large, so we can do a mysqldump before the changes, and restore if needed. What time on Thu 27th? I'm personally not available after 9 am Pacific time...
:sheeri Before 9am PST works for me (I am on UTC+1).
OK, any time between 6-9 am Pacific work for me. Maybe let's shoot for 7 am Pacific, 10 am Eastern, 2 pm UTC, 3 pm UTC+1? Tomorrow (Thu) 27 Aug?
Tomorrow (Thu) 27 Aug, 7am PST is OK for me.
OK, put it in my cal and invited :nemo and C.
FWIW the release looks good. I'm on until 9 am Pacific, noon Eastern. If for some reason there's a problem and we need to revert, the mozillians_org database export is at /root/mozillians_org.sql and Matt Pressman :mpressman can import it back into the generic phx1 production database.
Things look OK (no tracebacks/selenium tests are green). I will give it some time to see if something goes wrong in the next couple of hours and will close the bug after that. Thank you :sheeri :cyliang :)
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.