Closed
Bug 756242
Opened 13 years ago
Closed 12 years ago
staggered auto increments for marketplace/amo into puppet
Categories
(addons.mozilla.org Graveyard :: Code Quality, defect, P2)
Tracking
(Not tracked)
RESOLVED
FIXED
2012-06-21
People
(Reporter: scabral, Assigned: dustin)
References
Details
We need to test if this will work. We can change dev/staging/whatever to test this out.
Comment 1•13 years ago
|
||
Sheeri, I'm pretty sure this is not the component you're looking for.
Comment 2•13 years ago
|
||
Let us know which table you want to try first and we should just try it on dev or stage. We will just need to coordinate with krupa and other devs so as not to disrupt QA.
I think the only python code that we'll need to fix is for translations (don't ask): https://github.com/mozilla/zamboni/blob/master/apps/translations/models.py#L69
other than that, I can't think of any Python code that will break. We'll find out.
Assignee: kumar.mcmillan → nobody
Component: General → Code Quality
Product: Mozilla Services → addons.mozilla.org
QA Contact: general → code-quality
Reporter | ||
Comment 3•13 years ago
|
||
Can we try this out tomorrow (Thursday) morning? What about Sunday or Monday? (I know it's a US holiday)
Comment 4•13 years ago
|
||
We'll need to modify the translations model to work with this which will need dev time. This is a requirement for the milestone after next but not our next one (on the dev side of things) - is this time sensitive because you have hardware you've got on trial and need to give back or just because we need to get it done?
Reporter | ||
Comment 5•13 years ago
|
||
Just 'cause we need to get it done. The hardware we had was a different vendor and it's long gone, we're not going with them.
Comment 6•13 years ago
|
||
We can try for next week, will have to see how the end of this week shakes out
Target Milestone: --- → 2012-05-31
Updated•13 years ago
|
Assignee: nobody → kumar.mcmillan
Target Milestone: 2012-05-31 → 2012-06-07
Reporter | ||
Comment 7•12 years ago
|
||
[18:13:55] <kumar> we have that one table that selects next id + 1 to do an insert
My recommendation - to do an insert:
SELECT next id + @@global.auto_increment_increment
Comment 8•12 years ago
|
||
I'll try this on Monday and see if anything breaks! Sheeri, can I go ahead and use this global setting or do you need to turn it on in -dev first?
Target Milestone: 2012-06-07 → 2012-06-14
Reporter | ||
Comment 9•12 years ago
|
||
Kumar - right now auto_increment_increment is set to 1. Let's do this - try it Monday, with the current auto_increment_increment, see if the program still works (it should, but basically we're testing that the variable @@global.auto_increment_increment does the right thing).
Then, after it works, I'll change the setting of auto_increment_increment to 2, and we can verify again that it still works.
Target Milestone: 2012-06-14 → 2012-06-07
Updated•12 years ago
|
Target Milestone: 2012-06-07 → 2012-06-14
Comment 10•12 years ago
|
||
This is on -dev and has been working fine it seems. Sheeri, let me know when you want to try bumping it and I can test the translations.
Reporter | ||
Comment 11•12 years ago
|
||
[14:57:03] <sheeri> there's a problem tho
[14:57:19] <sheeri> it's a global parameter, and there are other dbs on there:
[14:57:20] <sheeri> | addons_allizom_org |
[14:57:20] <sheeri> | addons_dev_allizom_org |
[14:57:20] <sheeri> | builder_dev |
[14:57:20] <sheeri> | builder_stage |
[14:57:33] <sheeri> so changing it for dev would change it for stage too
[14:57:36] <sheeri> and for builder
Reporter | ||
Comment 12•12 years ago
|
||
Also fwiw:
[14:55:20] <kumar> I see addons_dev_allizom_org on db-zlb-rw.stage.addons.phx1.mozilla.com
Reporter | ||
Comment 13•12 years ago
|
||
Done!
mysql> set global auto_increment_increment=2;
Query OK, 0 rows affected (0.00 sec)
Comment 14•12 years ago
|
||
I did a quick spot check and translations are working fine and the id is incrementing by two as expected. Once we bang on it a little more this seems like it's a safe change.
Reporter | ||
Comment 15•12 years ago
|
||
FWIW this is on dev *and* stage. Let me know when you're ready to take on the staging server architecture that we can failover on (there's a separate heirarchy for that).
Comment 16•12 years ago
|
||
ok, that can be done in a separate bug. In response to this original bug, staggered auto increments looks like it will work.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment 17•12 years ago
|
||
Per QA we are letting this bake in -dev a few more days so I'm reverting the commit while we do a prod push today.
The fix: https://github.com/mozilla/zamboni/commit/4a4fdd43f0d9a8211d4c1fe5870821b40a25d94c
The revert https://github.com/mozilla/zamboni/commit/04c1caeae4c15297bc742b84280b94063c3d9e17
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: 2012-06-14 → 2012-06-21
Comment 18•12 years ago
|
||
The prod release has been pushed so I'm putting this back on -dev https://github.com/mozilla/zamboni/commit/cac83768f446184095268d0b0e0e87efe8e85d54
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Comment 19•12 years ago
|
||
our contractors have been testing this for a week and have found no issues with translations.
Status: RESOLVED → VERIFIED
Reporter | ||
Comment 20•12 years ago
|
||
Yay!
Reporter | ||
Comment 21•12 years ago
|
||
Note that this is on dev *and* stage, and if it works we can put it into production whenever you want us to, and enable automatic failover.
Comment 22•12 years ago
|
||
(In reply to Sheeri Cabral [:sheeri] from comment #21)
> Note that this is on dev *and* stage, and if it works we can put it into
> production whenever you want us to, and enable automatic failover.
How about putting it in production today after we push the code that switches us to the setting? The testing has been done already. Our push is at 2pm PST
Reporter | ||
Comment 23•12 years ago
|
||
Just updating - sounds good to me!
Reporter | ||
Comment 24•12 years ago
|
||
reopening, this is mine - I changed auto_increment_increment to 2 on all machines, and 1 or 2 depending on the machine in production in scl3 and phx1.
I need to put these into puppet so the changes persist across restarts.
Assignee: kumar.mcmillan → scabral
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Reporter | ||
Updated•12 years ago
|
Summary: will staggered auto increments work for marketplace/amo? → staggered auto increments for marketplace/amo into puppet
Reporter | ||
Comment 26•12 years ago
|
||
Hasn't been done yet, but the current configs are set, so the only problems are if a machine restarts.
Comment 27•12 years ago
|
||
Is there an ETA for this?
Reporter | ||
Comment 28•12 years ago
|
||
ETA is this week, barring tons of fires preventing the work from getting done.
Reporter | ||
Comment 29•12 years ago
|
||
Adding a link to bug 717638 - we're almost there, but we might spill over into next week.
See Also: → 717638
Reporter | ||
Comment 30•12 years ago
|
||
Giving to Dustin - this entails making variables for 2 options in the /etc/my.cnf:
auto_increment_increment
auto_increment_offset
They should default to not being set in /etc/my.cnf, and they should be specifically configured for the addons databases to be:
auto_increment_increment=2
and
auto_increment_offset either 1 or 2, depending on what's currently set in the running configs (mysql> SHOW GLOBAL VARIABLES LIKE 'auto_increment_offset';)
Reporter | ||
Updated•12 years ago
|
Assignee: scabral → dustin
Assignee | ||
Comment 31•12 years ago
|
||
So, these db's have been restarted - at least twice - in the last few weeks. Luckily, not the masters - at least, not by me.
Puppet only knows of addons DB's in phx1, and they're all running with an increment of 1. Hopefully scl3 has the increment of 2 so we're not duplicating id's or anything.
Incidentally, the options are already available in the puppet module, so they'll just need to be enabled -- and whatever hosts are in scl3 should be added to puppet!
So I can set this up, but clearly I lack some context, and it may already be broken. Pls advise :)
Reporter | ||
Comment 32•12 years ago
|
||
AMO's not in slc3, because scl3 is not sas70 compliant, so our failover is to the cloud (not built yet, in tickets).
let's have a vidyo meeting tomorrow, and I can explain how the auto_increment_increment/offset thing works. (e.g. the algorithm to make up good values).
Assignee | ||
Comment 33•12 years ago
|
||
I understand that part. I don't understand how you configured servers in scl3 if there are no servers in scl3 :)
Comment 34•12 years ago
|
||
Please clarify the current status of dev/stage/prod. Which are set to 1?
Assignee | ||
Comment 35•12 years ago
|
||
addons{1..7}.db.phx1 and addons{1..5}.stage.db.phx1 are set to 1. I don't know of any other addons servers.
Reporter | ||
Comment 36•12 years ago
|
||
Wil, we're updating the configs today, and after that we'll give you a full report.
Assignee | ||
Comment 37•12 years ago
|
||
So the plan is to set the odd servers to 1 and even servers to 2 across staging and prod, *except* addons7.db.phx1 (prod) is 2.
I'll set this in puppet and live.
Assignee | ||
Comment 38•12 years ago
|
||
OK. All increments are 2. Offset is 1 on
addons1.db.phx1
addons3.db.phx1
addons5.db.phx1
addons1.stage.db.phx1 (current stage master)
addons3.stage.db.phx1
addons5.stage.db.phx1
and 2 on
addons2.db.phx1
addons4.db.phx1
addons6.db.phx1
addons7.db.phx1 (current prod master)
addons2.stage.db.phx1
addons4.stage.db.phx1
The same settings are enforced from puppet.
Assignee | ||
Updated•12 years ago
|
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Updated•9 years ago
|
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•