Closed Bug 756242 Opened 12 years ago Closed 12 years ago

staggered auto increments for marketplace/amo into puppet

Categories

(addons.mozilla.org Graveyard :: Code Quality, defect, P2)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED
2012-06-21

People

(Reporter: scabral, Assigned: dustin)

References

Details

We need to test if this will work. We can change dev/staging/whatever to test this out.
Sheeri, I'm pretty sure this is not the component you're looking for.
Let us know which table you want to try first and we should just try it on dev or stage. We will just need to coordinate with krupa and other devs so as not to disrupt QA.

I think the only python code that we'll need to fix is for translations (don't ask): https://github.com/mozilla/zamboni/blob/master/apps/translations/models.py#L69

other than that, I can't think of any Python code that will break. We'll find out.
Assignee: kumar.mcmillan → nobody
Component: General → Code Quality
Product: Mozilla Services → addons.mozilla.org
QA Contact: general → code-quality
Can we try this out tomorrow (Thursday) morning? What about Sunday or Monday? (I know it's a US holiday)
We'll need to modify the translations model to work with this which will need dev time.  This is a requirement for the milestone after next but not our next one (on the dev side of things) - is this time sensitive because you have hardware you've got on trial and need to give back or just because we need to get it done?
Just 'cause we need to get it done. The hardware we had was a different vendor and it's long gone, we're not going with them.
Blocks: 749338
We can try for next week, will have to see how the end of this week shakes out
Target Milestone: --- → 2012-05-31
Assignee: nobody → kumar.mcmillan
Target Milestone: 2012-05-31 → 2012-06-07
[18:13:55] <kumar> we have that one table that selects next id + 1 to do an insert


My recommendation - to do an insert:

SELECT next id + @@global.auto_increment_increment
I'll try this on Monday and see if anything breaks! Sheeri, can I go ahead and use this global setting or do you need to turn it on in -dev first?
Target Milestone: 2012-06-07 → 2012-06-14
Kumar - right now auto_increment_increment is set to 1. Let's do this - try it Monday, with the current auto_increment_increment, see if the program still works (it should, but basically we're testing that the variable @@global.auto_increment_increment does the right thing).

Then, after it works, I'll change the setting of auto_increment_increment to 2, and we can verify again that it still works.
Target Milestone: 2012-06-14 → 2012-06-07
Target Milestone: 2012-06-07 → 2012-06-14
This is on -dev and has been working fine it seems. Sheeri, let me know when you want to try bumping it and I can test the translations.
[14:57:03] <sheeri> there's a problem tho
[14:57:19] <sheeri> it's a global parameter, and there are other dbs on there:
[14:57:20] <sheeri> | addons_allizom_org     |
[14:57:20] <sheeri> | addons_dev_allizom_org |
[14:57:20] <sheeri> | builder_dev            |
[14:57:20] <sheeri> | builder_stage          |
[14:57:33] <sheeri> so changing it for dev would change it for stage too
[14:57:36] <sheeri> and for builder
Also fwiw:
[14:55:20] <kumar> I see addons_dev_allizom_org on db-zlb-rw.stage.addons.phx1.mozilla.com
Done!

mysql> set global auto_increment_increment=2;
Query OK, 0 rows affected (0.00 sec)
I did a quick spot check and translations are working fine and the id is incrementing by two as expected. Once we bang on it a little more this seems like it's a safe change.
FWIW this is on dev *and* stage.  Let me know when you're ready to take on the staging server architecture that we can failover on (there's a separate heirarchy for that).
ok, that can be done in a separate bug. In response to this original bug, staggered auto increments looks like it will work.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Per QA we are letting this bake in -dev a few more days so I'm reverting the commit while we do a prod push today.

The fix: https://github.com/mozilla/zamboni/commit/4a4fdd43f0d9a8211d4c1fe5870821b40a25d94c
The revert https://github.com/mozilla/zamboni/commit/04c1caeae4c15297bc742b84280b94063c3d9e17
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: 2012-06-14 → 2012-06-21
The prod release has been pushed so I'm putting this back on -dev https://github.com/mozilla/zamboni/commit/cac83768f446184095268d0b0e0e87efe8e85d54
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
our contractors have been testing this for a week and have found no issues with translations.
Status: RESOLVED → VERIFIED
Yay!
Note that this is on dev *and* stage, and if it works we can put it into production whenever you want us to, and enable automatic failover.
(In reply to Sheeri Cabral [:sheeri] from comment #21)
> Note that this is on dev *and* stage, and if it works we can put it into
> production whenever you want us to, and enable automatic failover.

How about putting it in production today after we push the code that switches us to the setting? The testing has been done already. Our push is at 2pm PST
Just updating - sounds good to me!
reopening, this is mine - I changed auto_increment_increment to 2 on all machines, and 1 or 2 depending on the machine in production in scl3 and phx1. 

I need to put these into puppet so the changes persist across restarts.
Assignee: kumar.mcmillan → scabral
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Summary: will staggered auto increments work for marketplace/amo? → staggered auto increments for marketplace/amo into puppet
How's this going?
Priority: -- → P2
Hasn't been done yet, but the current configs are set, so the only problems are if a machine restarts.
Is there an ETA for this?
ETA is this week, barring tons of fires preventing the work from getting done.
Adding a link to bug 717638 - we're almost there, but we might spill over into next week.
See Also: → 717638
Giving to Dustin - this entails making variables for 2 options in the /etc/my.cnf:

auto_increment_increment
auto_increment_offset

They should default to not being set in /etc/my.cnf, and they should be specifically configured for the addons databases to be:

auto_increment_increment=2

and

auto_increment_offset either 1 or 2, depending on what's currently set in the running configs (mysql> SHOW GLOBAL VARIABLES LIKE 'auto_increment_offset';)
Assignee: scabral → dustin
So, these db's have been restarted - at least twice - in the last few weeks.  Luckily, not the masters - at least, not by me.

Puppet only knows of addons DB's in phx1, and they're all running with an increment of 1.  Hopefully scl3 has the increment of 2 so we're not duplicating id's or anything.

Incidentally, the options are already available in the puppet module, so they'll just need to be enabled -- and whatever hosts are in scl3 should be added to puppet!

So I can set this up, but clearly I lack some context, and it may already be broken.  Pls advise :)
AMO's not in slc3, because scl3 is not sas70 compliant, so our failover is to the cloud (not built yet, in tickets).

let's have a vidyo meeting tomorrow, and I can explain how the auto_increment_increment/offset thing works. (e.g. the algorithm to make up good values).
I understand that part.  I don't understand how you configured servers in scl3 if there are no servers in scl3 :)
Please clarify the current status of dev/stage/prod.  Which are set to 1?
addons{1..7}.db.phx1 and addons{1..5}.stage.db.phx1 are set to 1.  I don't know of any other addons servers.
Wil, we're updating the configs today, and after that we'll give you a full report.
So the plan is to set the odd servers to 1 and even servers to 2 across staging and prod, *except* addons7.db.phx1 (prod) is 2.  

I'll set this in puppet and live.
OK.  All increments are 2.  Offset is 1 on

 addons1.db.phx1
 addons3.db.phx1
 addons5.db.phx1
 addons1.stage.db.phx1 (current stage master)
 addons3.stage.db.phx1
 addons5.stage.db.phx1

and 2 on

 addons2.db.phx1
 addons4.db.phx1
 addons6.db.phx1
 addons7.db.phx1 (current prod master)
 addons2.stage.db.phx1
 addons4.stage.db.phx1

The same settings are enforced from puppet.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → FIXED
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.