Closed Bug 755180 Opened 13 years ago Closed 12 years ago

perform a trial run of the "qa migration" on bugzilla-stage

Categories

(Developer Services :: General, task)

Product:
Developer Services
Component:
General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: glob, Assigned: scabral)

References

Details

we need to perform a trial run of the "qa migration reclamation" migration code (bug 684088) on bugzilla-stage. steps: 1. update the database to a recent production copy (may 11th or later) 1a. set the mail delivery mechanism to Test 1b. disable LimitedEmail extension 2. run the migration script from bug 684701 (attachment 561989 [details]) 2a. verify successful completion of this step manually before proceeding 3. run the migration script from bug 684088 (attachment 604067 [details]) goals: - ensure both migration scripts execute without errors when processing all bugs (we have only been able to test with public bugs and components) - gather timings for both scripts - while bug 684088's migration script is running, which is expected to take a while, use and abuse bugzilla-stage by searching, filing and creating bugs, etc. this is to test how bugzilla behaves with the long running transaction this script employs notes: - after completion of this trial, the database may need to be reset back to a pre-migration state - this trail run can happen at any time, however earlier in the week would be preferred to avoid conflicting with out weekly push on late wed/early thu
do we need database sanitization from production for security reasons? Do we want it for other reasons (e.g. set everyone's e-mail to user@foo.com"@test" just in case? Who is responsible for each of these tests? What's the expected timeframe for this work? Are weekends acceptable for testing this migration?
(In reply to Sheeri Cabral [:sheeri] from comment #1) > do we need database sanitization from production for security reasons? no; bugzilla-stage runs a full production copy and is behind ldap auth. it's important for the purposes of this test that it's performed against a full db dump, as it's a test we're unable to perform ourselves. > Do we want it for other reasons (e.g. set everyone's e-mail to > user@foo.com"@test" just in case? no, point 1a will ensure emails aren't sent out. > Who is responsible for each of these tests? primarily dkl and i; but anyone with access to bugzilla-stage is welcome to join the party. > What's the expected timeframe for this work? i'd prefer it sooner rather than later :) > Are weekends acceptable for testing this migration? the db restore can happen on a weekend (after the thursday push), however the migration should happen during the week so dkl and i are available to perform the testing.
So to clarify, the only part you need from DBAs is: 1. update the database to a recent production copy (may 11th or later) and of course: - after completion of this trial, the database may need to be reset back to a pre-migration state Which means there's a step 0, backup the stage db.
(In reply to Sheeri Cabral [:sheeri] from comment #3) > So to clarify, the only part you need from DBAs is: > > 1. update the database to a recent production copy (may 11th or later) > > Which means there's a step 0, backup the stage db. More like just repeat step 1. We could use a more recent snapshot on stage anyway since the last one was from Sept. 2011.
Sheeri do you have time next week to work with us on steps in comment 0? I have test this again and we are looking good for a trial run on bugzilla-stage. Sooner we can do the final stage testing phase, we can start trying to schedule a time to update production. dkl
Sure, just let me know which bugzilla stage server you're using....
(In reply to Sheeri Cabral [:sheeri] from comment #6) > Sure, just let me know which bugzilla stage server you're using.... we only have one bugzilla-stage server, http://bugzilla-stage.mozilla.org/ (the other staging server we have is bugzilla-stage-tip).
I will restore the production backup tonight, to the staging server, if all goes well.
I have restored yesterday's backup of bugzilla to stage.
(In reply to Sheeri Cabral [:sheeri] from comment #9) > I have restored yesterday's backup of bugzilla to stage. bugzilla-stage no longer works: Can't connect to the database. Error: Host '10.22.70.209' is not allowed to connect to this MySQL server Is your database installed and up and running? Do you have the correct username and password selected in localconfig? once it's up again, we'll have to wait for work on bug 756946 to finish (see comment 30) before we can start this test run.
Ah, right, I overwrote the grants table when I restored the backup. I have restored the original stage grants (just the mysql tables) and all should be good now, pending the duplicate removals, which I shall do tomorrow, Finland time.
Still getting error when accessing either bugzilla-stage or bugzilla-stage-tip Can't connect to the database. Error: Unknown database 'bugstage02' Is your database installed and up and running? Do you have the correct username and password selected in localconfig? bug 759800 filed
I'd restored a physical backup, which changed grants and the database restored was "bugs". I am now copying over the logical backup, which I will import into the "bugs" database, and when it's done I will change the name of the databases so that the new data is in the staging database.
Importing to the bugs database now: [root@bugzilla1.stage.db.scl3 scabral]# ls -lrth bugs.2012.05.28.sql -rwx------ 1 scabral scabral 57G May 31 01:29 bugs.2012.05.28.sql [root@bugzilla1.stage.db.scl3 scabral]# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sda3 218G 129G 79G 63% / [root@bugzilla1.stage.db.scl3 scabral]# time mysql bugs < /home/scabral/bugs.2012.05.28.sql
Assignee: server-ops-devservices → server-ops-database
Component: Server Operations: Developer Services → Server Operations: Database
QA Contact: shyam → cshields
We're blocked on bug 756946 being a higher precedence.
Assignee: server-ops-database → scabral
Depends on: 756946
Now that we're not blocked, I can restore from production backup this weekend, if that's OK with folks.
Assignee: scabral → server-ops-devservices
Component: Server Operations: Database → Server Operations: Developer Services
QA Contact: cshields → shyam
Now that the blocker bug is done, I can restore a backup over the weekend, unless there's a problem with that.
I will wait for an OK from folks before doing this......please comment.
Works for me.
The data was loaded into bugstage01, and I am copying the database to bugstage02 now. Let me know if staging is completely broken in a few hours.
(In reply to Sheeri Cabral [:sheeri] from comment #20) > The data was loaded into bugstage01, and I am copying the database to > bugstage02 now. Let me know if staging is completely broken in a few hours. bugstage02 is bugzilla-stage-tip; that database needs to remain untouched.
i've set shutdownhtml on bugzilla-stage-tip to disable that instance until the sanitized database can be put back.
Putting the sanitized data back; will let you know when it's complete and stage-tip can be turned on again.
I believe this is all set and working now. Please be very explicit if there's more you need from me.
(In reply to Sheeri Cabral [:sheeri] from comment #24) > I believe this is all set and working now. indeed; i've reactivated bugzilla-stage-tip
the trial run on staging has been executed. there were two minor problems with the migrate_qa_to_component_watch.pl script (the boilerplate was before the #! line, and the bugzilla product wasn't excluded), both issues i have already fixed in revision 8220. we don't need another staging run to test these changes. the main migration script, which updates ~450,000 bugs, completed without error, and took under 15 minutes to complete. [21:21:18] Checking schema... [21:21:18] Reading watch-users... [21:21:18] Found 1027 watch-users... [21:21:18] Reading bugs... [21:21:19] Updating 449043 bugs... ............................................................60000/449043 (13%) ............................................................120000/449043 (26%) ............................................................180000/449043 (40%) ............................................................240000/449043 (53%) ............................................................300000/449043 (66%) ............................................................360000/449043 (80%) ............................................................420000/449043 (93%) .............................449043/449043 (100%) [21:33:19] Reading watchers... [21:33:19] Migrating 2988 watchers... [21:33:19] Migrating email settings for 570 users... [21:33:23] Updating components... [21:33:23] Committing transaction... [21:33:24] Done. a history entry was correctly inserted for each update, and an email was not triggered for each update, however an email is being triggered when updating a bug without making any changes. i am investigating this.
Assignee: server-ops-devservices → glob
(In reply to Byron Jones ‹:glob› from comment #26) > a history entry was correctly inserted for each update, and an email was not > triggered for each update, however an email is being triggered when updating > a bug without making any changes. i am investigating this. ah, the email payload was actually the automatic CC you get when saving changes; oops :) as far as i can tell, things happened as expected. i'll like to get a second set of eyes on this before i call it a success; so i'll wait for dkl to wake up and check things before i ask sheeri to restore the database, and we can close out this bug :)
After verification, all looks sane to me. dkl
\o/ sheeri: please restore the database for bugzilla-stage to any recent production dump.
Assignee: glob → scabral
Aye aye, copying over the backup from yesterday now.
Everything should be OK now. The email_setting table is still importing, but it's the last one, and I see traffic is flowing.
server's back to normal again; closing bug.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.