Last Comment Bug 458110 - Remove stop words off AMO MySQL servers
: Remove stop words off AMO MySQL servers
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
-- normal (vote)
: ---
Assigned To: Mark Smith [:xb95]
: matthew zeier [:mrz]
Depends on:
Blocks: 447933 458901
  Show dependency treegraph
Reported: 2008-10-01 13:29 PDT by Fred Wenzel [:wenzel]
Modified: 2013-10-09 10:29 PDT (History)
6 users (show)
mark: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Description User image Fred Wenzel [:wenzel] 2008-10-01 13:29:58 PDT
As discussed in bug 447933, we want to switch off stop words for MySQL's full text index on AMO altogether. This is a server-wide setting.

- Set the MySQL system variable "ft_stopword_file" to an empty string. That'll disable stop words (cf.
- restart the server(s)
- execute "REPAIR TABLE text_search_summary QUICK;" on the AMO DB in order to rebuild the full text index according to the new settings.

We need to do this in a maintenance window, as the server restart will mean a short downtime. Also: after the index has been rebuilt, we need to watch the server load for a little while and make sure that nothing is adversely affected (as the unfiltered index will grow larger than before).

Mark, do you want to suggest a window for that to happen? On the webdev side, I will be involved, and probably Steven in order to make sure search behaves as expected.

Comment 1 User image Mark Smith [:xb95] 2008-10-01 14:17:59 PDT
If you don't mind doing it this Thursday (tomorrow), then I'm already going to be doing some work so it'd work great to put the two tasks together.

Does that work for you?
Comment 2 User image Michael Morgan [:morgamic] 2008-10-01 14:18:45 PDT
Works for me -- I can be online around that time if nobody else can.
Comment 3 User image Fred Wenzel [:wenzel] 2008-10-01 14:28:51 PDT
What time? Since I am your time plus 3h, I can't do it too far in the evening, sorry (not tomorrow anyway, Tuesday would wfm). However, I mentioned everything in comment 0, Mike could pick it up too.
Comment 4 User image Stephen Donner [:stephend] 2008-10-01 16:54:53 PDT
I'll be around; I've also done baseline perf tests using my Selenium testsuite (, the Search testcase at, and Selenium reports that the script takes ~ 24 seconds to execute (don't know how accurate Selenium's internal timing is, but it's a good start).

Is there a better (i.e. definitely more accurate) measure we can apply to search query times?
Comment 5 User image Fred Wenzel [:wenzel] 2008-10-03 07:15:26 PDT
This did not happen last night, did it?
Comment 6 User image Mark Smith [:xb95] 2008-10-06 11:11:32 PDT
Apologies.  I didn't get the downtime window scheduled and announced so was unable to take AMO down for this.

Tuesday is the new scheduled evening - tomorrow.  I've pinged mrz to let him know that this will be happening, should be able to get this announced and out this time.

Again, sorry about the dropped ball.
Comment 7 User image Fred Wenzel [:wenzel] 2008-10-06 11:24:26 PDT
No problem. Let us know when you have a schedule.
Comment 8 User image matthew zeier [:mrz] 2008-10-07 16:02:36 PDT
This is going tonight - which db host is this?  Which databases/sites will be affected?
Comment 9 User image Michael Morgan [:morgamic] 2008-10-07 16:09:57 PDT
Should be AMO-specific, I believe that is mrdb03.  IT has more info on which specific servers, but it will require a restart for the AMO master, which puts a hold on:

For versioncheck, facebook and services, we should be able to effectively pause replication so that the master restart doesn't require downtime?  Mark?
Comment 10 User image Michael Morgan [:morgamic] 2008-10-07 16:10:58 PDT
Actually, facebook has writes going to the master, so the only true read-only subdomains are services and versioncheck at this point.
Comment 11 User image Fred Wenzel [:wenzel] 2008-10-07 16:14:02 PDT
We also need to make sure that the slaves realize their indexes are out of date also and rebuild them. Or, we first restart+execute "repair table" on the master, and later on the slaves also to be sure.
Comment 12 User image Mark Smith [:xb95] 2008-10-07 16:34:48 PDT
Yeah, I'll make sure all the databases get the indexes rebuilt.
Comment 13 User image Mark Smith [:xb95] 2008-10-07 19:10:49 PDT
After some fail on my part ('the' will never be indexed since it's in >50% of the data), this is done.


(Note: no slave01 exists right now.)
Comment 14 User image Fred Wenzel [:wenzel] 2008-10-07 19:14:43 PDT
verified, thanks!
Comment 15 User image Michael Morgan [:morgamic] 2008-10-07 21:01:48 PDT
Nice work guys! :)

Note You need to log in before you can comment on or make changes to this bug.