Closed Bug 663242 Opened 14 years ago Closed 13 years ago

Test ES-0.16.2 upgrade for BuildBot

Categories

(Mozilla Metrics :: Data/Backend Reports, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INVALID
Unreviewed

People

(Reporter: aphadke, Assigned: aphadke)

References

Details

BuildBot cluster upgrade with data until 6/8 from 0.15.2 to 0.16.2 is now complete, machines used: node8.generic.metrics.sjc1.mozilla.com,node9.generic.metrics.sjc1.mozilla.com,node10.generic.metrics.sjc1.mozilla.com There is a slight change in how we will index and search logs. Please use the following URL: -for indexing new content: http://node8.generic.metrics.sjc1.mozilla.com:9200/data_<index_name>/ -for searching all content: http://node8.generic.metrics.sjc1.mozilla.com:9200/all_<index_name>/ I have created the following two aliases for testing: http://node8.generic.metrics.sjc1.mozilla.com:9200/data_logs/_search?q=*&pretty=true http://node8.generic.metrics.sjc1.mozilla.com:9200/all_logs/_search?q=*&pretty=true Above changes are needed to avoid hotspots inside ElasticSearch. Under the hood, each week we will be creating a new index and pointing the data_<index_name> alias to this new index. In addition, the all_<index_name> alias will also point to newly created index. Creating weekly index allows ElasticSearch to better balance the load. Aliases allow us to create/search indexes without any change to the underlying code.
I'm setting up an instance of the logparser on brasstacks which will report to the new aliases for testing. I can't turn it on yet, as brasstacks can't see the new server, so I'll open a ticket to get a hole opened in the firewall.
Depends on: 663273
Anurag, just to confirm: the data_ alias should be used for all writing, and the all_ index should be used for all reading? Or should all_ only be used when you want to search historical data, and not, for example, if you want to read something that may be only a few minutes old?
Jonathan, the former: data_ alias should be used for all writing and all_ alias for reading/searching. Some background: The data_ alias points to a single weekly index, say data_logs_20110601 The all_ alias points to all indexes including data_logs_20110601, data_logs_20110523 etc.
I have a copy of logparser running on brasstacks that's using the new data_logs and all_logs indices for writing/reading. I'll let it run over the weekend, and review the data on Monday.
Data collection on the test server over the weekend went well. Have you rotated the indices underlying all_logs and data_logs? We should probably do that to make sure it doesn't introduce any problems in data collection. Assuming that doesn't show any problems, in order to go live with this, we'll just have to coordinate our timing, since there are several moving pieces that will need minor updates pushed at the same time as this change.
The safest thing is to always use all_ for reading. Technically, you could use data_ for short term reading, but the problem is a potential race condition. If you did a short term query right after the alias had moved to a new partition then you would not get the desired results. If that is not a concern for you then there is no harm in using the data_ alias for short term reading and it would probably be a bit quicker since it would have less data to sort through.
Created alias + index for 2011-06-13 http://node8.generic.metrics.sjc1.mozilla.com:9200/data_logs/_search?q=*&pretty=true now points to the newly created index: http://node8.generic.metrics.sjc1.mozilla.com:9200/logs2011_06_13/_search?q=*&pretty=true all_logs alias will continue to point to all existing and newly created indexes. http://node8.generic.metrics.sjc1.mozilla.com:9200/all_logs/_search?q=*&pretty=true @jgriffin and @aphadke will test this once again tomorrow before marking this bug as resolved.
Created daily indexes and respective aliases. @jgriffin: Can u please verify whether things are working as intended? We can deploy on Monday or Tuesday (6/20; 6/21) to production if it looks good to you.
Things look good. For the rollout, what would help to avoid any interruptions, would be to create a 'logs' alias that points to 'all_logs', so that both indices work while I migrate a bunch of code in various libraries we have running. Then when I'm done, you could delete the 'logs' alias. Ditto for the rest of the indices you'll be moving to this new system (others we're using are: bzcache, autolog, tbpl, bugs). Monday afternoon (after the MoCo meetings and such) is fine with me.
The "logs" index will work without the need for creating a new alias named "logs". I will also create an alias named "old_logs" that points to "all_logs", that way, all old and new data can be searched through the "all_logs" alias. Will do the same for other indices: bzcache, autolog, tbpl, bugs Confirmed upgrade for Monday, 2pm PST.
Assignee: nobody → aphadke
Status: NEW → ASSIGNED
no longer valid as IT is handling ES now.
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.