663242 - Test ES-0.16.2 upgrade for BuildBot

Status: RESOLVED INVALID

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[Anurag Phadke[:[e-mail]]]

BuildBot cluster upgrade with data until 6/8 from 0.15.2 to 0.16.2 is now complete, machines used:
node8.generic.metrics.sjc1.mozilla.com,node9.generic.metrics.sjc1.mozilla.com,node10.generic.metrics.sjc1.mozilla.com

There is a slight change in how we will index and search logs.
Please use the following URL:

-for indexing new content:
http://node8.generic.metrics.sjc1.mozilla.com:9200/data_<index_name>/

-for searching all content:
http://node8.generic.metrics.sjc1.mozilla.com:9200/all_<index_name>/
I have created the following two aliases for testing:
http://node8.generic.metrics.sjc1.mozilla.com:9200/data_logs/_search?q=*&pretty=true

http://node8.generic.metrics.sjc1.mozilla.com:9200/all_logs/_search?q=*&pretty=true


Above changes are needed to avoid hotspots inside ElasticSearch. Under the hood, each week we will be creating a new index and pointing the data_<index_name> alias to this new index.
In addition, the all_<index_name> alias will also point to newly created index.

Creating weekly index allows ElasticSearch to better balance the load. Aliases allow us to create/search indexes without any change to the underlying code.

Comment 1

[Jonathan Griffin (:jgriffin)]

I'm setting up an instance of the logparser on brasstacks which will report to the new aliases for testing.  I can't turn it on yet, as brasstacks can't see the new server, so I'll open a ticket to get a hole opened in the firewall.

Comment 2

[Jonathan Griffin (:jgriffin)]

Anurag, just to confirm:  the data_ alias should be used for all writing, and the all_ index should be used for all reading?  Or should all_ only be used when you want to search historical data, and not, for example, if you want to read something that may be only a few minutes old?

Comment 3

[Anurag Phadke[:[e-mail]]]

Jonathan,
the former:
data_ alias should be used for all writing and all_ alias for reading/searching. 

Some background:
The data_ alias points to a single weekly index, say data_logs_20110601

The all_ alias points to all indexes including data_logs_20110601, data_logs_20110523 etc.

Comment 4

[Jonathan Griffin (:jgriffin)]

I have a copy of logparser running on brasstacks that's using the new data_logs and all_logs indices for writing/reading.  I'll let it run over the weekend, and review the data on Monday.

Comment 5

[Jonathan Griffin (:jgriffin)]

Data collection on the test server over the weekend went well.  Have you rotated the indices underlying all_logs and data_logs?  We should probably do that to make sure it doesn't introduce any problems in data collection.

Assuming that doesn't show any problems, in order to go live with this, we'll just have to coordinate our timing, since there are several moving pieces that will need minor updates pushed at the same time as this change.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

The safest thing is to always use all_ for reading.  Technically, you could use data_ for short term reading, but the problem is a potential race condition.  If you did a short term query right after the alias had moved to a new partition then you would not get the desired results.

If that is not a concern for you then there is no harm in using the data_ alias for short term reading and it would probably be a bit quicker since it would have less data to sort through.

Comment 7

[Anurag Phadke[:[e-mail]]]

Created alias + index for 2011-06-13

http://node8.generic.metrics.sjc1.mozilla.com:9200/data_logs/_search?q=*&pretty=true 
now points to the newly created index:
http://node8.generic.metrics.sjc1.mozilla.com:9200/logs2011_06_13/_search?q=*&pretty=true 

all_logs alias will continue to point to all existing and newly created indexes.

http://node8.generic.metrics.sjc1.mozilla.com:9200/all_logs/_search?q=*&pretty=true 

@jgriffin and @aphadke will test this once again tomorrow before marking this bug as resolved.

Comment 8

[Anurag Phadke[:[e-mail]]]

Created daily indexes and respective aliases. 
@jgriffin: Can u please verify whether things are working as intended? We can deploy on Monday or Tuesday (6/20; 6/21) to production if it looks good to you.

Comment 9

[Jonathan Griffin (:jgriffin)]

Things look good.  For the rollout, what would help to avoid any interruptions, would be to create a 'logs' alias that points to 'all_logs', so that both indices work while I migrate a bunch of code in various libraries we have running.  Then when I'm done, you could delete the 'logs' alias.  Ditto for the rest of the indices you'll be moving to this new system (others we're using are:  bzcache, autolog, tbpl, bugs).

Monday afternoon (after the MoCo meetings and such) is fine with me.

Comment 10

[Anurag Phadke[:[e-mail]]]

The "logs" index will work without the need for creating a new alias named "logs". I will also create an alias named "old_logs" that points to "all_logs", that way, all old and new data can be searched through the "all_logs" alias.

Will do the same for other indices: bzcache, autolog, tbpl, bugs

Confirmed upgrade for Monday, 2pm PST.

Comment 11

[Anurag Phadke[:[e-mail]]]

no longer valid as IT is handling ES now.