573513 - Increase hlog blocksize config parameter from 2MB to 16MB to relieve node load

Status: RESOLVED FIXED

(Mozilla Metrics :: Hadoop/HBase Operations, defect)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Currently, we are rolling over HLogs about 10 per second because this value is so low.  It was set low because if a server crashes, then any HLogs that are not written to durable HDFS storage could be lost and result in lost data.

This setting was okay when we had just the collectors, but the processors generate a lot more Put requests and caused this constant rolling of the files which in turn caused some nodes to have very high OS load values and eventually caused timeouts that brought the Socorro Collectors down.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

Changed and the cluster was restarted at 9AM PDT 2010-06-21.

The HLogs are now rolling over about once per minute and most of them have about 900 entries in them.  Some region servers that host different data such as our metrics counters table or the queue index tables have much higher entry counts, but still rolling over at a reasonable rate.

Note that in the next major version of Hadoop+HBase, we will have an append functionality that will eliminate this possible dataloss issue.