963311 - Bug in bz_cache_refresh.py ?

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Brandon Burton [:solarce]]

Attachment: es_bzcache_errors_2014-01-23.log

Earlier today the ES cluster in SCL3 which hosts the 'bzcache' index had some OOM issues that required a restart of the cluster

Unfortunately this seems to cause corruption of the 'bzcache' index

I reran the cronjob to recreate the index, after troubleshooting confirmed it was not recoverable from the cluster.

But during the cron job run, the ES cluster logged a couple errors, which may be a bug in the script or how it's making a query to ES, so I wanted to bring it to your attention

Attached is the log output from ES

Comment 1

[Jonathan Griffin (:jgriffin)]

Thanks, it looks like we're trying to update bzcache with too many entries in one request.  I'll make a fix for this.

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Request bugs from ES in chunks of 200,

ES seems to default to a limit of 1024 boolean clauses per search; this will cause us to request no more than 250 at once.

Comment 3

[Mark Côté [:mcote]]

Comment on attachment 8371145 [details] [diff] [review]
Request bugs from ES in chunks of 200,

Review of attachment 8371145 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me!

Comment 4

[Jonathan Griffin (:jgriffin)]

http://hg.mozilla.org/users/jgriffin_mozilla.com/bzcache/rev/bc2086025056

Comment 5

[Jonathan Griffin (:jgriffin)]

...and deployed.