1263959 - Presto master is dead

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

The Presto master is dead and the cluster is in an unusable state. Wesley, could you please have a look at it?

Comment 1

[Anthony Zhang [:azhang] (last day at Mozilla: 2016-04-29)]

How we fixed the issue, at least temporarily:

1. Free enough memory for Presto to start.
2. Run sudo /etc/lib/presto/bin/launcher start --config /etc/presto/conf/config.properties
3. Check /var/log/presto/launcher.log to confirm it worked.

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Anthony Zhang [:azhang] from comment #1)
> How we fixed the issue, at least temporarily:
> 
> 1. Free enough memory for Presto to start.
> 2. Run sudo /etc/lib/presto/bin/launcher start --config
> /etc/presto/conf/config.properties
> 3. Check /var/log/presto/launcher.log to confirm it worked.

I think what happened here is the following:
1. I killed the Presto service thinking that it would restart on its own, which it usually does
2. Presto couldn't restart as redash_celery was eating a large fraction of the available memory
3. I restarted redash_celery with supervisorctl which freed up memory
4. Presto started up correctly automatically

In conclusion I don't think we need any manual steps to start Presto next time this happens.