1313319 - re:dash/PrestoDB queries fail do to workers not being available

Status: RESOLVED WORKSFORME

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Alessio Placitelli [:Dexter]]

I'm trying to run re:dash queries by they keep failing due to:

> Error running query: MaxRetryError("HTTPConnectionPool(host='ec2-54-149-111-215.us-west-2.compute.amazonaws.com', port=8889): Max retries exceeded with url: /v1/statement/20161027_101843_02116_wnz5u/3 (Caused by <class 'httplib.BadStatusLine'>: '')",) is not JSON serializable

> Error running query: {"errorCode":65541,"message":"No worker nodes available","errorType":"INTERNAL_ERROR","failureInfo":{"type":"com.facebook.presto.spi.PrestoException","message":"No worker nodes available","suppressed":[],"stack":["com.facebook.presto.util.Failures.checkCondition(Failures.java:76)","com.facebook.presto.sql.planner.SystemPartitioningHandle.getNodePartitionMap(SystemPartitioningHandle.java:149)","com.facebook.presto.sql.planner.NodePartitioningManager.getNodePartitioningMap(NodePartitioningManager.java:96)","com.facebook.presto.execution.scheduler.SqlQueryScheduler.lambda$null$0(SqlQueryScheduler.java:128)","java.util.HashMap.computeIfAbsent(HashMap.java:1126)","com.facebook.presto.execution.scheduler.SqlQueryScheduler.lambda$new$1(SqlQueryScheduler.java:128)","com.facebook.presto.execution.scheduler.SqlQueryScheduler.createStages(SqlQueryScheduler.java:220)","com.facebook.presto.execution.scheduler.SqlQueryScheduler.<init>(SqlQueryScheduler.java:119)","com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:322)","com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:226)","com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:62)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)","java.lang.Thread.run(Thread.java:745)"]},"errorName":"NO_NODES_AVAILABLE"}

Comment 1

[Alessio Placitelli [:Dexter]]

Blake, any idea about what's going on?

Comment 2

[Alessio Placitelli [:Dexter]]

It looks like, after a bit, it started to behave again.