1324639 - Queries on Presto are failing with "Encountered too many errors talking to a worker node"

Status: RESOLVED DUPLICATE of bug 1319086

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Francesco Lodolo [:flod]]

For the past 10 hours or so I haven't been able to run queries on Presto.

Right now I've forked one existing query, when I try to run it I get a huge error message starting with

Error running query: {"errorCode":65537,"message":"Encountered too many errors talking to a worker node. The node may have crashed or be under too much load. This is probably a transient issue, so please retry your query in a few minutes. (getting task status http://172.31.18.17:8889/v1/task/20161220_045726_00046_q4taj.1.5 - 20 failures, time since last success 122.78s)","errorType":"INTERNAL_ERROR","failureInfo":{"type":"com.facebook.presto.spi.PrestoException","message":"Encountered too many errors talking to a worker node. The node may have crashed or be under too much load. This is probably a transient issue, so please retry your query in a few minutes. (getting task status http://172.31.18.17:8889/v1/task/20161220_045726_00046_q4taj.1.5 - 20 failures, time since last success 122.78s)","suppressed":[{"type":"com.facebook.presto.server.remotetask.SimpleHttpResponseHandler.ServiceUnavailableException","message":"Server returned SERVICE_UNAVAILABLE: http://172.31.18.17:8889/v1/task/20161220_045726_00046_q4taj.1.5/status","suppressed":[],"stack":["com.facebook.presto.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:52)","com.facebook.presto.server.remotetask.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:27)","com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)","io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:77)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)","java.lang.Thread.run(Thread.java:745)"]},{"type":"com.facebook.presto.server.remotetask.SimpleHttpResponseHandler.ServiceUnavailableException","message":"Server returned SERVICE_UNAVAILABLE: http://172.31.18.17:8889/v1/task/20161220_045726_00046_q4taj.1.5/status","suppressed":[],"stack":

Comment 1

[Matthew N. [:MattN]]

The other error I was seeing intermittently is:
> Error running query: Should not have nextUri if failed 

If I reduce the time my query takes by throwing in some `LIMIT NNNN` then the problem happens less it seems.

Comment 2

[Blake Imsland [:robotblake]]

This is the same underlying issue so marking it as a dupe.