Closed Bug 1236831 Opened 8 years ago Closed 8 years ago

TaskCluster Index service outage: ~ 1:00am Pacific Time 5 Jan 2016

Categories

(Taskcluster :: Services, defect)

Product:
Taskcluster
Component:
Services
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: pmoore, Unassigned)

Details

From papertrail (CET timezone):


 Jan 05 10:02:16 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.54bed4f0e4c6f0a75ba9d5958615c005" host=index.taskcluster.net request_id=42f4042a-8ada-4297-905c-72754aeb5670 fwd="54.157.255.246" dyno=web.4 connect=0ms service=10ms status=500 bytes=622 
Jan 05 10:02:16 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:15 GMT body-parser:text skip empty body 
Jan 05 10:02:16 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:15 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.54bed4f0e4c6f0a75ba9d5958615c005, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: b46d67a0-224c-4948-9262-9caa7d353274 Error: Unknown Server Error 
Jan 05 10:02:16 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:16 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:16 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:16 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:16 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce" host=index.taskcluster.net request_id=e3a27d0b-5f12-4000-b34f-7849e449851d fwd="54.157.255.246" dyno=web.2 connect=1ms service=3ms status=500 bytes=622 
Jan 05 10:02:16 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:16 GMT body-parser:text skip empty body 
Jan 05 10:02:16 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:16 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: a542a9d7-5017-4279-9e8a-030e96ee3dac Error: timeout of 30000ms exceeded 
Jan 05 10:02:16 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce" host=index.taskcluster.net request_id=9d0f2624-1e75-400c-82ba-faf1c9e8326b fwd="54.157.255.246" dyno=web.3 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:16 taskcluster-index app/web.2:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:16 taskcluster-index app/web.2:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:17 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:16 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: fdc6f57d-facb-4bf0-aa80-f1bf7478e455 Error: timeout of 30000ms exceeded 
Jan 05 10:02:17 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:16 GMT body-parser:text skip empty body 
Jan 05 10:02:17 taskcluster-index app/web.3:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:17 taskcluster-index app/web.3:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:17 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce" host=index.taskcluster.net request_id=e38bae8f-5c02-44d4-b18e-f5b6d72687b6 fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:17 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:17 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: ecab4bf4-f37c-439f-9c20-2716e8c84e7f Error: Unknown Server Error 
Jan 05 10:02:17 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:17 GMT body-parser:text skip empty body 
Jan 05 10:02:17 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:17 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:17 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:17 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:17 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:17 GMT base:stats Sending points to influxdb 
Jan 05 10:02:17 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:17 GMT base:stats Sending points to influxdb 
Jan 05 10:02:17 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:17 GMT base:stats Sending points to influxdb 
Jan 05 10:02:18 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce" host=index.taskcluster.net request_id=70f322ff-6ae1-4d86-9302-94c7896795dc fwd="54.157.255.246" dyno=web.2 connect=1ms service=5ms status=500 bytes=622 
Jan 05 10:02:18 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:18 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: cdc9c459-2e65-42ba-85ad-244071a224e4 Error: timeout of 30000ms exceeded 
Jan 05 10:02:18 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:18 GMT body-parser:text skip empty body 
Jan 05 10:02:18 taskcluster-index app/web.2:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:18 taskcluster-index app/web.2:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:18 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.54bed4f0e4c6f0a75ba9d5958615c005" host=index.taskcluster.net request_id=c024d871-c52b-4ffd-9046-8be1927710c0 fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:18 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:18 GMT body-parser:text skip empty body 
Jan 05 10:02:18 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:18 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.54bed4f0e4c6f0a75ba9d5958615c005, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 5cad8d6c-64ad-4322-9cd3-5b590111aa1d Error: Unknown Server Error 
Jan 05 10:02:18 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:18 GMT base:stats Sending points to influxdb 
Jan 05 10:02:18 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:18 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:18 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:18 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:19 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce" host=index.taskcluster.net request_id=2c72916f-324f-40fd-afaa-2d0373fe47e5 fwd="54.157.255.246" dyno=web.1 connect=2ms service=4ms status=500 bytes=622 
Jan 05 10:02:19 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:19 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: d709278d-f254-4951-b515-e712de8e5c22 Error: timeout of 30000ms exceeded 
Jan 05 10:02:19 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:19 GMT body-parser:text skip empty body 
Jan 05 10:02:19 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:19 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:21 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:21 GMT base:stats Sending points to influxdb 
Jan 05 10:02:23 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:22 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.390b565498eb684ed3a59b9eb5f4c2ce, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: 5e691ab0-e0b7-419a-bdf8-2726ebd9b881 Error: timeout of 30000ms exceeded 
Jan 05 10:02:23 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:23 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:23 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:22 GMT body-parser:text skip empty body 
Jan 05 10:02:29 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=bf4d2354-5419-4fe1-83e1-f638ca89e0c4 fwd="54.157.255.246" dyno=web.2 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:29 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:29 GMT body-parser:text skip empty body 
Jan 05 10:02:29 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:29 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: e00b7751-d541-4f15-86bd-5ca56e1418d9 Error: timeout of 30000ms exceeded 
Jan 05 10:02:29 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=4bfa871a-e445-41f0-86e5-e565e67bbeec fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:29 taskcluster-index app/web.2:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:29 taskcluster-index app/web.2:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:29 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:29 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 8ec4942b-7ade-4e23-befa-cfb099fb03c3 Error: Unknown Server Error 
Jan 05 10:02:29 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:29 GMT body-parser:text skip empty body 
Jan 05 10:02:29 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:29 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:29 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:29 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:29 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:29 GMT base:stats Sending points to influxdb 
Jan 05 10:02:30 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=9f6c9729-9042-4c2c-a13a-6ac5ac833aea fwd="54.157.255.246" dyno=web.4 connect=1ms service=3ms status=500 bytes=622 
Jan 05 10:02:30 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:29 GMT body-parser:text skip empty body 
Jan 05 10:02:30 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:29 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 304608b9-7f03-4b98-8222-df9fcdb83c4c Error: Unknown Server Error 
Jan 05 10:02:30 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:30 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:30 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:30 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:31 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=52a21b76-74ed-4fae-be2c-d91bdf4b5413 fwd="54.157.255.246" dyno=web.1 connect=0ms service=3ms status=500 bytes=622 
Jan 05 10:02:31 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:30 GMT body-parser:text skip empty body 
Jan 05 10:02:31 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:30 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: b157fa7b-dbc5-4d5f-85a9-a8a7552ba108 Error: timeout of 30000ms exceeded 
Jan 05 10:02:31 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:31 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:31 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:31 GMT base:stats Sending points to influxdb 
Jan 05 10:02:32 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:32 GMT body-parser:text skip empty body 
Jan 05 10:02:32 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=2ae40534-c19d-45b9-90bf-de21259e6839 fwd="54.157.255.246" dyno=web.1 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:32 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:32 GMT body-parser:text skip empty body 
Jan 05 10:02:32 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:32 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: cd6c41d0-c2bf-48bd-a552-1fda9836b8e6 Error: timeout of 30000ms exceeded 
Jan 05 10:02:32 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:32 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:32 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=1f2637d7-82c1-4c36-ab9b-4e4cf1d4b920 fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:32 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:32 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: 7b972d24-05d1-43b9-b4a6-3c26f77a979c Error: timeout of 30000ms exceeded 
Jan 05 10:02:32 taskcluster-index app/web.3:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:32 taskcluster-index app/web.3:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:32 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:32 GMT base:stats Sending points to influxdb 
Jan 05 10:02:32 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:32 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 632dc4f3-a79b-4c06-8bef-6e91ea90f426 Error: Unknown Server Error 
Jan 05 10:02:32 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:32 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:32 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:32 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:32 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:32 GMT base:stats Sending points to influxdb 
Jan 05 10:02:32 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:32 GMT body-parser:text skip empty body 
Jan 05 10:02:33 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=a6436383-83da-45af-9be5-2aaf422c5483 fwd="54.157.255.246" dyno=web.3 connect=1ms service=5ms status=500 bytes=622 
Jan 05 10:02:33 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:32 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: b8fd0ef8-a689-46c8-ae58-5106ae948a08 Error: Unknown Server Error 
Jan 05 10:02:33 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:33 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:33 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:33 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:33 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=e70fec52-d59d-4cae-bf3a-eac6e2612157 fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:34 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=c547324b-00ed-4350-99ad-00d372fb2303 fwd="54.157.255.246" dyno=web.4 connect=1ms service=2ms status=500 bytes=622 
Jan 05 10:02:34 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:33 GMT body-parser:text skip empty body 
Jan 05 10:02:34 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:33 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 6dffce8c-686f-475f-8626-6721a607b84d Error: Unknown Server Error 
Jan 05 10:02:34 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:34 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:34 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:34 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:35 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:34 GMT base:stats Sending points to influxdb 
Jan 05 10:02:35 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:35 GMT base:stats Sending points to influxdb 
Jan 05 10:02:35 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:35 GMT body-parser:text skip empty body 
Jan 05 10:02:35 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.8e254c24285837144dfbfd6bddcaaff6" host=index.taskcluster.net request_id=f8b55f24-abcb-4e5a-ad5b-7e1603ffff00 fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:36 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=1ccae7cf-9489-45b7-8881-2ab84a1807af fwd="54.157.255.246" dyno=web.1 connect=2ms service=4ms status=500 bytes=622 
Jan 05 10:02:36 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:35 GMT base:stats Sending points to influxdb 
Jan 05 10:02:36 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:35 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: 0ad3f159-4b74-4086-91d5-49cb3f2edf25 Error: timeout of 30000ms exceeded 
Jan 05 10:02:36 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:35 GMT body-parser:text skip empty body 
Jan 05 10:02:36 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:36 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:38 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:37 GMT base:stats Sending points to influxdb 
Jan 05 10:02:38 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:38 GMT body-parser:text skip empty body 
Jan 05 10:02:38 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653" host=index.taskcluster.net request_id=d15656bd-e657-429a-ad5e-06179ae597a3 fwd="54.157.255.246" dyno=web.2 connect=1ms service=4ms status=500 bytes=622 
Jan 05 10:02:38 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:38 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.84258b4743eee0c4c167547a3e97e653, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: 76ff76e5-20b7-49f8-95fa-1e209a8178ba Error: timeout of 30000ms exceeded 
Jan 05 10:02:38 taskcluster-index app/web.2:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:38 taskcluster-index app/web.2:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:41 taskcluster-index app/web.2: Tue, 05 Jan 2016 09:02:41 GMT base:stats Sending points to influxdb 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:46 GMT body-parser:text skip empty body 
Jan 05 10:02:47 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e" host=index.taskcluster.net request_id=889d5a22-f11d-4ba1-bce7-858f6e77bc2a fwd="54.157.255.246" dyno=web.4 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:46 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 1938842e-b41f-4223-9921-cff7a3361e05 Error: Unknown Server Error 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:47 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:47 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:47 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 5cb2c12f-2774-4c05-8b20-273456ace7ca Error: Unknown Server Error 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:47 GMT body-parser:text skip empty body 
Jan 05 10:02:47 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e" host=index.taskcluster.net request_id=54ccc6db-5c64-4d42-b07f-8b1cc468b46d fwd="54.157.255.246" dyno=web.4 connect=0ms service=3ms status=500 bytes=622 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:47 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:47 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:47 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e" host=index.taskcluster.net request_id=8e1f506f-fba0-44e0-9c02-ebb9d39f6f77 fwd="54.157.255.246" dyno=web.4 connect=1ms service=3ms status=500 bytes=622 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:47 GMT body-parser:text skip empty body 
Jan 05 10:02:47 taskcluster-index app/web.4: Tue, 05 Jan 2016 09:02:47 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e, err: Error: Unknown Server Error, as JSON: {"body":{},"statusCode":503}, incidentId: 9ecb4aad-3d5f-4a4f-817d-1f7c3a25d70f Error: Unknown Server Error 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/taskcluster-client/lib/client.js:322:21 
Jan 05 10:02:47 taskcluster-index app/web.4:     at /app/node_modules/promise/lib/core.js:33:15 
Jan 05 10:02:47 taskcluster-index app/web.4:     at flush (/app/node_modules/promise/node_modules/asap/asap.js:27:13) 
Jan 05 10:02:47 taskcluster-index app/web.4:     at process._tickDomainCallback (node.js:381:11) 
Jan 05 10:02:48 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e" host=index.taskcluster.net request_id=e180ff72-22c0-47aa-b4e3-3dbe1a101a10 fwd="54.157.255.246" dyno=web.3 connect=0ms service=2ms status=500 bytes=622 
Jan 05 10:02:48 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:48 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: bab21789-dec7-4c39-b374-87baaae0423a Error: timeout of 30000ms exceeded 
Jan 05 10:02:48 taskcluster-index app/web.3: Tue, 05 Jan 2016 09:02:48 GMT body-parser:text skip empty body 
Jan 05 10:02:48 taskcluster-index app/web.3:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:48 taskcluster-index app/web.3:     at Timer.listOnTimeout (timers.js:119:15) 
Jan 05 10:02:50 taskcluster-index heroku/router: at=info method=GET path="/v1/task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e" host=index.taskcluster.net request_id=3bc01a53-4952-4688-9978-cc76456c3bd5 fwd="54.157.255.246" dyno=web.1 connect=2ms service=4ms status=500 bytes=622 
Jan 05 10:02:50 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:49 GMT body-parser:text skip empty body 
Jan 05 10:02:50 taskcluster-index app/web.1: Tue, 05 Jan 2016 09:02:49 GMT base:api Error occurred handling: /task/tc-vcs.v1.clones.5744ecf026fe1644bc98a4ee0371ca4e, err: Error: timeout of 30000ms exceeded, as JSON: {"timeout":30000,"code":"ECONNABORTED"}, incidentId: 7482979d-6bbf-4c1c-8c83-cc25b53bf76a Error: timeout of 30000ms exceeded 
Jan 05 10:02:50 taskcluster-index app/web.1:     at null.<anonymous> (/app/node_modules/superagent/lib/node/index.js:836:17) 
Jan 05 10:02:50 taskcluster-index app/web.1:     at Timer.listOnTimeout (timers.js:119:15)
Above is just a small extract from papertrail logs.

Please note, restarting the service (`heroku restart`) seemed to resolve the issue.
Hmm... Maybe we should upgrade index to use newer libs and node version...
I've certainly done some hacks to make these timeout less frequent...

But I doubt it made it to index..
This bug is closed as part of the Great Bugzilla Cleaning Sprint of 2016 by bstack and jonasfj, please reopen if you disagree.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: Index → Services
You need to log in before you can comment on or make changes to this bug.