Open
Bug 1523226
Opened 6 years ago
Updated 4 years ago
generic-worker: handle zero byte S3 artifacts
Categories
(Taskcluster :: Workers, defect)
Taskcluster
Workers
Tracking
(Not tracked)
REOPENED
People
(Reporter: glandium, Unassigned, Mentored)
References
Details
(Keywords: good-first-bug, Whiteboard: [lang=go])
e.g. https://taskcluster-artifacts.net/NKfWu7J2TC6kSFk9axwbdw/0/public/logs/live_backing.log
[taskcluster 2019-01-28T05:18:35.979Z] Task Duration: 7m45.4502018s
[taskcluster 2019-01-28T05:18:36.660Z] Uploading artifact public/build/llvm-symbolizer.gz from file public\build\llvm-symbolizer.gz with content encoding "", mime type "application/x-gzip" and expiry 2019-02-25T05:10:34.533Z
[taskcluster 2019-01-28T05:34:07.140Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/NKfWu7J2TC6kSFk9axwbdw/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-02-25T05:10:34.533Z
[taskcluster:error] exit status 1
The mentioned artifact is actually non-existent. I'm wondering if it's waiting for a timeout on the file spawning into existence or something.
15 minutes is a long (additional) time to wait for a job that failed in 7 minutes.
Flags: needinfo?(pmoore)
|
||
Comment 1•6 years ago
|
||
That is indeed very strange...
[taskcluster 2019-01-28T05:18:36.660Z] Uploading artifact public/build/llvm-symbolizer.gz from file public\build\llvm-symbolizer.gz with content encoding "", mime type "application/x-gzip" and expiry 2019-02-25T05:10:34.533Z
[taskcluster 2019-01-28T05:34:07.140Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/NKfWu7J2TC6kSFk9axwbdw/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-02-25T05:10:34.533Z
[taskcluster:error] exit status 1
It looks like the artifact exists according to the queue, but is missing in cloudfront:
$ curl -IL https://queue.taskcluster.net/v1/task/NKfWu7J2TC6kSFk9axwbdw/runs/0/artifacts/public/build/llvm-symbolizer.gz
HTTP/1.1 303 See Other
Server: Cowboy
Connection: keep-alive
Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
X-Content-Type-Options: nosniff
X-For-Request-Id: 4926ce24-306b-4280-8f92-86b8cade3974
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 900
Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
Access-Control-Request-Method: *
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
Cache-Control: no-store no-cache must-revalidate
X-Taskcluster-Artifact-Storage-Type: s3
Location: https://taskcluster-artifacts.net/NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz
Vary: Accept
Content-Type: text/plain; charset=utf-8
Content-Length: 116
Date: Tue, 29 Jan 2019 14:02:51 GMT
Via: 1.1 vegur
HTTP/2 403
content-type: application/xml
date: Tue, 29 Jan 2019 14:02:51 GMT
server: AmazonS3
x-cache: Error from cloudfront
via: 1.1 73f3a23156999272233949c078c30859.cloudfront.net (CloudFront)
x-amz-cf-id: liVrqtP0ThYBEO5LdJSBHoREmMoD2mJG5eB_99Y3YwfTJmInZ7Y8Vg==
We don't see a failed upload in the task log, and the Queue believes the artifact exists (presumably because the worker told it that it exists, so the Queue probably isn't at fault). At the moment I'm not quite sure what has happened here.
Is this a one-off, or have you seen it other times too?
Flags: needinfo?(pmoore)
|
|
||
Comment 2•6 years ago
|
||
From server logs:
$ papertrail --min-time "2019-01-28T05:18:36.660Z" --max-time "2019-01-28T05:34:07.140Z" --system i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com -- "program:generic-worker"
Jan 28 06:18:37 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:37 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:37 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:36 GMT Server: AmazonS3 X-Amz-Id-2: gSM40zkv/tRbm7czjdeDBh8nXSKyvhSiZmduqhJmQiIFAfL9aUbF2v0b3IADNR2o/UCKvrlurIg= X-Amz-Request-Id: 3209CF1DA608B7A2 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>3209CF1DA608B7A2</RequestId><HostId>gSM40zkv/tRbm7czjdeDBh8nXSKyvhSiZmduqhJmQiIFAfL9aUbF2v0b3IADNR2o/UCKvrlurIg=</HostId></Error> 0
Jan 28 06:18:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:36 GMT Server: AmazonS3 X-Amz-Id-2: BkQBPjvY19JhTtMFsL7qABlbA11YJ/gwwHVf8wlhb+yzQGujzb9bDh+AsHW94JED/5ind7VKUhk= X-Amz-Request-Id: EBB739824F0762B6 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>EBB739824F0762B6</RequestId><HostId>BkQBPjvY19JhTtMFsL7qABlbA11YJ/gwwHVf8wlhb+yzQGujzb9bDh+AsHW94JED/5ind7VKUhk=</HostId></Error> 0
Jan 28 06:18:39 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:39 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:39 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:38 GMT Server: AmazonS3 X-Amz-Id-2: bKehWmnUnLcPsxSXt0logBKDE7WV6AemzhfKPEc45A8De7Aor32+cI0R6cWlkv+MM6zS/fwyE5U= X-Amz-Request-Id: 1D7D5443D256CA7F 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>1D7D5443D256CA7F</RequestId><HostId>bKehWmnUnLcPsxSXt0logBKDE7WV6AemzhfKPEc45A8De7Aor32+cI0R6cWlkv+MM6zS/fwyE5U=</HostId></Error> 0
Jan 28 06:18:40 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:40 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:40 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:38 GMT Server: AmazonS3 X-Amz-Id-2: GOmeS2yqNTnmU1WDCfqpWGud8zexOw/Ro74YE7g0xPPzLHItYZIIPwB8SOwVofGuiT5XKSCznJo= X-Amz-Request-Id: FC00C09BC19CD9AA 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>FC00C09BC19CD9AA</RequestId><HostId>GOmeS2yqNTnmU1WDCfqpWGud8zexOw/Ro74YE7g0xPPzLHItYZIIPwB8SOwVofGuiT5XKSCznJo=</HostId></Error> 0
Jan 28 06:18:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:40 GMT Server: AmazonS3 X-Amz-Id-2: CbKnzNDicLG3tg0SBSmEyVHQ8TvIiFLEIgvJBR4Ck63Cxt0Zv2Tylp4FXubZYw2cQftG96A/S+M= X-Amz-Request-Id: 514CE97A1CBB68E5 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>514CE97A1CBB68E5</RequestId><HostId>CbKnzNDicLG3tg0SBSmEyVHQ8TvIiFLEIgvJBR4Ck63Cxt0Zv2Tylp4FXubZYw2cQftG96A/S+M=</HostId></Error> 0
Jan 28 06:18:44 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:44 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:44 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:43 GMT Server: AmazonS3 X-Amz-Id-2: Gu9hOfB8ofFW8BwdMNL6F9phYkD8EypH4LrydJGNHxZVuFBvYSX1RePZRWCyHLUyu6xqdKjmwkQ= X-Amz-Request-Id: 0614E9E1BAB5D9CC 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>0614E9E1BAB5D9CC</RequestId><HostId>Gu9hOfB8ofFW8BwdMNL6F9phYkD8EypH4LrydJGNHxZVuFBvYSX1RePZRWCyHLUyu6xqdKjmwkQ=</HostId></Error> 0
Jan 28 06:18:49 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:49 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:49 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:48 GMT Server: AmazonS3 X-Amz-Id-2: PLQ+B0ZBXDAecLyLwvM9j9gPoCVuDIR8wQZJABsSCORw9IqJakgoVjRFnHFHEE1/TFjhQsF5slU= X-Amz-Request-Id: D7F52D2FD76E78C7 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>D7F52D2FD76E78C7</RequestId><HostId>PLQ+B0ZBXDAecLyLwvM9j9gPoCVuDIR8wQZJABsSCORw9IqJakgoVjRFnHFHEE1/TFjhQsF5slU=</HostId></Error> 0
Jan 28 06:18:52 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:52 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:52 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:51 GMT Server: AmazonS3 X-Amz-Id-2: j/XExrxs18s7MvHDj1x3qGX7PDQHatazLq8mxdfRMWEOufr8DiL+rIRvtUjKSHRKThN5hRdtPL4= X-Amz-Request-Id: DB3A37F701203940 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>DB3A37F701203940</RequestId><HostId>j/XExrxs18s7MvHDj1x3qGX7PDQHatazLq8mxdfRMWEOufr8DiL+rIRvtUjKSHRKThN5hRdtPL4=</HostId></Error> 0
Jan 28 06:18:58 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:18:58 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:18:58 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:56 GMT Server: AmazonS3 X-Amz-Id-2: CTfGEh5B5uy1Ta7P46eusSLTieCNn9MfUv6C7AKYVFqiajTMmBlwNGYYw3JoRv4LgMlvnjKyyj8= X-Amz-Request-Id: CD84375FAB35793E 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>CD84375FAB35793E</RequestId><HostId>CTfGEh5B5uy1Ta7P46eusSLTieCNn9MfUv6C7AKYVFqiajTMmBlwNGYYw3JoRv4LgMlvnjKyyj8=</HostId></Error> 0
Jan 28 06:19:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:19:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:19:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:19:05 GMT Server: AmazonS3 X-Amz-Id-2: 5fiJL8E1YFlo8WZU8Et9IxaQd25GGnkDcP8lCqlN0vtI9MeZp2XiVCucpHcH+2UE+Gep0/ysTSQ= X-Amz-Request-Id: 1A3A68ADA5064ADF 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>1A3A68ADA5064ADF</RequestId><HostId>5fiJL8E1YFlo8WZU8Et9IxaQd25GGnkDcP8lCqlN0vtI9MeZp2XiVCucpHcH+2UE+Gep0/ysTSQ=</HostId></Error> 0
Jan 28 06:19:31 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:19:31 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:19:31 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:19:30 GMT Server: AmazonS3 X-Amz-Id-2: KPrdJWFi/lfuEGu0DXFD3UlK7XD0HK5KTKdEBlyanvra2kVqFhuKL3wRMzD9FvDDkIwLthukDCM= X-Amz-Request-Id: B8DFF53483B8E59E 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>B8DFF53483B8E59E</RequestId><HostId>KPrdJWFi/lfuEGu0DXFD3UlK7XD0HK5KTKdEBlyanvra2kVqFhuKL3wRMzD9FvDDkIwLthukDCM=</HostId></Error> 0
Jan 28 06:20:09 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:20:09 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:20:09 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:20:08 GMT Server: AmazonS3 X-Amz-Id-2: sz7ilmpbA3SvOuWuEwU/pt3IooZLMAGYEeFBKVqcPJswBLBR8sZg4Smh9Tv5LygUCZI5Dn4DwOE= X-Amz-Request-Id: DAD4048EE2FE118F 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>DAD4048EE2FE118F</RequestId><HostId>sz7ilmpbA3SvOuWuEwU/pt3IooZLMAGYEeFBKVqcPJswBLBR8sZg4Smh9Tv5LygUCZI5Dn4DwOE=</HostId></Error> 0
Jan 28 06:20:43 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:20:43 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:20:43 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:20:42 GMT Server: AmazonS3 X-Amz-Id-2: kjfsFn1KqWB/TZlgvfdEs+z24V4hZYksVAtzlxV+QIxL6O/p1+NrqG6qwErF+IolJ990rOss4pg= X-Amz-Request-Id: D2B2E37D3C4ACAC2 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>D2B2E37D3C4ACAC2</RequestId><HostId>kjfsFn1KqWB/TZlgvfdEs+z24V4hZYksVAtzlxV+QIxL6O/p1+NrqG6qwErF+IolJ990rOss4pg=</HostId></Error> 0
Jan 28 06:22:04 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:22:04 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:22:04 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:22:02 GMT Server: AmazonS3 X-Amz-Id-2: vkAAjbAvC6Vp8fk6cDjp8nDc3MIe5gdSRZgog0RkI2qgtO4Wv7z40Vtl1R4ALnmtXPJRqKcHHSE= X-Amz-Request-Id: 6D1259592F401963 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>6D1259592F401963</RequestId><HostId>vkAAjbAvC6Vp8fk6cDjp8nDc3MIe5gdSRZgog0RkI2qgtO4Wv7z40Vtl1R4ALnmtXPJRqKcHHSE=</HostId></Error> 0
Jan 28 06:23:03 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:23:03 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:23:03 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:23:01 GMT Server: AmazonS3 X-Amz-Id-2: /LcjIJ6A9uXJyK/K+45Dip7S9kXMHS2yM4lMuhphewyuGBwx3zVIRcwnZNImIHOIFrkDzkcr1vA= X-Amz-Request-Id: 7140B08DB64D8293 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>7140B08DB64D8293</RequestId><HostId>/LcjIJ6A9uXJyK/K+45Dip7S9kXMHS2yM4lMuhphewyuGBwx3zVIRcwnZNImIHOIFrkDzkcr1vA=</HostId></Error> 0
Jan 28 06:23:54 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:23:54 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:23:54 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:23:53 GMT Server: AmazonS3 X-Amz-Id-2: BguZXYgoFk0/Q+q1X30IqRbgIHVPNNBV+47vStucoxttxVMOjaNtKqvSgB0R7hom7yFLLSW/VHo= X-Amz-Request-Id: 90A72B91A733480F 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>90A72B91A733480F</RequestId><HostId>BguZXYgoFk0/Q+q1X30IqRbgIHVPNNBV+47vStucoxttxVMOjaNtKqvSgB0R7hom7yFLLSW/VHo=</HostId></Error> 0
Jan 28 06:24:33 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:24:33 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:24:33 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:24:31 GMT Server: AmazonS3 X-Amz-Id-2: eQkW4jmwFa2nId3ngjWqY3f0ZOjsR4slt3oy5ObAq1HQQ+xHlxq5JbHRbqw8gzhPAhwlBJFkodU= X-Amz-Request-Id: 5FB816C0272FA413 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>5FB816C0272FA413</RequestId><HostId>eQkW4jmwFa2nId3ngjWqY3f0ZOjsR4slt3oy5ObAq1HQQ+xHlxq5JbHRbqw8gzhPAhwlBJFkodU=</HostId></Error> 0
Jan 28 06:25:13 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:25:13 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:25:13 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:25:11 GMT Server: AmazonS3 X-Amz-Id-2: DTzizgysFO5rTb6QK7fAmZYWb/9P0dlSvCJRW2e3b/4cLKsgxgwlQ4JEqueEBhB7vzKtf1s9Gn8= X-Amz-Request-Id: B10F11ABF8B9F7C7 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>B10F11ABF8B9F7C7</RequestId><HostId>DTzizgysFO5rTb6QK7fAmZYWb/9P0dlSvCJRW2e3b/4cLKsgxgwlQ4JEqueEBhB7vzKtf1s9Gn8=</HostId></Error> 0
Jan 28 06:25:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:25:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:25:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:25:51 GMT Server: AmazonS3 X-Amz-Id-2: M7/KW4AgXcPKM1g4pV8OCw8l4v3fklS4HeZSmnf2oLZvj/VEWgYEs4Pve8oml+7eoWZvMSMb7KM= X-Amz-Request-Id: 81702B74DD267AE5 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>81702B74DD267AE5</RequestId><HostId>M7/KW4AgXcPKM1g4pV8OCw8l4v3fklS4HeZSmnf2oLZvj/VEWgYEs4Pve8oml+7eoWZvMSMb7KM=</HostId></Error> 0
Jan 28 06:26:24 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:26:24 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:26:24 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:26:23 GMT Server: AmazonS3 X-Amz-Id-2: Cp6OPee5fdSQxLxMFK1I+8sM9J0H2Dw76HShXudFZqe3pTvsjWvEq5VY8b6uv1osOFL3k52WbrA= X-Amz-Request-Id: 02864EB1BB0ECC38 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>02864EB1BB0ECC38</RequestId><HostId>Cp6OPee5fdSQxLxMFK1I+8sM9J0H2Dw76HShXudFZqe3pTvsjWvEq5VY8b6uv1osOFL3k52WbrA=</HostId></Error> 0
Jan 28 06:27:02 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:27:02 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:27:02 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:27:01 GMT Server: AmazonS3 X-Amz-Id-2: XQhUiqRtE03zaiampzkbEDJwjGLUpuNl8lSu+R4SslhLF+/842ADFt/2/sf9X6zlNrrXYVoTCrw= X-Amz-Request-Id: A6150621BA41458E 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>A6150621BA41458E</RequestId><HostId>XQhUiqRtE03zaiampzkbEDJwjGLUpuNl8lSu+R4SslhLF+/842ADFt/2/sf9X6zlNrrXYVoTCrw=</HostId></Error> 0
Jan 28 06:27:49 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: About to reclaim task NKfWu7J2TC6kSFk9axwbdw...
Jan 28 06:27:49 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Reclaiming task NKfWu7J2TC6kSFk9axwbdw...
Jan 28 06:27:50 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Reclaimed task NKfWu7J2TC6kSFk9axwbdw successfully.
Jan 28 06:27:50 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Successfully reclaimed task NKfWu7J2TC6kSFk9axwbdw
Jan 28 06:27:50 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Reclaiming task NKfWu7J2TC6kSFk9axwbdw at 2019-01-28 05:44:49.215 +0000 UTC
Jan 28 06:27:50 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Current task claim expires at 2019-01-28 05:47:49.215 +0000 UTC
Jan 28 06:27:50 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Reclaiming task NKfWu7J2TC6kSFk9axwbdw in 16m59.8396585s
Jan 28 06:27:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:27:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:27:53 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:27:52 GMT Server: AmazonS3 X-Amz-Id-2: dkmAxsY3MiPy2U8kUlxrztDhbi4jqpU9VoWR16rXGwdJ23jawWX3PXlYSuw7L6PLIi9s41Zk8mU= X-Amz-Request-Id: 489B72C19655F55D 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>489B72C19655F55D</RequestId><HostId>dkmAxsY3MiPy2U8kUlxrztDhbi4jqpU9VoWR16rXGwdJ23jawWX3PXlYSuw7L6PLIi9s41Zk8mU=</HostId></Error> 0
Jan 28 06:28:29 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:28:29 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:28:29 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:28:28 GMT Server: AmazonS3 X-Amz-Id-2: I6LgpH6bsH35Pfloq692D/TIzFqheW0bAUrHn0+X38DhVX24yM8uIZh6yluCIFTum0Td5YDErwc= X-Amz-Request-Id: B3C7C25ABD9C6DF4 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>B3C7C25ABD9C6DF4</RequestId><HostId>I6LgpH6bsH35Pfloq692D/TIzFqheW0bAUrHn0+X38DhVX24yM8uIZh6yluCIFTum0Td5YDErwc=</HostId></Error> 0
Jan 28 06:29:35 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:29:35 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:29:35 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:29:33 GMT Server: AmazonS3 X-Amz-Id-2: BC429ShaYC7tBQumlPHnR+SqLQhc3e/WkaLQazHsPT4PzZlRWoALyjpP+MsbdkUy1Pt9WzIOjv4= X-Amz-Request-Id: D317370F5E3CA56C 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>D317370F5E3CA56C</RequestId><HostId>BC429ShaYC7tBQumlPHnR+SqLQhc3e/WkaLQazHsPT4PzZlRWoALyjpP+MsbdkUy1Pt9WzIOjv4=</HostId></Error> 0
Jan 28 06:30:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:30:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:30:41 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:30:40 GMT Server: AmazonS3 X-Amz-Id-2: DX9Xyvv13tOlDvDJe0J/vbMd4Vi4gxlVqdZdJ3KUT2AFhkLsQfKxAkc147cgSAd+J2SqOkWPC7Q= X-Amz-Request-Id: E088D8100D2705CD 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>E088D8100D2705CD</RequestId><HostId>DX9Xyvv13tOlDvDJe0J/vbMd4Vi4gxlVqdZdJ3KUT2AFhkLsQfKxAkc147cgSAd+J2SqOkWPC7Q=</HostId></Error> 0
Jan 28 06:31:45 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:31:45 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:31:45 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:31:44 GMT Server: AmazonS3 X-Amz-Id-2: xz9FrtlCjdZ9tal09LDiUpzKc8f2CSyJbnYK6RMTk1Ur+Tg0WylToMlReTiMxDEYwbdFe6m8F5k= X-Amz-Request-Id: 18ABB6D31E58D74B 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>18ABB6D31E58D74B</RequestId><HostId>xz9FrtlCjdZ9tal09LDiUpzKc8f2CSyJbnYK6RMTk1Ur+Tg0WylToMlReTiMxDEYwbdFe6m8F5k=</HostId></Error> 0
Jan 28 06:32:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:32:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:32:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:32:37 GMT Server: AmazonS3 X-Amz-Id-2: vEb5WRop1YpFmUzC85yK2p3cCpKZda8fZf5YxLubcyKtbEAWoKHrEKZ+UXI7BLuZnyoztEgINoE= X-Amz-Request-Id: 47123D38E7D3D766 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>47123D38E7D3D766</RequestId><HostId>vEb5WRop1YpFmUzC85yK2p3cCpKZda8fZf5YxLubcyKtbEAWoKHrEKZ+UXI7BLuZnyoztEgINoE=</HostId></Error> 0
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Request
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: PUT /NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: 28 put requests issued to https://taskcluster-public-artifacts.s3.us-west-2.amazonaws.com/NKfWu7J2TC6kSFk9axwbdw/0/public/build/llvm-symbolizer.gz?AWSAccessKeyId=AKIAJQESBGXODWDRTZUA&Content-Type=application%2Fx-gzip&Expires=1548655426&Signature=CBTxxZ4E2NLCNwJAChCsbjVsShI%3D
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Response
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:34:05 GMT Server: AmazonS3 X-Amz-Id-2: TN4DNOSCx80+QZcbwZ2qpiEGayIWL3sFWRsBUYnj+d+6nAdODMz/iY2eiFMl+OBNEFWPxwVA6jI= X-Amz-Request-Id: 9B70081FAAF2B2FE 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>9B70081FAAF2B2FE</RequestId><HostId>TN4DNOSCx80+QZcbwZ2qpiEGayIWL3sFWRsBUYnj+d+6nAdODMz/iY2eiFMl+OBNEFWPxwVA6jI=</HostId></Error> 0
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Stopping task feature OS Groups...
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Stopping task feature Supersede...
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Stopping task feature Mounts/Caches...
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Stopping task feature Live Log...
Jan 28 06:34:06 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Redirecting public/logs/live.log to public/logs/live_backing.log
pmoore@Petes-iMac:~/go/src/github.com/taskcluster/generic-worker master $
|
|
||
Comment 3•6 years ago
|
||
What surprises me here is that we are making a PUT request with a 0-byte Content-Length.
I wonder if the file is zero bytes, and we can't upload 0 byte files....
HTTP/1.1 Host: taskcluster-public-artifacts.s3.us-west-2.amazonaws.com User-Agent: Go-http-client/1.1 Content-Length: 0 Content-Type: application/x-gzip Accept-Encoding: gzip
|
|
||
Comment 4•6 years ago
|
||
Note, if the file didn't exist, I'd expect an error artifact to be created, rather than a 0 byte s3 artifact.
I can test this theory by adding a test to upload a zero byte file, and see if we get a HTTP 501 status code back.
Also, we retry HTTP 501 failures, since we consider 5xx status codes to be intermittent issues, and 4xx codes to be permanent failures. We probably shouldn't do that, if 501 failures shouldn't be retried.
|
Reporter | |
Comment 5•6 years ago
|
||
So, the code that failed did something like:
gzip -c clang/bin/llvm-symbolizer > $UPLOAD_DIR/llvm-symbolizer.gz
And since llvm-symbolizer didn't exist, that created a 0-sized llvm-symbolizer.gz in the artifacts directory.
|
|
||
Updated•6 years ago
|
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
|
Assignee | |
Updated•6 years ago
|
Component: Generic-Worker → Workers
|
|
||
Updated•5 years ago
|
Assignee: pmoore → nobody
Status: ASSIGNED → NEW
|
|
||
Comment 6•5 years ago
|
||
The fix here isn't too hard, we should avoid publishing 0 byte S3 artifacts - it seems that isn't allowed. We should probably treat them with the same semantics as a missing file.
|
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
|
|
||
Updated•5 years ago
|
Summary: Long time spent uploading artifact after failed task → generic-worker: handle zero byte S3 artifacts
|
||
Comment 8•4 years ago
•
|
||
|
|
||
Updated•4 years ago
|
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → MOVED
|
|
||
Comment 9•4 years ago
|
||
Jan 28 06:18:38 i-0d74aa765acb92c2a.gecko-1-b-win2012.usw2.mozilla.com generic-worker: Error: (Intermittent) HTTP response code 501 HTTP/1.1 501 Not Implemented Connection: close Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 28 Jan 2019 05:18:36 GMT Server: AmazonS3 X-Amz-Id-2: BkQBPjvY19JhTtMFsL7qABlbA11YJ/gwwHVf8wlhb+yzQGujzb9bDh+AsHW94JED/5ind7VKUhk= X-Amz-Request-Id: EBB739824F0762B6 14d <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>EBB739824F0762B6</RequestId><HostId>BkQBPjvY19JhTtMFsL7qABlbA11YJ/gwwHVf8wlhb+yzQGujzb9bDh+AsHW94JED/5ind7VKUhk=</HostId></Error> 0
The issue here is the Transfer-Encoding header, not the zero length. Maybe this is (was?) a golang bug of using transfer-encoding: chunked on a zero-length body, or something like that? At any rate, let's test this with the new go-client. If it just works, great! If it fails, we can fix it in the client.
I added tests in https://github.com/taskcluster/taskcluster/pull/4599
Status: RESOLVED → REOPENED
Flags: needinfo?(pmoore)
Resolution: MOVED → ---
|
|
||
Comment 10•4 years ago
|
||
Many thanks!
Is the issue resolved now?
Flags: needinfo?(pmoore) → needinfo?(dustin)
|
|
||
Comment 11•4 years ago
|
||
No, it still needs testing and verification that a more recent Go version has fixed this.
Flags: needinfo?(dustin)
You need to log in
before you can comment on or make changes to this bug.
Description
•