1306040 - queue redirecting to cloudfront for EC2 instances

Status: RESOLVED INCOMPLETE

(Taskcluster :: Services, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Looking just at two hours of logs from the https://taskcluster-public-artifacts.taskcluster.net cloudformation logs, for distinct source IPs, I get about 2.4k distinct EC2 addresses, in both us-east-1 and us-west-2.

From https://github.com/taskcluster/taskcluster-queue/blob/master/src/artifacts.js#L358 it appears that all EC2 requests should be going either directly to the s3 bucket URL or to cloud-mirror.

The region metadata is dated 2016-09-26-16-49-06, and these two hours were the 3:00 hour UTC on the 27th and 28th, specifically logfiles

EXGGJTH3KS8NS.2016-09-27-03.1998f3eb  EXGGJTH3KS8NS.2016-09-27-03.f2a2f8ae  EXGGJTH3KS8NS.2016-09-28-03.3f5b4a34  EXGGJTH3KS8NS.2016-09-28-03.c91eb9fd
EXGGJTH3KS8NS.2016-09-27-03.234a9959  EXGGJTH3KS8NS.2016-09-27-03.f311188a  EXGGJTH3KS8NS.2016-09-28-03.4a18334b  EXGGJTH3KS8NS.2016-09-28-03.dd17876b
EXGGJTH3KS8NS.2016-09-27-03.9f5ff394  EXGGJTH3KS8NS.2016-09-27-03.fe2305ac  EXGGJTH3KS8NS.2016-09-28-03.8362d369  EXGGJTH3KS8NS.2016-09-27-03.ae0b2232  EXGGJTH3KS8NS.2016-09-28-03.18f6d377  EXGGJTH3KS8NS.2016-09-28-03.9f970f11

so the regions should not have changed since that time.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I can confirm this manually from lamport, which is in us-west-2:

(sandbox) dustin@lamport ~/p/m-c (bug1269443) $ curl -L -v -o /dev/null https://queue.taskcluster.net/v1/task/SDlWxl5OQTOX9u4ZNytuTA/artifacts/public/build/target.reftest.tests.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 54.235.141.80...
* Connected to queue.taskcluster.net (54.235.141.80) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: auth.taskcluster.net (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=California,L=Mountain View,O=Mozilla Corporation,CN=auth.taskcluster.net
*        start date: Thu, 17 Mar 2016 00:00:00 GMT
*        expire date: Fri, 22 Mar 2019 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server did not agree to a protocol
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> GET /v1/task/SDlWxl5OQTOX9u4ZNytuTA/artifacts/public/build/target.reftest.tests.zip HTTP/1.1
> Host: queue.taskcluster.net
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 303 See Other
< Server: Cowboy
< Connection: keep-alive
< X-Powered-By: Express
< Strict-Transport-Security: max-age=7776000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
< Access-Control-Request-Method: *
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin
< Location: https://public-artifacts.taskcluster.net/SDlWxl5OQTOX9u4ZNytuTA/0/public/build/target.reftest.tests.zip
< Vary: Accept
< Content-Type: text/plain; charset=utf-8
< Content-Length: 29
< Date: Wed, 28 Sep 2016 16:36:43 GMT
< Via: 1.1 vegur
<
* Ignoring the response-body
{ [29 bytes data]
100    29  100    29    0     0     34      0 --:--:-- --:--:-- --:--:--    34
* Connection #0 to host queue.taskcluster.net left intact
* Issue another request to this URL: 'https://public-artifacts.taskcluster.net/SDlWxl5OQTOX9u4ZNytuTA/0/public/build/target.reftest.tests.zip'
*   Trying 52.84.232.137...
* Connected to public-artifacts.taskcluster.net (52.84.232.137) port 443 (#1)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: auth.taskcluster.net (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=California,L=Mountain View,O=Mozilla Corporation,CN=auth.taskcluster.net
*        start date: Thu, 17 Mar 2016 00:00:00 GMT
*        expire date: Fri, 22 Mar 2019 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /SDlWxl5OQTOX9u4ZNytuTA/0/public/build/target.reftest.tests.zip HTTP/1.1
> Host: public-artifacts.taskcluster.net
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/zip
< Content-Length: 32102631
< Connection: keep-alive
< Date: Wed, 28 Sep 2016 15:12:33 GMT
< Last-Modified: Wed, 28 Sep 2016 03:45:18 GMT
< ETag: "098bdf06f370a606a75637840c4c0b3d"
< x-amz-version-id: jwmF11fp84D3PjyPIbbVORt_v1ecEQvR
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 5051
< X-Cache: Hit from cloudfront
< Via: 1.1 336f0e6ef9a3462f682d6ca49029b665.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: TdO_6aMqDx0H3Eg1X_bj91Umono9VRWe00bjTKAvj1iL-jO3S-2nZw==
<
{ [16384 bytes data]
100 30.6M  100 30.6M    0     0  18.8M      0  0:00:01  0:00:01 --:--:-- 66.8M
* Connection #1 to host public-artifacts.taskcluster.net left intact

I suspect this is, at the least, an opportunity for cost savings, but may also be related to bug 1305752 and bug 1305768.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: regions.txt

The 2438 distinct IPs I found, tagged with region.  I downloaded and unzipped the CF logs, then

cut -d'        ' -f 5 * | sort -u > ips

then ran

import requests
import json
from IPy import *

ips = [IP(ip.strip()) for ip in open("ips")]

ranges = requests.get('https://ip-ranges.amazonaws.com/ip-ranges.json').json()
ranges = [(IP(pfx['ip_prefix']), pfx['region']) for pfx in ranges['prefixes']]

for ip in ips:
    region = 'unknown'
    for pfx, rgn in ranges:
        if ip in pfx:
            region = rgn
            break
    print region, ip

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

I submitted https://github.com/taskcluster/taskcluster-queue/pull/120 to help figure out what the queue's view of the requester is

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

PR fixing the issue with us-west-2:
https://github.com/taskcluster/taskcluster-queue/pull/121

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

I merged above PR that should fix us-west-2 and give us some more debug statements.

I suspect we ought to refactor, but really, we want to move region resolution into cloud-mirror.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

I just tried to reproduce and got a valid zip file:

[ec2-user@ip-172-31-10-63 ~]$ curl -LOv https://queue.taskcluster.net/v1/task/SDlWxl5OQTOX9u4ZNytuTA/artifacts/public/build/target.reftest.tests.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to queue.taskcluster.net port 443 (#0)
*   Trying 54.235.141.80...
* Connected to queue.taskcluster.net (54.235.141.80) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=auth.taskcluster.net,O=Mozilla Corporation,L=Mountain View,ST=California,C=US
* 	start date: Mar 17 00:00:00 2016 GMT
* 	expire date: Mar 22 12:00:00 2019 GMT
* 	common name: auth.taskcluster.net
* 	issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
> GET /v1/task/SDlWxl5OQTOX9u4ZNytuTA/artifacts/public/build/target.reftest.tests.zip HTTP/1.1
> User-Agent: curl/7.29.0
> Host: queue.taskcluster.net
> Accept: */*
> 
< HTTP/1.1 303 See Other
< Server: Cowboy
< Connection: keep-alive
< X-Powered-By: Express
< Strict-Transport-Security: max-age=7776000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
< Access-Control-Request-Method: *
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin
< Location: https://cloud-mirror.taskcluster.net/v1/redirect/s3/us-west-1/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip
< Vary: Accept
< Content-Type: text/plain; charset=utf-8
< Content-Length: 29
< Date: Thu, 29 Sep 2016 14:22:23 GMT
< Via: 1.1 vegur
< 
* Ignoring the response-body
{ [data not shown]
100    29  100    29    0     0     37      0 --:--:-- --:--:-- --:--:--    36
* Connection #0 to host queue.taskcluster.net left intact
* Issue another request to this URL: 'https://cloud-mirror.taskcluster.net/v1/redirect/s3/us-west-1/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip'
* About to connect() to cloud-mirror.taskcluster.net port 443 (#1)
*   Trying 54.225.222.159...
* Connected to cloud-mirror.taskcluster.net (54.225.222.159) port 443 (#1)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=auth.taskcluster.net,O=Mozilla Corporation,L=Mountain View,ST=California,C=US
* 	start date: Mar 17 00:00:00 2016 GMT
* 	expire date: Mar 22 12:00:00 2019 GMT
* 	common name: auth.taskcluster.net
* 	issuer: CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US
> GET /v1/redirect/s3/us-west-1/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cloud-mirror.taskcluster.net
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0< HTTP/1.1 302 Found
< Server: Cowboy
< Connection: keep-alive
< X-Powered-By: Express
< Strict-Transport-Security: max-age=7776000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
< Access-Control-Request-Method: *
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin
< Location: https://cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip
< Content-Type: application/json; charset=utf-8
< Content-Length: 251
< Etag: W/"fb-c71862c4"
< Date: Thu, 29 Sep 2016 14:22:26 GMT
< Via: 1.1 vegur
< 
* Ignoring the response-body
{ [data not shown]
100   251  100   251    0     0     71      0  0:00:03  0:00:03 --:--:--   168
* Connection #1 to host cloud-mirror.taskcluster.net left intact
* Issue another request to this URL: 'https://cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip'
* About to connect() to cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com port 443 (#2)
*   Trying 54.231.236.44...
* Connected to cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com (54.231.236.44) port 443 (#2)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=*.s3-us-west-1.amazonaws.com,O=Amazon.com Inc.,L=Seattle,ST=Washington,C=US
* 	start date: Jul 18 00:00:00 2016 GMT
* 	expire date: Oct 26 12:00:00 2017 GMT
* 	common name: *.s3-us-west-1.amazonaws.com
* 	issuer: CN=DigiCert Baltimore CA-2 G2,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Ftaskcluster-public-artifacts%2FSDlWxl5OQTOX9u4ZNytuTA%2F0%2Fpublic%2Fbuild%2Ftarget.reftest.tests.zip HTTP/1.1
> User-Agent: curl/7.29.0
> Host: cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-amz-id-2: MD2xjLbEhABFlsWjfSq603Iax30/xmhCofgIBVW2kvkc+vTs1K79Mij1EEj9gL1nR9OrADqM1Bs=
< x-amz-request-id: BFB5FF7E04068788
< Date: Thu, 29 Sep 2016 14:22:27 GMT
< Last-Modified: Thu, 29 Sep 2016 14:22:26 GMT
< x-amz-expiration: expiry-date="Sat, 01 Oct 2016 00:00:00 GMT", rule-id="us-west-1-1-day"
< ETag: "098bdf06f370a606a75637840c4c0b3d"
< x-amz-meta-addresses: [{"code":200,"url":"https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/SDlWxl5OQTOX9u4ZNytuTA/0/public/build/target.reftest.tests.zip","t":"2016-09-29T14:22:23.984Z"}]
< x-amz-meta-url: https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/SDlWxl5OQTOX9u4ZNytuTA/0/public/build/target.reftest.tests.zip
< x-amz-meta-upstream-etag: "098bdf06f370a606a75637840c4c0b3d"
< x-amz-meta-stored: 2016-09-29T14:22:23.985Z
< Accept-Ranges: bytes
< Content-Type: application/zip
< Content-Length: 32102631
< Server: AmazonS3
< 
{ [data not shown]
100 30.6M  100 30.6M    0     0  6087k      0  0:00:05  0:00:05 --:--:-- 19.3M
* Connection #2 to host cloud-mirror-production-us-west-1.s3-us-west-1.amazonaws.com left intact
[ec2-user@ip-172-31-10-63 ~]$ unbzip^C
[ec2-user@ip-172-31-10-63 ~]$ unzip target.reftest.tests.zip 
<SNNIPPPP>
  inflating: reftest/tests/dom/xul/crashtests/extB1.xul  
[ec2-user@ip-172-31-10-63 ~]$ echo $?
0
[ec2-user@ip-172-31-10-63 ~]$ sha1sum target.reftest.tests.zip 
b572e1c3a2bba5f48236ae1b6e497dd25f6f3124  target.reftest.tests.zip
[ec2-user@ip-172-31-10-63 ~]$

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] CET/CEST Berlin Time from comment #6)

ignore this comment, it was meant for another bug

Comment 8

[Pete Moore [:pmoore][:pete]]

Dustin says I can close this.