1818571 - TreeHerder log viewer sometimes doesn't work due to missing CORS header in response from firefoxci.taskcluster-artifacts.net

Status: NEW

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Daniel Holbert [:dholbert]]

Some of the time, when I view https://treeherder.mozilla.org/logviewer?job_id=406575263 , I get an error page:

NetworkError when attempting to fetch resource.
An error occurred attempting to load the provided log.
Please check the URL and ensure it is reachable.
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/eBNYST7uS5GqmbW42HR3CA/runs/0/artifacts/public/logs/live_backing.log

If I check my web console, I see that it was blocked for CORS reasons:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://firefoxci.taskcluster-artifacts.net/eBNYST7uS5GqmbW42HR3CA/0/public/logs/live_backing.log. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 200.

Indeed, if I look in network devtools and compare a successful load vs. an unsuccessful load, I see that the response headers are missing Access-Control-Allow-Origin for cases when the load fails.

Here's the (slightly edited for readability) copy-as-curl command for the request:

curl 'https://firefoxci.taskcluster-artifacts.net/eBNYST7uS5GqmbW42HR3CA/0/public/logs/live_backing.log' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Origin: null' -H 'Connection: keep-alive' -H 'Referer: https://treeherder.mozilla.org/' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -D -

I just ran this twice in my terminal and got two different results for the headers -- one with access-control-allow-origin, one without. So this seems to be a server-side issue where the server is sometimes failing to send that header.

Comment 1

[Daniel Holbert [:dholbert]]

Here are the "bad" response headers I got the first time that I ran the above curl command:

HTTP/2 200 
x-amz-id-2: LBsUj31xXI0txK2H6GsiJUxyyQj2+rnI1HKsqxFJ34pg/mtJIsyP7E9l9LYk3bJCXy5uDMNK+qc=
x-amz-request-id: V9TS4P9EH6V2ZXBN
x-amz-server-side-encryption: AES256
content-encoding: gzip
x-amz-version-id: .2iNhNNhC_wQ7rM0lrOTpAdovz59Ja8O
accept-ranges: bytes
server: AmazonS3
content-length: 80721
via: 1.1 google
date: Thu, 23 Feb 2023 19:04:07 GMT
last-modified: Wed, 22 Feb 2023 21:49:56 GMT
etag: "7097f74ad155bf8eb9e28c1427f3fd37"
content-type: text/plain; charset=utf-8
age: 786
cache-control: public,max-age=604800
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Notice, no access-control-allow-origin header is present. So this response would result in Firefox showing the error page.

Then I ran the exact same curl command again and got this "good" set of response headers, now including access-control-allow-origin: *:

HTTP/2 200 
x-amz-id-2: SU/ub6RaOkA9glHCSYHJAwRuhEpFYj1j8+t7+tXsxrHZKvBI8q39tpTiHHl6AAdkdjWEhjM15F0=
x-amz-request-id: 8HY5S04ZECWXDJ66
date: Thu, 23 Feb 2023 19:17:23 GMT
access-control-allow-origin: *
access-control-allow-methods: GET, PUT, HEAD, POST, DELETE
access-control-max-age: 3600
vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
last-modified: Wed, 22 Feb 2023 21:49:56 GMT
etag: "7097f74ad155bf8eb9e28c1427f3fd37"
x-amz-server-side-encryption: AES256
content-encoding: gzip
x-amz-version-id: .2iNhNNhC_wQ7rM0lrOTpAdovz59Ja8O
accept-ranges: bytes
content-type: text/plain; charset=utf-8
server: AmazonS3
content-length: 80721
via: 1.1 google
cache-control: public,max-age=604800
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Comment 2

[Daniel Holbert [:dholbert]]

Attachment: screenshot of how the error looks in Firefox when the log fails to load

Comment 3

[Daniel Holbert [:dholbert]]

Note:

• I'm using Firefox Nightly 112.0a1 (2023-02-22) (but the browser doesn't matter too much, since this is reproducible using curl as noted above)
• This is extremely fiddly; in my main browsing profile, this is reproducing 100% of the time (when I reload the log, it always hits this error). Whereas in another Firefox profile, this reproduced only once out of 20 attempts. So there may be some randomness involved, and possibly a load balancer that can get stuck returning good/bad results.

Comment 4

[Daniel Holbert [:dholbert]]

Aside from the CORS headers, the only difference between the "good" vs "bad" results are the date and age headers:

• Date header:
  date: Thu, 23 Feb 2023 19:29:58 GMT in headers for "good" response
  date: Thu, 23 Feb 2023 19:04:07 GMT in headers for "bad" response

• Age header:
  ...is missing entirely in headers for "good" response
  age: 786 in above-quoted headers for "bad" response. And age: 1551 in another "bad" response that I just got form doing another run of the curl command.

The MDN page on "Age" suggests that it's an indication that the response is being served from a proxy cache:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Age