Open Bug 1576032 Opened 5 years ago Updated 10 months ago

sccache S3 bucket doesn't work in eu-central-1

Categories

(Release Engineering :: Firefox-CI Administration, defect)

defect

Tracking

(Not tracked)

People

(Reporter: dustin, Unassigned)

Details

Attachments

(1 file)

See bug 1562686 comment 69 -- for reasons unknown, sccache cannot write to buckets in eu-central-1. It appears that only windows try has workers in eu-central-1 (so, not linux, and not level 3), so this has flown under the radar.

From the IAM console:


Session Tokens from the STS endpoints

AWS recommends using regional STS endpoints to reduce latency. Session tokens from regional STS endpoints are valid in all AWS Regions. If you use regional STS endpoints, no action is required.

Session tokens from the global STS endpoint (https://sts.amazonaws.com) are valid only in AWS Regions that are enabled by default. If you intend to enable a new Region for your account, you can use session tokens from regional STS endpoints or activate the global STS endpoint to issue session tokens that are valid in all AWS Regions.


I bet the right fix here is to enable that.

Attached image image.png

..and I've done that.

Needinfo myself to check the result of https://tools.taskcluster.net/groups/bquElUX0Q9aoVe7b2Uar4w/tasks/ZABH_qE3TJyyZ0EarslLXw/details and see if it cached, and to update the TC docs about this setting, and to file a bug about changing how this is configured so it can use regional endpoints.

Flags: needinfo?(dustin)
Flags: needinfo?(dustin)

Still caches errors in that build, fwiw.

Well, that's disappointing.

Flags: needinfo?(dustin)

FWIW, This is still happening whenever we hit a worker in eu-central.
https://firefox-ci-tc.services.mozilla.com/tasks/fl8B4NBVTbS3_6toQ7x9sA

Found in triaging: have we still seen these lately?

Flags: needinfo?(rob)
Flags: needinfo?(mh+mozilla)

It seems the only jobs that use sccache and would be affected are those running on gecko-1/b-win2012, which still has eu-central-1 as a possible region.

I picked a random non-cross-compiled windows job on try, and it was with no cache hit, and large number of write errors, and guess what, it was running on eu-central. So yes, this is still happening. It's also not that uncommon, cf. https://treeherder.mozilla.org/perf.html?#/graphs?highlightAlerts=1&series=try,2106090,1,2&timerange=1209600

Flags: needinfo?(rob)
Flags: needinfo?(mh+mozilla)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: