1576032 - sccache S3 bucket doesn't work in eu-central-1

Status: NEW

(Release Engineering :: Firefox-CI Administration, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

See bug 1562686 comment 69 -- for reasons unknown, sccache cannot write to buckets in eu-central-1. It appears that only windows try has workers in eu-central-1 (so, not linux, and not level 3), so this has flown under the radar.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

From the IAM console:

---

Session Tokens from the STS endpoints

AWS recommends using regional STS endpoints to reduce latency. Session tokens from regional STS endpoints are valid in all AWS Regions. If you use regional STS endpoints, no action is required.

Session tokens from the global STS endpoint (https://sts.amazonaws.com) are valid only in AWS Regions that are enabled by default. If you intend to enable a new Region for your account, you can use session tokens from regional STS endpoints or activate the global STS endpoint to issue session tokens that are valid in all AWS Regions.

---

I bet the right fix here is to enable that.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: image.png

..and I've done that.

Needinfo myself to check the result of https://tools.taskcluster.net/groups/bquElUX0Q9aoVe7b2Uar4w/tasks/ZABH_qE3TJyyZ0EarslLXw/details and see if it cached, and to update the TC docs about this setting, and to file a bug about changing how this is configured so it can use regional endpoints.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/bquElUX0Q9aoVe7b2Uar4w/tasks/Pfx3xuSmS665hlVnDQNVVA/runs/0 specifically.

Comment 4

[Mike Hommey [:glandium]]

Still caches errors in that build, fwiw.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Well, that's disappointing.

Comment 6

[Rob Lemley [:rjl]]

FWIW, This is still happening whenever we hit a worker in eu-central.
https://firefox-ci-tc.services.mozilla.com/tasks/fl8B4NBVTbS3_6toQ7x9sA

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Found in triaging: have we still seen these lately?

Comment 8

[Mike Hommey [:glandium]]

It seems the only jobs that use sccache and would be affected are those running on gecko-1/b-win2012, which still has eu-central-1 as a possible region.

I picked a random non-cross-compiled windows job on try, and it was with no cache hit, and large number of write errors, and guess what, it was running on eu-central. So yes, this is still happening. It's also not that uncommon, cf. https://treeherder.mozilla.org/perf.html?#/graphs?highlightAlerts=1&series=try,2106090,1,2&timerange=1209600