1889836 - No decision tasks and cron task failures across multiple trees

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Cristian Tuns]

Link with no decisions tasks: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&tochange=cfe7bdcc2e1235e33fd8408b337be8d36e22b066&fromchange=89c3b4480a815f8d9acb1e61fdc8344bceba6e3f
Link with cron failures: https://firefox-ci-tc.services.mozilla.com/hooks/#project-releng/cron-task-mozilla-central. Task: https://firefox-ci-tc.services.mozilla.com/tasks/YRqkNmxdSoWcg_VhI_nkZQ

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This is closing all trees at the moment.

Comment 2

[Kelsey Gilbert [:jgilbert]]

Suspiciously, this is also from today: bug 1889764.

Comment 3

[Julien Cristau [:jcristau]]

Both this and bug 1889764 show connection timeouts between CI workers in gcp and pypi/hg in mdc1.

This could be either some kind of network/routing outage, or an issue with the load balancer that's fronting pypi and hg?

Comment 4

[Julien Cristau [:jcristau]]

Attachment: failed build-decision task log

Comment 5

[Julien Cristau [:jcristau]]

Attachment: failed build-decision task log

Attaching a sample failure log for the record.

Comment 6

[Julien Cristau [:jcristau]]

Andrew, any ideas if something on the gcp side could explain why some workers can't connect to services in mdc1?

Comment 7

[Andrew Erickson [:aerickson]]

Turned out to be a Mozilla firewall change done by the Security team. Post-mortem being tracked at https://mozilla-hub.atlassian.net/browse/RELOPS-917.