Enable live logging for GCP level 1 builders
Categories
(Taskcluster :: Workers, defect)
Tracking
(Not tracked)
People
(Reporter: coop, Assigned: tomprince)
References
Details
Attachments
(1 file)
Per https://bugzilla.mozilla.org/show_bug.cgi?id=1597996#c4, we need live logging enabled in GCP for level 1 for parity with AWS.
|
Reporter | |
Updated•4 years ago
|
|
||
Comment 1•4 years ago
|
||
From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?
|
||
Comment 2•4 years ago
|
||
That's correct, and public IPs (which I think are already in place)
|
|
Reporter | |
Comment 3•4 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #2)
That's correct, and public IPs (which I think are already in place)
Yes, and we can check this pretty easily once the port change is made by checking the logs for any running, tier-3, GCP task.
|
|
||
Comment 4•4 years ago
|
||
Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.
|
|
Reporter | |
Comment 5•4 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #4)
Added firewall exception to fxci-production-level1-workers and fxci-staging-level1-workers.
I just checked, and GCP live logs are still broken:
...complains about a "NetworkError when attempting to fetch resource." The raw log links wants points to:
...which is not found.
|
|
Reporter | |
Comment 6•4 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #1)
From the AWS security groups, it looks like we need port 60023 inbound opened, is that correct?
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
|
|
||
Comment 7•4 years ago
|
||
Sorry, I messed this up -- we need all ports 32768-65535 open.
|
|
||
Comment 9•4 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #6)
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.
|
|
Reporter | |
Comment 10•4 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #8)
Done.
Still can see the live logs. From IRC, we think this is because we're missing the stateless DNS config for the GCP worker pools.
dustin: 11:14 AM https://gecko-3-b-linux-gcp-kizjakhdr2anv058tm17iq.c.fxci-production-level3-workers.internal:32770/log/nkKdtMnVTIWcSRhWg4j4ng that isn't a stateless-dns hostname
that would mean that the worker doesn't have the stateless DNS config
I think that's DNS_SERVER_SECRET, hm
coop 11:15 AM so we'll need new images?
dustin: 11:15 AM no, I think that just needs to get put in the worker-pool config
https://firefox-ci-tc.services.mozilla.com/secrets/worker-pool%3Agecko-3%2Fb-linux probably has that secret, but there's no equivalent gcp secret
(I can't see that secret)
Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)
|
|
Reporter | |
Comment 11•4 years ago
|
||
(In reply to Kendall Libby [:fubar] (he/him) from comment #9)
(In reply to Chris Cooper [:coop] pronoun: he from comment #6)
fubar: I think we want live logging enabled for all tiers and levels. I was getting confused with the interactive tasks which should only ever be available on level 1 builders.
should read all of the bug updates instead of just the last. do we really need this on level 3 workers? having that many ports open to the internet seems like a problem waiting to happen.
Looks like we don't actually have this for level 3 builders right now, so no, let's not enable it for level 3 builders in GCP. Sorry for the churn.
|
|
Reporter | |
Comment 12•4 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #10)
Can I ask someone from releng to please check that secret for the existing (AWS) worker pools and copy/create the value(s) across for the GCP worker pools? NB: we'll need this secret for all levels of builders in GCP (1-3)
Mid-air ate my NI requests for comment #10
|
Assignee | |
Comment 13•4 years ago
|
||
|
||
Updated•4 years ago
|
|
||
Updated•4 years ago
|
|
|
Assignee | |
Comment 14•4 years ago
|
||
I've created the secrets (and the patch above changes our tools to do so by default).
Description
•