1829850 - Permanent failure line for successful Talos tasks: FATAL ERROR: Non-local network connections are disabled and a connection attempt to incoming.telemetry.mozilla.org (34.120.208.123) was made.

Status: RESOLVED FIXED

(Core :: DOM: Security, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=413701779&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CQL2rc4BRp-hq_47Yibm-Q/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-04-25T12:04:33.168Z] 12:04:33     INFO -  PID 8520 | *** You are running in headless mode.
[task 2023-04-25T12:04:33.246Z] 12:04:33     INFO -  PID 8520 | console.error: "C:\\Users\\task_1682422729\\AppData\\Local\\Temp\\tmp3s2qqb_e\\profile\\storage" "to-be-removed" 0 "" "Quota"
[task 2023-04-25T12:04:33.249Z] 12:04:33     INFO -  PID 8520 | console.error: "started removing C:\\Users\\task_1682422729\\AppData\\Local\\Temp\\tmp3s2qqb_e\\profile\\storage\\to-be-removed"
[task 2023-04-25T12:04:33.250Z] 12:04:33     INFO -  PID 8520 | console.error: "done removing C:\\Users\\task_1682422729\\AppData\\Local\\Temp\\tmp3s2qqb_e\\profile\\storage\\to-be-removed. removed 1 entries."
[task 2023-04-25T12:04:33.250Z] 12:04:33     INFO -  PID 8520 | console.error: "Pinged glean, waiting for submission."
[task 2023-04-25T12:04:33.305Z] 12:04:33     INFO -  PID 8520 | FATAL ERROR: Non-local network connections are disabled and a connection attempt to incoming.telemetry.mozilla.org (34.120.208.123) was made.

Comment 1

[Frederik Braun [:freddy]]

I can't reproduce due to talos tests not running locally :(

can't open file 'gecko-https-first/testing/mozharness/external_tools/tooltool.py': [Errno 2] No such file or directory

Comment 2

[Frederik Braun [:freddy]]

Jan-Erik helped me look at this. Might be this is an issue from bug 1788986, where the talos specific pref override to not use the production server is not properly mirrored into the QuotaManager background task.

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

It does follow the user preference to disable telemetry, but I guess Talos still needs the telemetry? In that case do we have some documentation about what to do here?

Comment 4

[Kagami Rosylight [:saschanaz] (they/them)]

Actually Jan-Erik may have better idea for the docs?

Comment 5

[Jan-Erik Rediger [:janerik]]

The Talos config sets the legacy telemetry endpoint to localhost: https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js#82
And also sets Glean to act as if enabled, but not sending out things: https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js#83

Is the latter pref also set in the background task? (I don't know exactly how background tasks get their preference, do they just load the full profile or does it need explicit mirroring of a subset of prefs?)

Comment 6

[Kagami Rosylight [:saschanaz] (they/them)]

It does not receive any specific prefs for now, it only receives the parameter whether to send telemetry or not. I'm not sure it's any worth to pass the Talos config in this case, perhaps it can just disable the telemetry? Does it make sense?

Comment 7

[Jan-Erik Rediger [:janerik]]

I don't know Talos enough, but I can imagine it wants to keep Telemetry on so that it tests the performance of the app as close as possible to real usage.
Whether that means background tasks should also keep it on, but not sending, is then up to you and Talos folks.

Comment 8

[Kagami Rosylight [:saschanaz] (they/them)]

Makes sense. Passing the question to the current Talos triage owner.

Comment 9

[Intermittent Failures Robot]

27 failures in 673 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 6
• autoland: 11
• try: 3
• mozilla-beta: 7

Platform and build breakdown:

• linux1804-64-shippable-qr: 25
  • opt: 25
• windows10-64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1829850&startday=2023-04-27&endday=2023-04-27&tree=all

Comment 10

[Intermittent Failures Robot]

51 failures in 3177 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 18
• mozilla-beta: 12
• mozilla-central: 12
• try: 9

Platform and build breakdown:

• windows10-64-shippable-qr: 5
  • opt: 5
• linux1804-64-shippable-qr: 43
  • opt: 43
• linux1804-64-qr: 1
  • opt: 1
• linux1804-64-devedition-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1829850&startday=2023-04-24&endday=2023-04-30&tree=all

Comment 11

[Andrej (:aglavic)]

investigating the issue will respond when I have some more information

Comment 12

[Kagami Rosylight [:saschanaz] (they/them)]

(It's about having telemetry from asynchronous cleanup that happens after Firefox closes. I'm not sure this is in the scope of Talos)

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I am 99% sure that we do not want to collect metrics from talos (or other CI test/perf jobs). There might be an exception or two.

Comment 14

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: Bug 1829850 - Disable telemetry from removeDirectory in performance tests r=nalexander!,jmaher!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a53000efdd8c
Disable telemetry from removeDirectory in performance tests r=nalexander,jmaher

Comment 16

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a53000efdd8c

Comment 17

[Intermittent Failures Robot]

1 failures in 3411 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1829850&startday=2023-05-01&endday=2023-05-07&tree=all