1910834 - Perma [tier 2] tgdiff [taskcluster:error] Task timeout after 1800 seconds. Force killing container.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: tszentpeteri [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=468569431&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PtIF9aSKSZi_YUUMjBOUIA/runs/0/artifacts/public/logs/live_backing.log

---

[vcs 2024-07-30T19:21:03.180Z] updating [================================================> ] 370745/372179 01s
[vcs 2024-07-30T19:21:05.066Z]                                                                                 
[vcs 2024-07-30T19:21:05.066Z] 372179 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2024-07-30T19:21:05.208Z] updated to b0b107c799b5bd4c694a4f3c70a058e2e0a9c8d2
[vcs 2024-07-30T19:21:05.208Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 95.27965903282166}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "update", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 91.6293933391571}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 187.706387758255}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 187.706387758255}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 187.706387758255}, {"extraOptions": ["projects/887720501152/machineTypes/n2-standard-4"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 187.706387758255}]}
[vcs 2024-07-30T19:21:05.538Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/autoland/rev/b0b107c799b5bd4c694a4f3c70a058e2e0a9c8d2 title='Built from autoland revision b0b107c799b5bd4c694a4f3c70a058e2e0a9c8d2'>b0b107c799b5bd4c694a4f3c70a058e2e0a9c8d2</a>
[setup 2024-07-30T19:21:05.538Z] GECKO_PATH is /builds/worker/checkouts/gecko
[setup 2024-07-30T19:21:05.538Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[setup 2024-07-30T19:21:05.538Z] MOZ_PYTHON_HOME is /builds/worker/fetches/python
[fetches 2024-07-30T19:21:05.538Z] fetching artifacts
[fetches 2024-07-30T19:21:05.538Z] executing ['/usr/bin/python3', '-u', '/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content', 'task-artifacts']
[fetches 2024-07-30T19:21:05.618Z] attempt 1/5
[fetches 2024-07-30T19:21:05.618Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bo-Sb3qaQZSJk4OGMh1IOw/artifacts/public/build/python.tar.zst to /builds/worker/fetches/python.tar.zst
[fetches 2024-07-30T19:21:05.618Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bo-Sb3qaQZSJk4OGMh1IOw/artifacts/public/build/python.tar.zst
[fetches 2024-07-30T19:21:06.156Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bo-Sb3qaQZSJk4OGMh1IOw/artifacts/public/build/python.tar.zst resolved to 104420247 bytes with sha256 254b78b70cfd47dc35b014e58df4ab3b6584ca010845263b6f7c483c0a35ff3c in 0.538s
[fetches 2024-07-30T19:21:06.156Z] Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bo-Sb3qaQZSJk4OGMh1IOw/artifacts/public/build/python.tar.zst
[fetches 2024-07-30T19:21:06.156Z] Extracting /builds/worker/fetches/python.tar.zst to /builds/worker/fetches
[fetches 2024-07-30T19:21:07.014Z] /builds/worker/fetches/python.tar.zst extracted in 0.858s
[fetches 2024-07-30T19:21:07.014Z] Removing /builds/worker/fetches/python.tar.zst
[fetches 2024-07-30T19:21:07.030Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 1.413868109999953, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-07-30T19:21:07.044Z] finished fetching artifacts
[setup 2024-07-30T19:21:07.044Z] Setting up local python environment
[setup 2024-07-30T19:21:07.044Z] updated PATH with python artifact: /builds/worker/fetches/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[task 2024-07-30T19:21:07.044Z] executing ['bash', '-cx', 'mkdir -p /builds/worker/artifacts/diffs && ./mach taskgraph target --fast -p "taskcluster/test/params" --force-local-files-changed -o "/builds/worker/artifacts/diffs/diff.txt" --diff fc3b217a4880db888add63d88c4b7027c888d619 && python taskcluster/scripts/misc/summarize-tgdiff.py /builds/worker/artifacts/diffs/ 20']in /builds/worker/checkouts/gecko
[task 2024-07-30T19:21:07.046Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[task 2024-07-30T19:21:07.046Z] + mkdir -p /builds/worker/artifacts/diffs
[task 2024-07-30T19:21:07.048Z] + ./mach taskgraph target --fast -p taskcluster/test/params --force-local-files-changed -o /builds/worker/artifacts/diffs/diff.txt --diff fc3b217a4880db888add63d88c4b7027c888d619
[task 2024-07-30T19:21:10.798Z] Creating default state directory: /builds/worker/.mozbuild
[task 2024-07-30T19:21:10.798Z] Creating local state directory: /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69
[task 2024-07-30T19:21:10.798Z] Generating target_task_set @ b0b107c799b5
[task 2024-07-30T19:32:55.396Z] Looking for existing index gecko.v2.mozilla-central.revision.d9466bebb4c96b99a227f35b5a7e86bd0c8f5bb5.taskgraph.decision-nightly-all to avoid triggering multiple nightlies off the same revision...
[task 2024-07-30T19:32:55.396Z] Index gecko.v2.mozilla-central.revision.d9466bebb4c96b99a227f35b5a7e86bd0c8f5bb5.taskgraph.decision-nightly-all exists: taskId bAuTiJV6T1WjfKxWzyWFOg
[task 2024-07-30T19:32:55.396Z] Looking for existing index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-desktop to avoid triggering multiple nightlies off the same revision...
[task 2024-07-30T19:32:55.396Z] Index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-desktop exists: taskId V2cNf7S3QFKQ319PxsrbQA
[task 2024-07-30T19:32:55.396Z] Looking for existing index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-android to avoid triggering multiple nightlies off the same revision...
[task 2024-07-30T19:32:55.396Z] Index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-android doesn't exist.
[task 2024-07-30T19:32:55.396Z] Looking for existing index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-all to avoid triggering multiple nightlies off the same revision...
[task 2024-07-30T19:32:55.396Z] Index gecko.v2.mozilla-central.revision.ed20a8b98a8f2d8593dbbbd9b5df993e3e91433f.taskgraph.decision-nightly-all doesn't exist.
[task 2024-07-30T19:32:59.911Z] Generating target_task_set @ fc3b217a4880
[taskcluster:error] Task timeout after 1800 seconds. Force killing container.
[taskcluster 2024-07-30 19:47:57.361Z] === Task Finished ===
[taskcluster 2024-07-30 19:47:57.362Z] Unsuccessful task run with exit code: -1 completed in 1800.889 seconds

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1910179

:ahal, since you are the author of the regressor, bug 1910179, could you take a look?

For more information, please visit BugBot documentation.

Comment 2

[Intermittent Failures Robot]

35 failures in 3839 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #41 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 14
• mozilla-central: 21

Platform and build breakdown:

• gecko-decision: 35
  • opt: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1910834&startday=2024-07-29&endday=2024-08-04&tree=all

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1910179

Comment 4

[Intermittent Failures Robot]

47 failures in 3459 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #36 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-central: 20
• mozilla-beta: 2

Platform and build breakdown:

• gecko-decision: 47
  • opt: 47

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1910834&startday=2024-08-05&endday=2024-08-11&tree=all

Comment 5

[Intermittent Failures Robot]

23 failures in 2154 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 15
• autoland: 8

Platform and build breakdown:

• gecko-decision: 23
  • opt: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1910834&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 6

[Julien Cristau [:jcristau]]

Attachment: Bug 1910834 - don't reload `taskgraph.main` in `mach taskgraph --diff`. r?#releng

Bug 1910179 caused gecko_taskgraph.main to use logic from taskgraph.main,
including the format_taskgraph function, that gets passed to subprocesses
when generating task graphs in parallel.
Reloading the taskgraph.main module means the original format_taskgraph
function is no longer pickleable:

"""
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/queues.py", line 264, in _feed
obj = _ForkingPickler.dumps(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function format_taskgraph at 0x7f47ab1cbce0>: it's not the same object as taskgraph.main.format_taskgraph
"""

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b476832986a
don't reload `taskgraph.main` in `mach taskgraph --diff`. r=releng-reviewers,taskgraph-reviewers,hneiva

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/3b476832986a

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:jcristau, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox130 to wontfix.

For more information, please visit BugBot documentation.

Comment 10

[Julien Cristau [:jcristau]]

Attachment: Bug 1910834 - don't reload `taskgraph.main` in `mach taskgraph --diff`. r?#releng

Bug 1910179 caused gecko_taskgraph.main to use logic from taskgraph.main,
including the format_taskgraph function, that gets passed to subprocesses
when generating task graphs in parallel.
Reloading the taskgraph.main module means the original format_taskgraph
function is no longer pickleable:

"""
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/queues.py", line 264, in _feed
obj = _ForkingPickler.dumps(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function format_taskgraph at 0x7f47ab1cbce0>: it's not the same object as taskgraph.main.format_taskgraph
"""

Original Revision: https://phabricator.services.mozilla.com/D219416

Comment 11

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: n/a
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: test-only
• String changes made/needed: none
• Is Android affected?: no

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/e71bc8b41a31

Comment 13

[Intermittent Failures Robot]

3 failures in 3218 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform and build breakdown:

• gecko-decision: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1910834&startday=2024-08-19&endday=2024-08-25&tree=all