1868998 - Perma mozilla-release custom-car [taskcluster:error] Task aborted - max run time exceeded

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=439243313&repo=mozilla-release
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/exdVuUgkRn-w4iEzT0UHbg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-12-08T04:22:30.283Z] [58360/58534] AR obj/chrome/browser/apps/platform_apps/api/libapi_registration.a
[task 2023-12-08T04:22:30.284Z] [58361/58534] STAMP obj/chrome/browser/extensions/api/api.stamp
[task 2023-12-08T04:22:30.289Z] [58362/58534] STAMP obj/chrome/browser/apps/platform_apps/platform_apps.stamp
[task 2023-12-08T04:22:36.468Z] [58363/58534] AR obj/chrome/browser/ui/libui.a
[task 2023-12-08T04:22:36.477Z] [58364/58534] STAMP obj/chrome/browser/renderer_host/history_swiper.stamp
[task 2023-12-08T04:22:36.478Z] [58365/58534] STAMP obj/chrome/browser/updates/announcement_notification/announcement_notification.stamp
[task 2023-12-08T04:22:36.478Z] [58366/58534] STAMP obj/chrome/browser/ui/webui/configs.stamp
[task 2023-12-08T04:22:36.484Z] [58367/58534] STAMP obj/chrome/app_shim/app_shim.stamp
[task 2023-12-08T04:22:40.560Z] [58368/58534] AR obj/chrome/browser/libbrowser.a
[task 2023-12-08T04:26:12.178Z] [58369/58534] LINK ./v8_context_snapshot_generator
[task 2023-12-08T04:26:12.178Z] warning: no debug symbols in executable (-arch x86_64)
[task 2023-12-08T04:26:12.431Z] [58370/58534] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/mac:clang_x64)
[task 2023-12-08T04:26:12.437Z] [58371/58534] STAMP obj/tools/v8_context_snapshot/generate_v8_context_snapshot.stamp
[task 2023-12-08T04:26:12.444Z] [58372/58534] STAMP obj/tools/v8_context_snapshot/v8_context_snapshot.stamp
[task 2023-12-08T04:26:12.451Z] [58373/58534] STAMP obj/chrome/chrome_framework_resources.stamp
[task 2023-12-08T04:26:12.614Z] [58374/58534] AR obj/chrome/renderer/librenderer.a
[task 2023-12-08T04:26:12.621Z] [58375/58534] STAMP obj/chrome/dependencies.stamp
[task 2023-12-08T04:26:12.784Z] [58376/58534] AR obj/headless/libheadless_non_renderer.a
[task 2023-12-08T04:26:12.806Z] [58377/58534] AR obj/headless/libheadless_shell_lib.a
[task 2023-12-08T04:26:12.813Z] [58378/58534] STAMP obj/chrome/chrome_dll.stamp
[taskcluster 2023-12-08T04:37:08.447Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/exdVuUgkRn-w4iEzT0UHbg/0/on/mdc1/macmini-r8-198/until/1702011428.457
[taskcluster:error] Aborting task...
[taskcluster:warn 2023-12-08T04:44:37.061Z] no such process
[taskcluster 2023-12-08T04:44:37.061Z] === Task Finished ===
[taskcluster 2023-12-08T04:44:37.061Z] Task Duration: 8h20m0.918629s
[taskcluster:error] Uploading error artifact public/build from file public/build with message "Could not read directory '/opt/worker/tasks/task_169748250232954/public/build'", reason "file-missing-on-worker" and expiry 2024-03-06T18:04:27.443Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/opt/worker/tasks/task_169748250232954/public/build'
[taskcluster 2023-12-08T04:44:38.323Z] Uploading artifact public/logs/certified.log from file /opt/worker/tasks/task_169748250232954/generic-worker/certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2024-03-06T18:04:27.443Z
[taskcluster 2023-12-08T04:44:39.019Z] Uploading artifact public/chain-of-trust.json from file /opt/worker/tasks/task_169748250232954/generic-worker/chain-of-trust.json with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2024-03-06T18:04:27.443Z
[taskcluster 2023-12-08T04:44:39.356Z] Uploading artifact public/chain-of-trust.json.sig from file /opt/worker/tasks/task_169748250232954/generic-worker/chain-of-trust.json.sig with content encoding "gzip", mime type "application/octet-stream" and expiry 2024-03-06T18:04:27.443Z
[taskcluster 2023-12-08T04:44:39.588Z] [mounts] Preserving cache: Moving "/opt/worker/tasks/task_169748250232954/checkouts" to "/opt/worker/cache/MoCaUrYPRpCz8IJ3NQZA3w"
[taskcluster 2023-12-08T04:44:39.588Z] [mounts] Denying task_169748250232954 access to '/opt/worker/cache/MoCaUrYPRpCz8IJ3NQZA3w'
[taskcluster 2023-12-08T04:44:39.673Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2024-03-06T18:04:27.443Z
[taskcluster:error] Task aborted - max run time exceeded

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

AFAIK this task shouldn't be running on mozilla-release
Why was this triggered on mozilla-release?
(and fwiw, this time out was resolved in https://bugzilla.mozilla.org/show_bug.cgi?id=1867382 which is in mozilla central)

:cosmin do you happen to know why this would was triggered on mozilla-release?

Comment 2

[Intermittent Failures Robot]

2 failures in 3671 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2

Platform and build breakdown:

• toolchains: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1868998&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 3

[Julien Cristau [:jcristau]]

As part of bug 1868558, all cached tasks are being rebuilt, which includes toolchains.

Comment 4

[Mike Hommey [:glandium]]

(In reply to Julien Cristau [:jcristau] from comment #3)

As part of bug 1868558, all cached tasks are being rebuilt, which includes toolchains.

This makes me wonder: are tasks with the cache_task attribute set to false skipped? (I would assume so). Leading to the question: should me set the attribute to false on the custom-car tasks?

Comment 5

[Julien Cristau [:jcristau]]

(In reply to Mike Hommey [:glandium] from comment #4)

(In reply to Julien Cristau [:jcristau] from comment #3)

As part of bug 1868558, all cached tasks are being rebuilt, which includes toolchains.

This makes me wonder: are tasks with the cache_task attribute set to false skipped? (I would assume so).

Yes: https://searchfox.org/mozilla-central/rev/69bcb46e73b6a30ee9261cedead161ae7a9a8051/taskcluster/gecko_taskgraph/actions/rebuild_cached_tasks.py#27

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1868998 - Mark custom-car jobs as running nowhere and not being cached.

They are triggered via cron, so an empty run-on-projects should be the
appropriate config.

As they are not idempotent (they will pick the last from upstream on
each run), they also shouldn't be cached.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ab9b9a557a9e
Mark custom-car jobs as running nowhere and not being cached. r=taskgraph-reviewers,kshampur,jcristau

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/ab9b9a557a9e

Comment 9

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/1d19517b592a

Comment 10

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/3c276b433666

Comment 11

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/b6fde2e8b099