1913546 - Perma Symbols [tier 2] [taskcluster:error] Task timeout after 1800 seconds. Force killing container.

Status: RESOLVED INCOMPLETE

(Firefox Build System :: Task Configuration, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=470690032&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LlxDo_n2TfWLyJUjfbssbg/runs/0/artifacts/public/logs/live_backing.log

---

[taskcluster 2024-08-16 16:19:56.834Z] === Task Starting ===
[setup 2024-08-16T16:19:58.793Z] run-task started in /builds/worker
[cache 2024-08-16T16:19:58.795Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-08-16T16:19:58.795Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2024-08-16T16:19:58.795Z] volume /builds/worker/checkouts is a cache
[setup 2024-08-16T16:19:58.795Z] running as worker:worker
[vcs 2024-08-16T16:19:58.796Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2024-08-16T16:19:58.874Z] region google/us-central1 not yet supported; using public hg.mozilla.org service
[vcs 2024-08-16T16:19:58.875Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--sparseprofile', 'build/sparse-profiles/upload-symbols', '--revision', '512e5aae19cf44bb81758fb79d66237ae04fc90d', 'https://hg.mozilla.org/integration/autoland', '/builds/worker/checkouts/gecko']
[vcs 2024-08-16T16:19:59.007Z] (using Mercurial 6.8.1)
[vcs 2024-08-16T16:19:59.007Z] ensuring https://hg.mozilla.org/integration/autoland@512e5aae19cf44bb81758fb79d66237ae04fc90d is available at /builds/worker/checkouts/gecko
[vcs 2024-08-16T16:19:59.008Z] (existing repository shared store: /builds/worker/checkouts/hg-store/8ba995b74e18334ab3707f27e9eb8f4e37ba3d29/.hg)
[vcs 2024-08-16T16:19:59.865Z] (purging working directory)
[vcs 2024-08-16T16:20:02.608Z] (replacing existing sparse config with profile build/sparse-profiles/upload-symbols)
[vcs 2024-08-16T16:20:03.833Z] 1 files added, 31 files dropped, 0 files conflicting
[vcs 2024-08-16T16:20:03.834Z] (sparse refresh complete)
[vcs 2024-08-16T16:20:03.995Z] 34 files updated, 0 files merged, 7 files removed, 0 files unresolved
[vcs 2024-08-16T16:20:04.087Z] updated to 512e5aae19cf44bb81758fb79d66237ae04fc90d
[vcs 2024-08-16T16:20:04.087Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "purge", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 2.445969343185425}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "sparse_update_config", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 1.2245607376098633}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "update_sparse", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.2525784969329834}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 5.079995393753052}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "overall_nopull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 5.079995393753052}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "overall_nopull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 5.079995393753052}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.8.1", "lowerIsBetter": true, "name": "overall_nopull_populatedwdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 5.079995393753052}]}
[vcs 2024-08-16T16:20:04.382Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/autoland/rev/512e5aae19cf44bb81758fb79d66237ae04fc90d title='Built from autoland revision 512e5aae19cf44bb81758fb79d66237ae04fc90d'>512e5aae19cf44bb81758fb79d66237ae04fc90d</a>
[setup 2024-08-16T16:20:04.382Z] GECKO_PATH is /builds/worker/checkouts/gecko
[task 2024-08-16T16:20:04.382Z] executing ['bash', '-cx', './mach python toolkit/crashreporter/tools/upload_symbols.py --task-id bkelEPd-S_6NatYWAZitew --ignore-missing']in /builds/worker/checkouts/gecko
[task 2024-08-16T16:20:04.383Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[task 2024-08-16T16:20:04.384Z] + ./mach python toolkit/crashreporter/tools/upload_symbols.py --task-id bkelEPd-S_6NatYWAZitew --ignore-missing
[task 2024-08-16T16:20:06.189Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-08-16T16:20:06.269Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bkelEPd-S_6NatYWAZitew/artifacts/public/build/target.crashreporter-symbols.0.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-08-16T16:20:06.269Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-08-16T16:25:06.295Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT
[task 2024-08-16T16:25:06.295Z] ERROR:upload-symbols:Response body:
[task 2024-08-16T16:25:06.295Z] ====================
[task 2024-08-16T16:25:06.295Z] 
[task 2024-08-16T16:25:06.295Z] ====================
[task 2024-08-16T16:25:06.295Z] 
[task 2024-08-16T16:25:06.295Z] INFO:upload-symbols:Retrying...
[task 2024-08-16T16:25:15.751Z] INFO:upload-symbols:Attempt 2 of 7...
[task 2024-08-16T16:28:55.702Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-08-16T16:28:55.703Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-08-16T16:28:55.768Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bkelEPd-S_6NatYWAZitew/artifacts/public/build/target.crashreporter-symbols.1.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-08-16T16:28:55.768Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-08-16T16:33:52.914Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-08-16T16:33:52.915Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-08-16T16:33:53.000Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bkelEPd-S_6NatYWAZitew/artifacts/public/build/target.crashreporter-symbols.2.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-08-16T16:33:53.000Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-08-16T16:38:52.296Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT
[task 2024-08-16T16:38:52.296Z] ERROR:upload-symbols:Response body:
[task 2024-08-16T16:38:52.296Z] ====================
[task 2024-08-16T16:38:52.296Z] 
[task 2024-08-16T16:38:52.296Z] ====================
[task 2024-08-16T16:38:52.296Z] 
[task 2024-08-16T16:38:52.296Z] INFO:upload-symbols:Retrying...
[task 2024-08-16T16:39:02.899Z] INFO:upload-symbols:Attempt 2 of 7...
[task 2024-08-16T16:44:00.220Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-08-16T16:44:00.221Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-08-16T16:44:00.283Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bkelEPd-S_6NatYWAZitew/artifacts/public/build/target.crashreporter-symbols.3.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-08-16T16:44:00.283Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-08-16T16:48:51.989Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-08-16T16:48:51.990Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-08-16T16:48:52.052Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bkelEPd-S_6NatYWAZitew/artifacts/public/build/target.crashreporter-symbols.4.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-08-16T16:48:52.052Z] INFO:upload-symbols:Attempt 1 of 7...
[taskcluster:error] Task timeout after 1800 seconds. Force killing container.
[taskcluster 2024-08-16 16:49:57.076Z] === Task Finished ===
[taskcluster 2024-08-16 16:49:57.076Z] Unsuccessful task run with exit code: -1 completed in 1821.951 seconds

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Task Configuration' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1913154

:saschanaz, since you are the author of the regressor, bug 1913154, could you take a look?

For more information, please visit BugBot documentation.

Comment 3

[Kagami Rosylight [:saschanaz] (they/them) (OOO until 2025-03-29)]

[task 2024-08-16T16:33:53.000Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-08-16T16:38:52.296Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT

This is not a code regression.

Comment 4

[Intermittent Failures Robot]

6 failures in 2154 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform and build breakdown:

• symbols: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1913546&startday=2024-08-12&endday=2024-08-18&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.