1905466 - Perma system-symbols-fedora [taskcluster:error] Task timeout after 1200 seconds. Force killing container.

Status: NEW

(Tecken :: Upload, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=464497785&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZmVma8XfRFOA9hcQmw4Bmw/runs/3/artifacts/public/logs/live_backing.log

---

[vcs 2024-06-29T04:16:38.401Z] clone [=======================================>     ] 5070747857/5675006431 08s
[vcs 2024-06-29T04:16:39.401Z] clone [=========================================>   ] 5383970771/5675006431 04s
[vcs 2024-06-29T04:16:39.420Z] clone [===========================================> ] 5668686063/5675006431 01s
[vcs 2024-06-29T04:16:39.420Z]                                                                                 
[vcs 2024-06-29T04:16:39.420Z] transferred 5.29 GB in 67.1 seconds (80.6 MB/sec)
[vcs 2024-06-29T04:16:40.045Z] finished applying clone bundle
[vcs 2024-06-29T04:16:40.529Z] searching for changes
[vcs 2024-06-29T04:16:41.121Z] adding changesets
[vcs 2024-06-29T04:16:41.269Z] adding manifests
[vcs 2024-06-29T04:16:43.154Z] 
[vcs 2024-06-29T04:16:45.601Z] manifests [                                                     ]   2/185 5m45s
[vcs 2024-06-29T04:16:47.671Z] manifests [>                                                    ]   4/185 4m22s
[vcs 2024-06-29T04:16:51.518Z] manifests [=======>                                               ]  27/185 39s
[vcs 2024-06-29T04:16:51.534Z] manifests [=======>                                             ]  28/185 1m00s
[vcs 2024-06-29T04:16:51.534Z]                                                                                 
[vcs 2024-06-29T04:16:51.535Z] adding file changes
[vcs 2024-06-29T04:16:53.417Z] added 185 changesets with 1550 changes to 1316 files
[vcs 2024-06-29T04:16:53.417Z] new changesets b6a4670d77b8:2ede043748da
[vcs 2024-06-29T04:16:57.426Z] searching for changes
[vcs 2024-06-29T04:16:57.511Z] no changes found
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark aurora
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark autoland
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark beta
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark central
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr10
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr102
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr115
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr128
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr17
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr24
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr31
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr38
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr45
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr52
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr60
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr68
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr78
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark esr91
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark fx-team
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark inbound
[vcs 2024-06-29T04:16:57.826Z] adding remote bookmark release
[vcs 2024-06-29T04:16:58.297Z] (setting sparse config to profile build/sparse-profiles/upload-symbols)
[vcs 2024-06-29T04:16:58.301Z] 0 files added, 0 files dropped, 0 files conflicting
[vcs 2024-06-29T04:16:58.301Z] (sparse refresh complete)
[vcs 2024-06-29T04:17:00.634Z] warning: sparse profile 'build/sparse-profiles/upload-symbols' not found in rev 000000000000 - ignoring it
[vcs 2024-06-29T04:17:02.469Z] 
[vcs 2024-06-29T04:17:02.817Z] updating [                                                          ]  100/6420
[vcs 2024-06-29T04:17:02.853Z]                                                                                 
[vcs 2024-06-29T04:17:02.853Z] 6420 files updated, 0 files merged, 0 files removed, 0 files unresolved
[vcs 2024-06-29T04:17:02.953Z] updated to 30e83f26e442b92b8959e2ab876b25015c0f4012
[vcs 2024-06-29T04:17:02.954Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "clone", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 86.35345935821533}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "sparse_update_config", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 0.003228425979614258}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "update_sparse", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 4.652248859405518}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 92.63479781150818}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 92.63479781150818}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_fullcheckout", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 92.63479781150818}, {"extraOptions": ["projects/970387039909/machineTypes/c2-standard-16"], "hgVersion": "6.4.3", "lowerIsBetter": true, "name": "overall_pull_emptywdir", "serverUrl": "hg.mozilla.org", "shouldAlert": false, "subtests": [], "value": 92.63479781150818}]}
[vcs 2024-06-29T04:17:03.231Z] TinderboxPrint:<a href=https://hg.mozilla.org/mozilla-central/rev/30e83f26e442b92b8959e2ab876b25015c0f4012 title='Built from mozilla-central revision 30e83f26e442b92b8959e2ab876b25015c0f4012'>30e83f26e442b92b8959e2ab876b25015c0f4012</a>
[setup 2024-06-29T04:17:03.231Z] GECKO_PATH is /builds/worker/checkouts/gecko
[task 2024-06-29T04:17:03.231Z] executing ['bash', '-cx', './mach python toolkit/crashreporter/tools/upload_symbols.py --task-id WvPUSo3MTMSscfVEe84fpQ --ignore-missing']in /builds/worker/checkouts/gecko
[task 2024-06-29T04:17:03.233Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[task 2024-06-29T04:17:03.233Z] + ./mach python toolkit/crashreporter/tools/upload_symbols.py --task-id WvPUSo3MTMSscfVEe84fpQ --ignore-missing
[task 2024-06-29T04:17:05.246Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:17:05.262Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.0.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:17:05.262Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-06-29T04:20:38.988Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-06-29T04:20:38.989Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:20:39.012Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.1.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:20:39.012Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-06-29T04:24:14.054Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-06-29T04:24:14.055Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:24:14.082Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.2.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:24:14.082Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-06-29T04:28:07.524Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-06-29T04:28:07.526Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:28:07.548Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.3.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:28:07.548Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-06-29T04:31:22.058Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-06-29T04:31:22.059Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:31:22.080Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.4.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:31:22.080Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2024-06-29T04:34:36.165Z] INFO:upload-symbols:Uploaded successfully!
[task 2024-06-29T04:34:36.166Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2024-06-29T04:34:36.209Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/WvPUSo3MTMSscfVEe84fpQ/artifacts/public/build/target.crashreporter-symbols.5.zip" to "https://symbols.mozilla.org/upload/"
[task 2024-06-29T04:34:36.209Z] INFO:upload-symbols:Attempt 1 of 7...
[taskcluster:error] Task timeout after 1200 seconds. Force killing container.
[taskcluster 2024-06-29 04:35:27.801Z] === Task Finished ===
[taskcluster 2024-06-29 04:35:27.802Z] Unsuccessful task run with exit code: -1 completed in 1221.695 seconds

Comment 1

[Cristian Tuns]

Hi, can you please take a look at this?
Might be similar with Bug 1869712?

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

From the logs, there's no indication of how many symbols files uploads it's planning to do, but it looks like it had more work to do than 1200 seconds allotted for. None of the symbol uploads it did failed or had to be retried.

Bug #1869712 looks like the upload is failing and needs to be retried over and over again until it exceeds the task timeout. I don't know if that's the same or different. It'd be nice if the size of each symbols zip file was echoed to the log as well as how many zip files there were.

I don't think either of these are Tecken bugs. I think this is something you'll need to fix in the system-symbols-fedora task. Maybe increase the timeout?

Comment 3

[Cosmin Sabou [:CosminS]]

The previous job had turned green after 2 mins here - https://treeherder.mozilla.org/logviewer?job_id=464499250&repo=mozilla-central
Similar jobs shows that this task normally takes under 5 mins to finish, in some rare cases actually goes to 10 min. I don't think increasing the task timeout is gonna solve the problem, it's just sweeping it under the rug.
In the failed run I see hitting: ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT several times, green runs don't have that: https://treeherder.mozilla.org/logviewer?job_id=464354790&repo=mozilla-central.
It needs further investigation as it's permafailing atm.

Comment 4

[Cosmin Sabou [:CosminS]]

Maybe an issue with Mozilla Symbols Server as stated here: https://bugzilla.mozilla.org/show_bug.cgi?id=1869712#c3

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

If I recall, both the system-symbols-fedora and system-symbols-ubuntu tasks do wildly different amounts of work from run to run. It's likely some runs have more work than it has 1200 seconds for, but I think it self-heals the next run.

I think it's worth spending time improving the tasks to improve logging output and maybe adjusting how much work they do or how they break up the work across runs.

Comment 6

[Intermittent Failures Robot]

4 failures in 4116 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• symbols: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905466&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 7

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Table

	no_variant
linux-x86/unknown build	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905466&startday=2025-05-12&endday=2025-05-18&tree=all

Comment 8

[Intermittent Failures Robot]

3 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Table

	no_variant
linux64-x86/opt	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905466&startday=2025-07-21&endday=2025-07-27&tree=all

Comment 9

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2

Table

	no_variant
linux1804-x86_64/debug	1
linux1804-x86_64/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905466&startday=2026-03-02&endday=2026-03-08&tree=all