1869712 - Intermittent Symsys-symbols-ubuntu [taskcluster:error] Task timeout after 1200 seconds. Force killing container.

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=439902525&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bKjWy5b8T5iOlu3rQKJitg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-12-13T03:24:25.968Z] bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[task 2023-12-13T03:24:25.968Z] + ./mach python toolkit/crashreporter/tools/upload_symbols.py https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W6b_q0ekQ6CTAe9i_JG2-w/artifacts/public/build/target.crashreporter-symbols.zip --ignore-missing
[task 2023-12-13T03:24:27.209Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2023-12-13T03:24:27.287Z] INFO:upload-symbols:Uploading symbol file "https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W6b_q0ekQ6CTAe9i_JG2-w/artifacts/public/build/target.crashreporter-symbols.zip" to "https://symbols.mozilla.org/upload/"
[task 2023-12-13T03:24:27.287Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2023-12-13T03:29:27.559Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT
[task 2023-12-13T03:29:27.559Z] ERROR:upload-symbols:Response body:
[task 2023-12-13T03:29:27.559Z] ====================
[task 2023-12-13T03:29:27.559Z] 
[task 2023-12-13T03:29:27.559Z] ====================
[task 2023-12-13T03:29:27.559Z] 
[task 2023-12-13T03:29:27.559Z] INFO:upload-symbols:Retrying...
[task 2023-12-13T03:29:37.644Z] INFO:upload-symbols:Attempt 2 of 7...
[task 2023-12-13T03:34:37.014Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT
[task 2023-12-13T03:34:37.014Z] ERROR:upload-symbols:Response body:
[task 2023-12-13T03:34:37.014Z] ====================
[task 2023-12-13T03:34:37.014Z] 
[task 2023-12-13T03:34:37.014Z] ====================
[task 2023-12-13T03:34:37.014Z] 
[task 2023-12-13T03:34:37.014Z] INFO:upload-symbols:Retrying...
[task 2023-12-13T03:34:53.361Z] INFO:upload-symbols:Attempt 3 of 7...
[task 2023-12-13T03:39:53.013Z] ERROR:upload-symbols:Error: got HTTP response 504: GATEWAY_TIMEOUT
[task 2023-12-13T03:39:53.013Z] ERROR:upload-symbols:Response body:
[task 2023-12-13T03:39:53.013Z] ====================
[task 2023-12-13T03:39:53.013Z] 
[task 2023-12-13T03:39:53.013Z] ====================
[task 2023-12-13T03:39:53.013Z] 
[task 2023-12-13T03:39:53.013Z] INFO:upload-symbols:Retrying...
[task 2023-12-13T03:40:15.393Z] INFO:upload-symbols:Attempt 4 of 7...
[taskcluster:error] Task timeout after 1200 seconds. Force killing container.

Comment 1

[Cosmin Sabou [:CosminS]]

Weird, it fails only on this merge. On the following merge it was green.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

That file is 2.2GB which is definitely on the big side and more likely to hit timeouts. There's a large number of upload api requests around that time. The cluster scaled up as far as it could go, but I can see cases where the gunicorn worker timed out trying to handle uploads.

This issue is ephemeral and you'll see it when the Mozilla Symbols Server is under heavy load. Having it work fine later is entirely plausible.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving this to the right product. Socorro doesn't do symbols upload--the Mozilla Symbols Server does.

Comment 4

[Intermittent Failures Robot]

2 failures in 4224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• symbols: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 5

[Intermittent Failures Robot]

6 failures in 3217 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform and build breakdown:

• symbols: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 6

[Intermittent Failures Robot]

4 failures in 3599 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• symbols: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 7

[Intermittent Failures Robot]

2 failures in 3952 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• symbols: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 8

[Intermittent Failures Robot]

7 failures in 4336 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• symbols: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-04-22&endday=2024-04-28&tree=all

Comment 9

[Intermittent Failures Robot]

4 failures in 4059 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• symbols: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 10

[Intermittent Failures Robot]

3 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• symbols: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 4625 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't think this is a Tecken bug. I think this is something you'll need to fix in the symsys-symbols-ubuntu task. Maybe increase the timeout because even if it's not retrying, it's still doing more work than the 1200 seconds allows for?

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #12)

I don't think this is a Tecken bug. I think this is something you'll need to fix in the symsys-symbols-ubuntu task. Maybe increase the timeout because even if it's not retrying, it's still doing more work than the 1200 seconds allows for?

Comment 14

[Gabriele Svelto [:gsvelto]]

Indeed, these tasks are uploading a huge amount of symbol files, let's give them more time to do it.

Comment 15

[Gabriele Svelto [:gsvelto]]

Attachment: Bug 1869712 - Increase the maximum duration of system symbol upload tasks to avoid timeouts when encountering 5000+ new symbols at once r=gerard-majax

Comment 16

[Intermittent Failures Robot]

1 failures in 3199 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 3780 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1869712&startday=2024-07-08&endday=2024-07-14&tree=all

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

There is an r+ patch which didn't land and no activity in this bug for 2 weeks.
:gsvelto, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 19

[Gabriele Svelto [:gsvelto]]

Duh, I forgot to land this. Landing it now.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3eac2f1ec013
Increase the maximum duration of system symbol upload tasks to avoid timeouts when encountering 5000+ new symbols at once r=gerard-majax

Comment 21

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/3eac2f1ec013