1751303 - Intermittent Symbols [taskcluster:error] Task timeout after 900 seconds. Force killing container. | The write operation timed out

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=364889175&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KlSxNSlLQKKl721zYUdOKQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-01-20T22:10:07.958Z] INFO:upload-symbols:Using symbol upload token from the secrets service: "http://taskcluster/secrets/v1/secret/project/releng/gecko/build/level-3/gecko-symbol-upload"
[task 2022-01-20T22:10:08.118Z] INFO:upload-symbols:Uploading symbol file "/tmp/tmpmspf_syl/symbols.zip" to "https://symbols.mozilla.org/upload/"
[task 2022-01-20T22:10:08.118Z] INFO:upload-symbols:Attempt 1 of 7...
[task 2022-01-20T22:15:12.427Z] ERROR:upload-symbols:Error: ('Connection aborted.', timeout('The write operation timed out'))
[task 2022-01-20T22:15:12.427Z] INFO:upload-symbols:Retrying...
[task 2022-01-20T22:15:22.681Z] INFO:upload-symbols:Attempt 2 of 7...
[taskcluster:error] Task timeout after 900 seconds. Force killing container.
[taskcluster 2022-01-20 22:16:15.223Z] === Task Finished ===
[taskcluster 2022-01-20 22:16:15.224Z] Unsuccessful task run with exit code: -1 completed in 904.304 seconds

Comment 1

[Julien Cristau [:jcristau]]

Will, do you know if there was any issue with the symbols server yesterday?

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't see anything curious with Tecken at that time.

However, looking at that log, I think the task timeout isn't long enough and doesn't allow for more than one write timeout. There's a write timeout between 22:10 and 22:15 (5 minutes) and then it retries, but the retry is killed at 22:16 by the task timeout.

Comment 3

[Intermittent Failures Robot]

1 failures in 3444 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-01-17&endday=2022-01-23&tree=all

Comment 4

[Intermittent Failures Robot]

2 failures in 3445 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1
• windows2012-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-01-24&endday=2022-01-30&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 3341 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-03-21&endday=2022-03-27&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 3323 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-5-0-x86_64: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-03-28&endday=2022-04-03&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 2922 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-04-04&endday=2022-04-10&tree=all

Comment 8

[Intermittent Failures Robot]

4 failures in 2948 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• mozilla-beta: 1

Platform and build breakdown:

• osx-cross-devedition: 2
  • opt: 2
• windows2012-32-shippable: 1
  • opt: 1
• windows2012-aarch64-devedition: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-04-11&endday=2022-04-17&tree=all

Comment 9

[Intermittent Failures Robot]

1 failures in 3540 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows2012-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-05-02&endday=2022-05-08&tree=all

Comment 10

[Intermittent Failures Robot]

4 failures in 3317 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-esr91: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows2012-32: 1
  • debug: 1
• linux64-shippable: 1
  • opt: 1
• android-5-0-x86_64: 1
  • debug-isolated-process: 1
• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 11

[Intermittent Failures Robot]

4 failures in 2694 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 2

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1
• windows2012-64: 1
  • debug: 1
• osx-aarch64-devedition: 1
  • opt: 1
• osx-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 12

[Intermittent Failures Robot]

1 failures in 2624 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-05-23&endday=2022-05-29&tree=all

Comment 13

[Intermittent Failures Robot]

3 failures in 3336 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 1

Platform and build breakdown:

• windows2012-32: 1
  • debug: 1
• windows2012-32-devedition: 1
  • opt: 1
• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 3312 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• windows2012-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-06-27&endday=2022-07-03&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 3341 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• pine: 1

Platform and build breakdown:

• windows2012-64-pinebuild: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-07-11&endday=2022-07-17&tree=all

Comment 16

[Intermittent Failures Robot]

2 failures in 3497 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• linux64: 1
  • debug: 1
• windows2012-aarch64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-07-18&endday=2022-07-24&tree=all

Comment 17

[Intermittent Failures Robot]

3 failures in 3057 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2
• mozilla-esr102: 1

Platform and build breakdown:

• windows2012-64-shippable: 1
  • opt: 1
• linux64-devedition: 1
  • opt: 1
• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-08-01&endday=2022-08-07&tree=all

Comment 18

[Intermittent Failures Robot]

8 failures in 3339 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 2
• mozilla-beta: 2
• mozilla-central: 3
• try: 1

Platform and build breakdown:

• windows2012-32: 2
  • debug: 2
• android-5-0-x86-shippable: 1
  • opt: 1
• linux32: 1
  • debug: 1
• windows2012-aarch64: 2
  • debug: 2
• osx-cross: 1
  • debug: 1
• windows2012-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-08-08&endday=2022-08-14&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 3537 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• windows2012-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-08-29&endday=2022-09-04&tree=all

Comment 20

[Markus Stange [:mstange]]

The timeout is being bumped up in bug 1789414.

Comment 21

[Intermittent Failures Robot]

3 failures in 3616 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1
• mozilla-release: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows2012-aarch64: 1
  • debug: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• linux64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-09-05&endday=2022-09-11&tree=all

Comment 22

[Intermittent Failures Robot]

1 failures in 2909 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-09-19&endday=2022-09-25&tree=all

Comment 23

[Intermittent Failures Robot]

1 failures in 2793 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-10-17&endday=2022-10-23&tree=all

Comment 24

[Intermittent Failures Robot]

2 failures in 3743 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• symbols: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-10-24&endday=2022-10-30&tree=all

Comment 25

[Intermittent Failures Robot]

1 failures in 3938 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• symbols: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2022-11-28&endday=2022-12-04&tree=all

Comment 26

[Intermittent Failures Robot]

2 failures in 3606 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 2

Platform and build breakdown:

• windows2012-32: 1
  • debug: 1
• windows2012-32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2023-01-30&endday=2023-02-05&tree=all

Comment 27

[Intermittent Failures Robot]

1 failures in 3766 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• osx-cross: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2023-02-06&endday=2023-02-12&tree=all

Comment 28

[Intermittent Failures Robot]

1 failures in 3966 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 30

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=418587269&repo=mozilla-esr102

Comment 31

[Intermittent Failures Robot]

1 failures in 3885 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2023-06-05&endday=2023-06-11&tree=all

Comment 32

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 33

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=426033551&repo=comm-central

Comment 34

[Intermittent Failures Robot]

1 failures in 3076 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• windows2012-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 36

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=463792463&repo=comm-central

Comment 37

[Intermittent Failures Robot]

1 failures in 4116 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• osx-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 38

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 39

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=479483698&repo=comm-central

Comment 40

[Intermittent Failures Robot]

2 failures in 4456 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 2

Platform and build breakdown:

• osx-aarch64-shippable: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 41

[Intermittent Failures Robot]

2 failures in 4456 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 2

Platform and build breakdown:

• osx-aarch64-shippable: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures in 4227 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• osx-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1751303&startday=2024-11-11&endday=2024-11-17&tree=all

Comment 43

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.