1821513 - [meta] High frequency [taskcluster:error] Task timeout after 2700 seconds / 1800 seconds / 3600 seconds / 7200 seconds. Force killing container. -> Hang during various stages of the build

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=408420613&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SxIBvUG0T0aoVj9ZBkrtwg/runs/0/artifacts/public/logs/live_backing.log

---

[taskcluster 2023-03-09 17:21:13.355Z] === Task Starting ===
 * Starting system message bus dbus       
[setup 2023-03-09T17:21:16.027Z] run-task started in /builds/worker
[cache 2023-03-09T17:21:16.030Z] cache /builds/worker/checkouts is empty; writing requirements: gid=1000 uid=1000 version=1
[cache 2023-03-09T17:21:16.030Z] cache /builds/worker/tooltool-cache is empty; writing requirements: gid=1000 uid=1000 version=1
[volume 2023-03-09T17:21:16.030Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2023-03-09T17:21:16.030Z] volume /builds/worker/checkouts is a cache
[volume 2023-03-09T17:21:16.030Z] volume /builds/worker/tooltool-cache is a cache
[volume 2023-03-09T17:21:16.030Z] changing ownership of volume /builds/worker/workspace to 1000:1000
[setup 2023-03-09T17:21:16.030Z] running as worker:worker
[vcs 2023-03-09T17:21:16.030Z] fetching hgmointernal config from http://taskcluster/secrets/v1/secret/project/taskcluster/gecko/hgmointernal
[vcs 2023-03-09T17:21:16.127Z] region google/us-central1 not yet supported; using public hg.mozilla.org service
[vcs 2023-03-09T17:21:16.127Z] executing ['hg', 'robustcheckout', '--sharebase', '/builds/worker/checkouts/hg-store', '--purge', '--upstream', 'https://hg.mozilla.org/mozilla-unified', '--revision', '8aea0e78341420231309f814d26a9819e600c94f', 'https://hg.mozilla.org/integration/autoland', '/builds/worker/checkouts/gecko']
[vcs 2023-03-09T17:21:16.479Z] (using Mercurial 5.8.1)
[vcs 2023-03-09T17:21:16.480Z] ensuring https://hg.mozilla.org/integration/autoland@8aea0e78341420231309f814d26a9819e600c94f is available at /builds/worker/checkouts/gecko
[vcs 2023-03-09T17:21:17.007Z] (cloning from upstream repo https://hg.mozilla.org/mozilla-unified)
[vcs 2023-03-09T17:21:17.724Z] (sharing from new pooled repository 8ba995b74e18334ab3707f27e9eb8f4e37ba3d29)
[vcs 2023-03-09T17:21:18.377Z] applying clone bundle from https://storage.googleapis.com/moz-hg-bundles-gcp-us-central1/mozilla-unified/4d5e640759776d678d17822c70cece3e395e72c1.stream-v2.hg
[vcs 2023-03-09T17:21:18.460Z] 703336 files to transfer, 4.39 GB of data
[vcs 2023-03-09T17:21:20.466Z] 
[vcs 2023-03-09T17:21:21.463Z] clone [=>                                           ]  269189773/4713057607 34s
[vcs 2023-03-09T17:21:22.463Z] clone [==>                                          ]  359632057/4713057607 37s
<...>
[vcs 2023-03-09T17:22:25.470Z] clone [==================================>          ] 3673829845/4713057607 19s
[vcs 2023-03-09T17:22:26.470Z] clone [==================================>          ] 3762565975/4713057607 17s
[vcs 2023-03-09T17:22:27.502Z] clone [===================================>         ] 3823083326/4713057607 16s
[taskcluster:error] Task timeout after 2700 seconds. Force killing container.
[taskcluster 2023-03-09 18:06:13.764Z] === Task Finished ===
[taskcluster 2023-03-09 18:06:13.765Z] Unsuccessful task run with exit code: -1 completed in 2779.496 seconds

Comment 1

[Iulian Moraru]

Hi Andrew! Can you please take a look at this? In the last few days we saw an increase in frequency in "run" build bustages.
They fail like this or with failure lines that would go to Bug 1705852.
Thank you!

Comment 2

[Iulian Moraru]

From what I can see, so far, the ones that fail because of the hang during cloning are all on Android 5.0 x86 pgo.
And the ones that fail with

[task 2023-03-09T13:43:26.860Z] adb Using adb 1.0.41
[task 2023-03-09T13:48:26.863Z] 13:48:26     INFO - Sleeping 10 seconds
[task 2023-03-09T13:48:36.867Z] 13:48:36     INFO - >> Verify Android boot completed: Attempt #2 of 30
[task 2023-03-09T13:48:36.872Z] adb Using adb 1.0.41
[taskcluster:error] Task timeout after 2700 seconds. Force killing container.
[taskcluster 2023-03-09 13:53:02.734Z] === Task Finished ===
[taskcluster 2023-03-09 13:53:02.735Z] Unsuccessful task run with exit code: -1 completed in 2780.163 seconds

are on Android 5.0 x86-64 pgo.

Comment 3

[Iulian Moraru]

I guess this is not just Android related, found the same hang on OSX and Windows.

Comment 4

[Intermittent Failures Robot]

25 failures in 748 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 12

Platform and build breakdown:

• android-5-0-x86: 12
  • pgo: 12
• android-5-0-x86_64: 7
  • pgo: 7
• gecko-decision: 3
  • opt: 3
• linux64-shippable: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-09&endday=2023-03-09&tree=all

Comment 5

[Intermittent Failures Robot]

28 failures in 720 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 7

Platform and build breakdown:

• android-5-0-x86: 6
  • pgo: 5
  • opt: 1
• android-5-0-x86_64: 9
  • debug-isolated-process: 2
  • opt: 2
  • debug: 4
  • asan: 1
• linux64: 1
  • tsan: 1
• windows2012-32: 2
  • opt: 2
• windows-mingw32: 1
  • all: 1
• gecko-decision: 4
  • opt: 4
• windows2012-64: 2
  • debug: 2
• linux64-shippable: 1
  • opt: 1
• windows2012-aarch64: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-10&endday=2023-03-10&tree=all

Comment 6

[Intermittent Failures Robot]

73 failures in 3966 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 40
• mozilla-central: 33

Platform and build breakdown:

• android-5-0-x86: 18
  • pgo: 17
  • opt: 1
• android-5-0-x86_64: 17
  • pgo: 7
  • debug-isolated-process: 2
  • opt: 3
  • debug: 4
  • asan: 1
• gecko-decision: 7
  • opt: 7
• linux64-shippable: 10
  • opt: 10
• linux64: 3
  • tsan: 1
  • asan: 2
• windows2012-32: 2
  • opt: 2
• windows-mingw32: 1
  • all: 1
• windows2012-64: 3
  • debug: 3
• windows2012-aarch64: 2
  • debug: 2
• osx-shippable: 2
  • opt: 2
• android-5-0-aarch64: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• windows2012-aarch64-shippable: 2
  • opt: 2
• updatebot: 1
  • all: 1
• windows2012-32-shippable: 1
  • opt: 1
• symbols: 1
  • opt: 1
• android-4-1-geckoview-fat-aar-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-06&endday=2023-03-12&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Could this hangs be explained by the upgrade of Python to version 3.7 in bug 1734402? The failures got initially classified against other, existing bugs for intermittent task timeouts.

Comment 8

[Intermittent Failures Robot]

50 failures in 735 pushes (0.068 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 42
• mozilla-esr102: 1
• mozilla-beta: 7

Platform and build breakdown:

• windows2012-32: 4
  • debug: 3
  • opt: 1
• android-5-0-x86: 3
  • pgo: 2
  • opt: 1
• windows2012-64: 3
  • opt: 1
  • debug: 1
  • asan: 1
• osx-cross: 6
  • opt: 3
  • debug: 2
  • plain: 1
• android-5-0-x86_64: 9
  • debug: 2
  • asan: 5
  • debug-isolated-process: 1
  • opt: 1
• android-5-0-aarch64: 2
  • opt: 2
• android-5-0-armv7: 1
  • opt: 1
• linux64: 4
  • debug: 1
  • opt: 2
  • asan: 1
• gecko-decision: 6
  • opt: 6
• android-5-0-aarch64-shippable: 1
  • opt: 1
• android-5-0-aarch64-shippable-lite: 1
  • opt: 1
• windows2012-32-devedition: 1
  • opt: 1
• toolchains: 3
  • opt: 3
• linux64-shippable: 1
  • opt: 1
• windows2012-64-devedition: 2
  • opt: 2
• linux64-devedition: 1
  • opt: 1
• linux32-devedition: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-13&endday=2023-03-13&tree=all

Comment 9

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

It's possible. The build script for mercurial 5.8.1 was slightly altered to work with python 3.7 for ubuntu1804. There could just be an unknown incompatibility there causing the hangs. I will investigate. I'll start by trying to build different (newer) versions of mercurial to see if that changes the behavior here.

Comment 13

[Intermittent Failures Robot]

80 failures in 800 pushes (0.1 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 32
• mozilla-central: 21
• try: 1
• mozilla-beta: 26

Platform and build breakdown:

• android-5-0-x86_64: 16
  • opt: 4
  • asan: 3
  • debug: 3
  • debug-isolated-process: 3
  • pgo: 3
• android-5-0-aarch64: 3
  • opt: 3
• linux64: 3
  • tsan: 1
  • asan: 1
  • opt: 1
• diff: 1
  • opt: 1
• gecko-decision: 16
  • opt: 16
• linux32: 1
  • asan: 1
• osx-shippable: 3
  • opt: 3
• windows2012-64-shippable: 3
  • opt: 3
• win64-aarch64-shippable: 1
  • opt: 1
• windows2012-64: 6
  • asan: 3
  • debug: 3
• osx-aarch64-shippable: 1
  • opt: 1
• windows2012-32: 1
  • opt: 1
• android-5-0-x86: 3
  • pgo: 3
• android-4-1-armv7: 1
  • debug: 1
• macosx1015-64-shippable: 2
  • opt: 2
• linux32-shippable: 2
  • opt: 2
• windows2012-aarch64-shippable: 1
  • opt: 1
• linux64-shippable: 2
  • opt: 2
• windows2012-32-shippable: 5
  • opt: 5
• windows2012-64-devedition: 3
  • opt: 3
• windows2012-aarch64-devedition: 3
  • opt: 3
• windows2012-32-devedition: 1
  • opt: 1
• symbols: 1
  • opt: 1
• toolchains: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-14&endday=2023-03-14&tree=all

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

• Bug 1821537: S2

:ahal, could you consider increasing the severity of this bug to S2?

For more information, please visit auto_nag documentation.

Comment 15

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - add worker image/pool for testing. r?masterwayz

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/1e48ac78e98c
add worker image/pool for testing. r=MasterWayZ

Comment 17

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - bump disk size for test pool. r?masterwayz,#releng

For some reason b-linux-gcp works fine with a 20gb disk but we were getting errors:
Invalid value for field 'resource.disks[0].initializeParams.diskSizeGb': '20'. Requested disk size cannot be smaller than the image size (75 GB)

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/da908765ecd3
bump disk size for test pool.

Comment 19

[Intermittent Failures Robot]

116 failures in 865 pushes (0.134 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 77
• firefox-android: 11
• mozilla-central: 26
• mozilla-beta: 2

Platform and build breakdown:

• gecko-decision: 7
  • opt: 7
• osx-cross: 2
  • asan: 1
  • opt: 1
• windows2012-32-shippable: 3
  • opt: 3
• android-5-0-aarch64: 1
  • opt: 1
• android-5-0-armv7: 2
  • opt: 1
  • debug: 1
• android-4-1-armv7: 2
  • opt: 1
  • debug: 1
• AC-android-all: 11
  • opt: 11
• windows2012-64: 13
  • debug: 5
  • opt: 4
  • asan: 4
• android-5-0-x86_64: 19
  • opt: 7
  • debug-isolated-process: 2
  • asan: 3
  • debug: 5
  • pgo: 2
• windows2012-32: 7
  • opt: 2
  • debug: 5
• linux64: 24
  • tsan: 5
  • debug: 9
  • opt: 8
  • asan: 2
• android-5-0-x86: 7
  • pgo: 7
• diff: 1
  • opt: 1
• windows2012-aarch64: 2
  • opt: 2
• android-5-0-aarch64-shippable-lite: 1
  • opt: 1
• linux64-shippable: 4
  • opt: 4
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1
• win32-shippable: 1
  • opt: 1
• windows10-64: 1
  • opt: 1
• linux32: 1
  • debug: 1
• win64-aarch64-shippable: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• linux64-asan-reporter: 1
  • opt: 1
• android-5-0-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-15&endday=2023-03-15&tree=all

Comment 20

[Michelle Goossens [:masterwayz]]

Attachment: Bug 1821513 - Revert disk size change and use new image r=jcristau

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/d444721848b0
Revert disk size change and use new image r=jcristau

Comment 22

[Intermittent Failures Robot]

126 failures in 726 pushes (0.174 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 91
• firefox-android: 17
• mozilla-central: 10
• try: 1
• mozilla-esr102: 2
• mozilla-release: 1
• mozilla-beta: 4

Platform and build breakdown:

• osx-cross: 7
  • debug: 3
  • opt: 3
  • asan: 1
• linux64: 27
  • opt: 15
  • asan: 7
  • debug: 5
• windows2012-64: 7
  • debug: 5
  • asan: 1
  • opt: 1
• android-5-0-x86_64: 22
  • debug: 5
  • opt: 10
  • pgo: 2
  • debug-isolated-process: 2
  • asan: 3
• lint: 1
  • opt: 1
• android-5-0-armv7: 5
  • debug: 2
  • opt: 3
• windows2012-32: 6
  • debug: 5
  • opt: 1
• windows2012-32-shippable: 3
  • opt: 3
• gecko-decision: 8
  • opt: 8
• android-5-0-aarch64: 6
  • debug: 2
  • opt: 4
• windows-mingw32: 3
  • all: 3
• android-5-0-x86: 4
  • pgo: 3
  • opt: 1
• linux32-shippable: 2
  • opt: 2
• windows2012-64-shippable: 3
  • opt: 3
• linux64-shippable: 3
  • opt: 3
• windows2012-aarch64-shippable: 2
  • opt: 2
• windows2012-aarch64: 1
  • debug: 1
• fenix-android-all: 2
  • opt: 2
• AC-android-all: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-16&endday=2023-03-16&tree=all

Comment 23

[Intermittent Failures Robot]

79 failures in 321 pushes (0.246 failures/push) were associated with this bug yesterday.

Repository breakdown:

• firefox-android: 18
• autoland: 24
• mozilla-central: 29
• try: 1
• mozilla-beta: 7

Platform and build breakdown:

• AC-android-all: 13
  • opt: 13
• focus-android-all: 3
  • opt: 3
• android-5-0-x86_64: 5
  • opt: 3
  • asan: 1
  • debug: 1
• android-4-1-armv7: 2
  • debug: 2
• android-em-7-0-x86: 1
  • opt: 1
• linux32-shippable: 5
  • opt: 5
• android-5-0-aarch64: 3
  • opt: 2
  • debug: 1
• android-5-0-x86-shippable: 2
  • opt: 2
• windows2012-64: 4
  • opt: 2
  • debug: 1
  • asan: 1
• gecko-decision: 2
  • opt: 2
• linux64: 9
  • opt: 5
  • debug: 2
  • asan: 2
• linux32: 2
  • debug: 2
• android-5-0-armv7: 1
  • debug: 1
• windows2012-64-shippable: 4
  • opt: 4
• osx-cross: 5
  • debug: 1
  • opt: 2
  • plain: 2
• linux64-shippable: 2
  • opt: 2
• android-5-0-geckoview-fat-aar-shippable: 1
  • opt: 1
• osx-shippable: 2
  • opt: 2
• lint: 2
  • opt: 2
• windows2012-32-shippable: 1
  • opt: 1
• linux64-devedition: 1
  • opt: 1
• android-5-0-x86: 1
  • pgo: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• android-5-0-armv7-shippable: 2
  • opt: 2
• osx-cross-devedition: 1
  • opt: 1
• osx-aarch64-devedition: 1
  • opt: 1
• windows2012-32: 1
  • opt: 1
• windows2012-aarch64: 1
  • debug: 1
• win64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-17&endday=2023-03-17&tree=all

Comment 24

[Intermittent Failures Robot]

16 failures in 143 pushes (0.112 failures/push) were associated with this bug yesterday.

Repository breakdown:

• firefox-android: 9
• autoland: 2
• try: 1
• mozilla-central: 4

Platform and build breakdown:

• AC-android-all: 9
  • opt: 9
• android-5-0-x86_64: 1
  • opt: 1
• android-5-0-aarch64: 1
  • opt: 1
• windows2012-aarch64-devedition: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1
• windows2012-64-shippable: 2
  • opt: 2
• toolchains: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-18&endday=2023-03-18&tree=all

Comment 25

[Intermittent Failures Robot]

560 failures in 3761 pushes (0.149 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• firefox-android: 85
• autoland: 305
• mozilla-central: 96
• mozilla-beta: 65
• mozilla-esr102: 3
• try: 4
• mozilla-release: 2

Platform and build breakdown:

• AC-android-all: 75
  • opt: 75
• linux64: 69
  • debug: 18
  • opt: 32
  • asan: 13
  • tsan: 6
• windows2012-32: 21
  • debug: 13
  • opt: 8
• android-5-0-x86: 19
  • pgo: 17
  • opt: 2
• windows2012-64: 38
  • opt: 8
  • debug: 19
  • asan: 11
• osx-cross: 24
  • opt: 10
  • debug: 9
  • plain: 3
  • asan: 2
• android-5-0-x86_64: 82
  • debug: 18
  • asan: 15
  • pgo: 9
  • debug-isolated-process: 8
  • opt: 32
• android-5-0-aarch64: 16
  • opt: 13
  • debug: 3
• gecko-decision: 46
  • opt: 46
• android-5-0-armv7: 9
  • opt: 5
  • debug: 4
• windows2012-aarch64: 7
  • opt: 5
  • debug: 2
• linux64-aarch64: 1
  • opt: 1
• linux64-shippable: 18
  • opt: 18
• linux32: 5
  • debug: 4
  • asan: 1
• android-5-0-aarch64-shippable: 2
  • opt: 2
• android-5-0-aarch64-shippable-lite: 2
  • opt: 2
• windows2012-32-devedition: 3
  • opt: 3
• toolchains: 9
  • opt: 9
• windows2012-64-devedition: 6
  • opt: 6
• linux32-devedition: 4
  • opt: 4
• windows2012-aarch64-devedition: 6
  • opt: 6
• linux64-devedition: 3
  • opt: 3
• osx-cross-devedition: 2
  • opt: 2
• osx-shippable: 11
  • opt: 11
• diff: 2
  • opt: 2
• windows2012-64-shippable: 14
  • opt: 14
• win64-aarch64-shippable: 2
  • opt: 2
• fenix-android-all: 4
  • opt: 4
• osx-aarch64-shippable: 2
  • opt: 2
• android-4-1-armv7: 5
  • debug: 4
  • opt: 1
• macosx1015-64-shippable: 2
  • opt: 2
• linux32-shippable: 11
  • opt: 11
• windows2012-aarch64-shippable: 4
  • opt: 4
• windows2012-32-shippable: 13
  • opt: 13
• symbols: 1
  • opt: 1
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• win32-shippable: 1
  • opt: 1
• windows10-64: 1
  • opt: 1
• linux64-asan-reporter: 1
  • opt: 1
• lint: 3
  • opt: 3
• windows-mingw32: 3
  • all: 3
• focus-android-all: 3
  • opt: 3
• android-em-7-0-x86: 1
  • opt: 1
• android-5-0-x86-shippable: 2
  • opt: 2
• android-5-0-geckoview-fat-aar-shippable: 1
  • opt: 1
• android-4-1-x86-shippable-lite: 1
  • opt: 1
• android-5-0-armv7-shippable: 2
  • opt: 2
• osx-aarch64-devedition: 1
  • opt: 1
• win64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-13&endday=2023-03-19&tree=all

Comment 26

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - update l1 docker-worker image. r?masterwayz

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/90efbd04c604
update l1 docker-worker image. r=MasterWayZ

Comment 28

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - update robustcheckout hg extension. r?#releng

Imported from version-control-tools revision 0698b5050920.

Comment 29

[Intermittent Failures Robot]

25 failures in 780 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• firefox-android: 14
• autoland: 6
• mozilla-central: 1
• mozilla-beta: 2
• mozilla-release: 1
• toolchains: 1

Platform and build breakdown:

• focus-android-all: 1
  • opt: 1
• AC-android-all: 12
  • opt: 12
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• android-5-0-x86_64: 1
  • pgo: 1
• windows2012-32: 3
  • debug: 1
  • opt: 2
• android-5-0-x86: 1
  • pgo: 1
• windows2012-aarch64: 1
  • debug: 1
• fenix-android-all: 1
  • opt: 1
• android-5-0-aarch64: 1
  • debug: 1
• windows2012-64-devedition: 1
  • opt: 1
• gecko-decision: 1
  • opt: 1
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-20&endday=2023-03-20&tree=all

Comment 30

[Julien Cristau [:jcristau]]

So far I've been able to reproduce hangs during robustcheckout; a combination of updating robustcheckout to include the change from bug 1822044 and updating mercurial per bug 1822044 comment 3 will hopefully resolve that.
I haven't yet been able to root-cause these connections hanging, or reproduce other causes of hangs, though.

Comment 31

[Intermittent Failures Robot]

51 failures in 743 pushes (0.069 failures/push) were associated with this bug yesterday.

Repository breakdown:

• firefox-android: 6
• mozilla-central: 2
• autoland: 35
• mozilla-release: 2
• mozilla-beta: 6

Platform and build breakdown:

• lint: 1
  • opt: 1
• android-5-0-x86_64: 12
  • debug: 5
  • asan: 1
  • opt: 5
  • debug-isolated-process: 1
• AC-android-all: 3
  • opt: 3
• osx-cross: 5
  • debug: 1
  • opt: 3
  • plain: 1
• linux64: 4
  • opt: 2
  • debug: 2
• windows2012-64: 2
  • opt: 1
  • debug: 1
• windows-mingw32: 1
  • all: 1
• windows2012-aarch64-shippable: 2
  • opt: 2
• windows2012-32-shippable: 2
  • opt: 2
• linux64-shippable: 3
  • opt: 3
• osx-aarch64-shippable: 1
  • opt: 1
• windows2012-aarch64: 1
  • opt: 1
• linux1804-64-qr: 3
  • debug: 3
• osx-shippable: 2
  • opt: 2
• linux32-shippable: 1
  • opt: 1
• windows2012-64-shippable: 1
  • opt: 1
• android-5-0-aarch64: 2
  • opt: 1
  • debug: 1
• android-4-1-armv7: 1
  • opt: 1
• android-5-0-armv7: 1
  • opt: 1
• focus-android-all: 2
  • opt: 2
• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-21&endday=2023-03-21&tree=all

Comment 32

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - update l3 docker-worker image. r?MasterWayZ

Comment 33

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - patch mercurial debian package to make http.timeout option work

Cherry-pick from https://www.mercurial-scm.org/repo/hg/rev/3bb7c56e8fe6

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/5e146d52ea54
update l3 docker-worker image. r=MasterWayZ

Comment 35

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - set a timeout for fetch-content downloads. r?#releng

Setting a timeout on the download means we can retry if the connection
hangs, instead of sitting around idle until the task itself hits its
maxRunTime.

Comment 36

[Intermittent Failures Robot]

41 failures in 821 pushes (0.05 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 29
• firefox-android: 6
• mozilla-central: 5
• comm-central: 1

Platform and build breakdown:

• gecko-decision: 14
  • opt: 14
• AC-android-all: 5
  • opt: 5
• android-5-0-x86_64: 5
  • opt: 2
  • debug: 1
  • asan: 1
  • pgo: 1
• linux64: 2
  • asan: 1
  • opt: 1
• windows2012-64: 5
  • debug: 3
  • asan: 2
• osx-shippable: 1
  • opt: 1
• windows2012-32: 2
  • opt: 1
  • debug: 1
• osx-cross: 1
  • debug: 1
• linux64-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1
• thunderbird-release: 1
  • opt: 1
• windows2012-32-shippable: 1
  • opt: 1
• linux32: 1
  • debug: 1
• fenix-android-all: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-22&endday=2023-03-22&tree=all

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b3d278d4a6d8
update robustcheckout hg extension. r=releng-reviewers,bhearsum

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/528f46a1abf8
patch mercurial debian package to make http.timeout option work r=releng-reviewers,bhearsum

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c7a5550cc22f
set a timeout for fetch-content downloads. r=releng-reviewers,gbrown

Comment 40

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - bump decision docker image version. r?#releng

Comment 41

[Julien Cristau [:jcristau]]

The patches from comments 37-39 should help with the most common issues we've seen here. If that's confirmed I should be able to move on to remaining places we see timeouts next week.

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebb447880cf2
bump decision docker image version. r=releng-reviewers,gbrown DONTBUILD

Comment 43

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/b3d278d4a6d8
https://hg.mozilla.org/mozilla-central/rev/528f46a1abf8
https://hg.mozilla.org/mozilla-central/rev/c7a5550cc22f

Comment 44

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/ebb447880cf2

Comment 45

[Intermittent Failures Robot]

127 failures in 4161 pushes (0.031 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• firefox-android: 26
• autoland: 79
• mozilla-central: 8
• mozilla-beta: 8
• mozilla-release: 3
• toolchains: 1
• comm-central: 1
• mozilla-esr102: 1

Platform and build breakdown:

• focus-android-all: 3
  • opt: 3
• AC-android-all: 20
  • opt: 20
• android-5-0-x86_64-shippable-lite: 1
  • opt: 1
• android-5-0-x86_64: 18
  • pgo: 2
  • debug: 6
  • asan: 2
  • opt: 7
  • debug-isolated-process: 1
• windows2012-32: 5
  • debug: 2
  • opt: 3
• android-5-0-x86: 1
  • pgo: 1
• windows2012-aarch64: 2
  • debug: 1
  • opt: 1
• fenix-android-all: 2
  • opt: 2
• android-5-0-aarch64: 3
  • debug: 2
  • opt: 1
• windows2012-64-devedition: 1
  • opt: 1
• gecko-decision: 19
  • opt: 19
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1
• lint: 1
  • opt: 1
• osx-cross: 6
  • debug: 2
  • opt: 3
  • plain: 1
• linux64: 6
  • opt: 3
  • debug: 2
  • asan: 1
• windows2012-64: 7
  • opt: 1
  • debug: 4
  • asan: 2
• windows-mingw32: 1
  • all: 1
• windows2012-aarch64-shippable: 2
  • opt: 2
• windows2012-32-shippable: 3
  • opt: 3
• linux64-shippable: 4
  • opt: 4
• linux1804-64-qr: 5
  • debug: 5
• osx-aarch64-shippable: 1
  • opt: 1
• android-4-1-armv7: 1
  • opt: 1
• android-5-0-armv7: 1
  • opt: 1
• osx-shippable: 3
  • opt: 3
• linux32-shippable: 2
  • opt: 2
• windows2012-64-shippable: 1
  • opt: 1
• thunderbird-release: 1
  • opt: 1
• linux32: 1
  • debug: 1
• linux1804-64-asan-qr: 1
  • opt: 1
• toolchains: 1
  • opt: 1
• linux1804-64-tsan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-20&endday=2023-03-26&tree=all

Comment 46

[Julien Cristau [:jcristau]]

It looks like this is pretty much gone now. I see 2 tasks starred with this bug so far this week, both of which don't actually show a hang, just tests that legitimately did run longer than their maxRunTime.

Comment 47

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/13caf33f49e6
https://hg.mozilla.org/releases/mozilla-esr102/rev/f814b6f367ea
https://hg.mozilla.org/releases/mozilla-esr102/rev/e402ffc75cb5

Comment 48

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-beta/rev/af658fd3f204
https://hg.mozilla.org/releases/mozilla-beta/rev/3bfad2337556
https://hg.mozilla.org/releases/mozilla-beta/rev/3a45228e40b6

Comment 49

[Intermittent Failures Robot]

8 failures in 3863 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3
• mozilla-esr102: 1
• firefox-android: 1
• try: 1
• autoland: 2

Platform and build breakdown:

• osx-cross: 1
  • asan: 1
• linux32: 1
  • debug: 1
• windows2012-aarch64: 1
  • debug: 1
• linux1804-64-tsan-qr: 2
  • opt: 2
• fenix-android-all: 1
  • opt: 1
• linux1804-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-03-27&endday=2023-04-02&tree=all

Comment 50

[Julien Cristau [:jcristau]]

Attachment: Bug 1821513 - remove test pool, no longer necessary. r?masterwayz

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/981118187641
remove test pool, no longer necessary. r=MasterWayZ

Comment 52

[Julien Cristau [:jcristau]]

Calling this done.

Comment 53

[Intermittent Failures Robot]

1 failures in 3386 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• firefox-android: 1

Platform and build breakdown:

• AC-ui-test: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-04-03&endday=2023-04-09&tree=all

Comment 54

[Intermittent Failures Robot]

1 failures in 3206 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-04-10&endday=2023-04-16&tree=all

Comment 55

[Intermittent Failures Robot]

1 failures in 4004 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-04-17&endday=2023-04-23&tree=all

Comment 56

[Intermittent Failures Robot]

4 failures in 3671 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4

Platform and build breakdown:

• linux64-devedition: 1
  • opt: 1
• linux64-shippable: 1
  • opt: 1
• osx-shippable: 1
  • opt: 1
• linux32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1821513&startday=2023-12-04&endday=2023-12-10&tree=all