1842651 - Frequent Android xpcshell [taskcluster:error] Task timeout after 5400 seconds. Force killing container.

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=421965797&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Bal5dqlyT9iSwmWQebdkRw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-07-08T08:15:37.226Z] 08:15:37     INFO -  TEST-SKIP | netwerk/test/unit/test_http3_server.js | took 0ms
[task 2023-07-08T08:15:37.226Z] 08:15:37     INFO -  TEST-START | netwerk/test/unit/test_http3_dns_retry.js
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  TEST-SKIP | netwerk/test/unit/test_http3_dns_retry.js | took 0ms
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  INFO | Result summary:
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  INFO | Passed: 403
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  INFO | Failed: 0
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  INFO | Todo: 1
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  INFO | Retried: 32
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  SUITE-END | took 852s
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  Node moz-http2 server shutting down ...
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  Process stdout
[task 2023-07-08T08:15:37.227Z] 08:15:37     INFO -  forked process without handler sent: {"error":"","errorStack":""}
[task 2023-07-08T08:15:37.228Z] 08:15:37     INFO -  error: Command failed: /builds/worker/fetches/android-sdk-linux/platform-tools/adb reverse tcp:35914 tcp:35914
[task 2023-07-08T08:15:37.228Z] 08:15:37     INFO -  adb: error: cannot bind listener: Address already in use
[task 2023-07-08T08:15:37.228Z] 08:15:37     INFO -  error: Command failed: /builds/worker/fetches/android-sdk-linux/platform-tools/adb reverse tcp:46547 tcp:46547
[task 2023-07-08T08:15:37.228Z] 08:15:37     INFO -  adb: error: cannot bind listener: Address already in use
[taskcluster:error] Task timeout after 5400 seconds. Force killing container.
[taskcluster 2023-07-08 09:29:21.069Z] === Task Finished ===
[taskcluster 2023-07-08 09:29:21.070Z] Unsuccessful task run with exit code: -1 completed in 5400.941 seconds

Comment 1

[Cosmin Sabou [:CosminS]]

Coming from: https://bugzilla.mozilla.org/show_bug.cgi?id=1842167#c6

Comment 2

[Geoff Brown [:gbrown]]

This reminds me of bug 1824708, but I don't know if it is actually related.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1842167

:standard8, since you are the author of the regressor, bug 1842167, could you take a look?

For more information, please visit BugBot documentation.

Comment 4

[Mark Banner (:standard8)]

The interesting bits are these:

INFO -  error: Command failed: /builds/worker/fetches/android-sdk-linux/platform-tools/adb reverse tcp:35914 tcp:35914
adb: error: cannot bind listener: Address already in use

That seems to be new and shows up in the affected builds. I tried hunting down where the reverse happens, but came up with multiple different places.

The change from my bug is effectively upgrading from Node 12 to Node 16. So maybe there's some different characteristics for shutdown?

I think these are all triggered by the netwerk/test/unit/ tests, so maybe there's something in there that's causing it.

Kershaw, would you happen to have any ideas here?

Comment 5

[Intermittent Failures Robot]

81 failures in 647 pushes (0.125 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 77
• mozilla-central: 4

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 42
  • debug: 20
  • opt: 22
• android-em-7-0-x86_64-lite-qr: 36
  • opt: 36
• android-em-7-0-x86_64-shippable-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1842651&startday=2023-07-11&endday=2023-07-11&tree=all

Comment 6

[Kershaw Chang [:kershaw] PTO until Jan 2]

(In reply to Mark Banner (:standard8) from comment #4)

The interesting bits are these:

INFO -  error: Command failed: /builds/worker/fetches/android-sdk-linux/platform-tools/adb reverse tcp:35914 tcp:35914
adb: error: cannot bind listener: Address already in use

That seems to be new and shows up in the affected builds. I tried hunting down where the reverse happens, but came up with multiple different places.

The change from my bug is effectively upgrading from Node 12 to Node 16. So maybe there's some different characteristics for shutdown?

I think these are all triggered by the netwerk/test/unit/ tests, so maybe there's something in there that's causing it.

Kershaw, would you happen to have any ideas here?

I think it's likely that the error is from this line.
I'll try to debug this and see if I can find a solution for this.

Comment 7

[Intermittent Failures Robot]

47 failures in 655 pushes (0.072 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 43
• mozilla-central: 4

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 27
  • debug: 11
  • opt: 16
• android-em-7-0-x86_64-lite-qr: 17
  • opt: 17
• android-em-7-0-x86_64-shippable-qr: 2
  • opt: 2
• android-em-7-0-x86_64-shippable-lite-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1842651&startday=2023-07-12&endday=2023-07-12&tree=all

Comment 8

[Kershaw Chang [:kershaw] PTO until Jan 2]

Attachment: Bug 1842651 - Retry adb reverse if the port can not be used, r=Standard8

Comment 9

[Kershaw Chang [:kershaw] PTO until Jan 2]

https://treeherder.mozilla.org/jobs?repo=try&revision=48a7c63578252dd8e645a1c562c2c58c5f4178b8&selectedTaskRun=ZF3lXmqOTQC2taDeTqzb0g.0

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/716992a687c0
Retry adb reverse if the port can not be used, r=Standard8

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/716992a687c0

Comment 12

[Intermittent Failures Robot]

47 failures in 706 pushes (0.067 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 47

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 29
  • debug: 12
  • opt: 17
• android-em-7-0-x86_64-lite-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1842651&startday=2023-07-13&endday=2023-07-13&tree=all

Comment 13

[Intermittent Failures Robot]

189 failures in 3309 pushes (0.057 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 181
• mozilla-central: 8

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 77
  • opt: 77
• android-em-7-0-x86_64-qr: 106
  • debug: 47
  • opt: 59
• android-em-7-0-x86_64-shippable-qr: 4
  • opt: 4
• android-em-7-0-x86_64-shippable-lite-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1842651&startday=2023-07-10&endday=2023-07-16&tree=all