1875115 - taskcluster issues, potentially cause by pulse issues, e.g. requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://taskcluster/queue/v1/task/G1TWngcxR5-elWhxEqP9pw

Status: NEW

(Taskcluster :: Operations and Service Requests, defect)

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yarik investigates an issue with high CPU load on both firefox-ci and the community instance as far as I understood, some pulse queues had an unacknowledged backlog but clearing them did not resolve it.

Tasks e.g. fail with https://treeherder.mozilla.org/logviewer?job_id=443648451&repo=autoland requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://taskcluster/queue/v1/task/G1TWngcxR5-elWhxEqP9pw, attempts to replay hg pushes or retrigger tasks fail with internal server error. There are failed tasks without obvious error (example) which might also be explained by this situation.

Trees are closed for this.

Comment 2

[[e-mail] (:bhearsum)]

The problem in this instance was that the Pernosco consumer stopped working, leading to an extremely large queue of unacked messages. To remedy this we removed permissions from this account and purged the queue until the Pernosco folks can fix the issue on their side.

https://bugzilla.mozilla.org/show_bug.cgi?id=1875132 has been filed about the general issue of unacked messages taking down Pulse.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Try has been reopened, autoland will follow once it has been verified setting a task as failed works as expected.

Comment 4

[Intermittent Failures Robot]

419 failures in 709 pushes (0.591 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-release: 21
• autoland: 306
• mozilla-central: 92

Platform and build breakdown:

• linux1804-32-shippable-qr: 1
  • opt: 1
• linux1804-64-shippable-qr: 14
  • opt: 14
• macosx1015-64-shippable-qr: 4
  • opt: 4
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows10-64-2009-shippable-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 34
  • opt: 34
• linux1804-64-qr: 142
  • debug: 96
  • opt: 46
• linux2204-64-wayland: 2
  • debug: 2
• android-em-7-0-x86_64-qr: 53
  • debug: 37
  • debug-isolated-process: 2
  • opt: 14
• windows11-64-2009-asan-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 88
  • opt: 88
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-qr: 25
  • debug: 24
  • opt: 1
• android-5-0-x86_64: 2
  • pgo: 2
• linux1804-64-ccov-qr: 18
  • opt: 18
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• linux64: 8
  • asan: 5
  • debug: 3
• windows2012-64: 2
  • asan: 2
• macosx1015-64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-lite-qr: 9
  • opt: 9
• linux64-aarch64: 1
  • opt: 1
• osx-cross-aarch64: 1
  • debug: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows2012-32: 1
  • debug: 1
• windows11-32-2009-qr: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875115&startday=2024-01-17&endday=2024-01-17&tree=all

Comment 5

[Intermittent Failures Robot]

419 failures in 3599 pushes (0.116 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 306
• mozilla-central: 92
• mozilla-release: 21

Platform and build breakdown:

• linux1804-64-qr: 142
  • opt: 46
  • debug: 96
• linux2204-64-wayland: 2
  • debug: 2
• linux64: 8
  • asan: 5
  • debug: 3
• windows2012-64: 2
  • asan: 2
• android-em-7-0-x86_64-qr: 53
  • debug: 37
  • debug-isolated-process: 2
  • opt: 14
• windows11-64-2009-asan-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 34
  • opt: 34
• linux1804-64-tsan-qr: 88
  • opt: 88
• windows11-32-2009-mingwclang-qr: 1
  • debug: 1
• windows11-64-2009-qr: 25
  • debug: 24
  • opt: 1
• macosx1015-64-qr: 1
  • debug: 1
• osx-cross-aarch64: 1
  • debug: 1
• windows2012-64-shippable: 1
  • opt: 1
• windows2012-32: 1
  • debug: 1
• android-5-0-x86_64: 2
  • pgo: 2
• android-em-7-0-x86_64-lite-qr: 9
  • opt: 9
• linux1804-32-shippable-qr: 1
  • opt: 1
• linux1804-64-ccov-qr: 18
  • opt: 18
• windows11-32-2009-qr: 7
  • debug: 7
• linux1804-64-shippable-qr: 14
  • opt: 14
• macosx1015-64-shippable-qr: 4
  • opt: 4
• linux64-aarch64: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1
• windows10-64-2009-shippable-qr: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875115&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 6

[[e-mail] (:bhearsum)]

This is long fixed.

Today we re-enabled the Pernosco account, and had the folks that run it subscribe to only the events they need. This doesn't fix any root causes (like their consumer breaking and not acking messages, or https://bugzilla.mozilla.org/show_bug.cgi?id=1875132), but it does limit the damage if this particular queue starts to build up messages again.

Comment 7

[Intermittent Failures Robot]

1 failures in 3805 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875115&startday=2024-04-01&endday=2024-04-07&tree=all

Comment 8

[:gerard-majax]

happening right now for me?

https://treeherder.mozilla.org/jobs?repo=try&revision=95a9ea78d65c056c804a7330e7ed5a7a1c6c23ba
https://treeherder.mozilla.org/jobs?repo=try&revision=162fdbc6e4e66b69e1fc152ac7c0dbccfe32223a

Comment 9

[[e-mail] (:bhearsum)]

(In reply to :gerard-majax from comment #8)

happening right now for me?

https://treeherder.mozilla.org/jobs?repo=try&revision=95a9ea78d65c056c804a7330e7ed5a7a1c6c23ba
https://treeherder.mozilla.org/jobs?repo=try&revision=162fdbc6e4e66b69e1fc152ac7c0dbccfe32223a

unlikely to be the same root cause. if it keeps up, please file something new or ping us in Matrix and we'll file it.

Comment 10

[Intermittent Failures Robot]

3 failures in 4625 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• gecko-decision: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1875115&startday=2024-06-10&endday=2024-06-16&tree=all