1905006 - Perma [tier 2] linux1804-64-clang-trunk-qr opt mass failures that affect awsy and btime

Status: NEW

(Testing :: General, defect, P2)

Description

[Iulian Moraru]

This started happening all at once and only on linux1804-64-clang-trunk-qr opt.

• Push where this started
• Failure log for awsy jobs
• Failure log for btime jobs example 1
• Failure log for btime jobs example 2

Comment 1

[Iulian Moraru]

Hi Joel! Can you please take a look at this? Maybe you can figure out what is going on here.
Thank you!

Comment 2

[Intermittent Failures Robot]

243 failures in 654 pushes (0.372 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 243

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 243
  • opt: 243

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-06-26&endday=2024-06-26&tree=all

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

the push prior to the awsy failures seems to be where we turned on perf tests on clang builds. I don't know why we even run perf tests there as this isn't shippable, according to perfherder this has been running on m-c for >1 year, so why did we not have data for 20+ builds on m-c?

these tasks appear to be scheduled by a cron job:
Decision Task for cron job linux64-clang-trunk-perf

and even on previous commits this would run and generate target tasks, but those tasks wouldn't show up.

I am not aware of any other testing done on linux clang builds, so probably the builds are not functional in some way?

things to figure out:

1. why did these just start running (june 19th was the last run, then a break for a week?)
2. why do we run perf tests on here, but no unit tests?
3. can we add to autoland and bisect down?

I will start with the perf tooling team as we could understand the why and maybe be aware of more history of on/off tests or why they just started back up.

Comment 4

[Intermittent Failures Robot]

164 failures in 818 pushes (0.2 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 1
• mozilla-central: 163

Platform and build breakdown:

• windows11-64-qr: 1
  • opt: 1
• linux1804-64-clang-trunk-qr: 163
  • opt: 163

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-06-27&endday=2024-06-27&tree=all

Comment 5

[Intermittent Failures Robot]

99 failures in 710 pushes (0.139 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 99

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 99
  • opt: 99

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-06-28&endday=2024-06-28&tree=all

Comment 6

[Intermittent Failures Robot]

82 failures in 140 pushes (0.586 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 81
• autoland: 1

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 81
  • opt: 81
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-06-29&endday=2024-06-29&tree=all

Comment 7

[Intermittent Failures Robot]

588 failures in 4116 pushes (0.143 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 586
• autoland: 2

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 586
  • opt: 586
• windows11-64-qr: 1
  • opt: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 8

[Greg Mierzwinski [:sparky]]

:jmaher, I'm not sure why they broke like that, or why the scheduling is odd. We don't monitor these much on our side.

:andi, have these clang-trunk builds/tests been useful for anything in the time that they've been running (~3 years now)? I'm wondering if we could turn them off.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Greg Mierzwinski [:sparky] from comment #8)

:andi, have these clang-trunk builds/tests been useful for anything in the time that they've been running (~3 years now)? I'm wondering if we could turn them off.

Comment 10

[Mike Hommey [:glandium]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #3)

things to figure out:

I can answer these.

1. why did these just start running (june 19th was the last run, then a break for a week?)

Because the build they depend on, build-linux64-plain-clang-trunk/opt, didn't happen because of bug 1903956.

2. why do we run perf tests on here, but no unit tests?

Because the goal was to test the evolution of performance as the clang/LLVM trunk progresses.

3. can we add to autoland and bisect down?

Backfills on these jobs are meaningless because they always pull the latest clang/LLVM trunk. And here, the bustage comes from a change there, that hasn't been reverted yet. Essentially, this is bug 1906026.

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #9)

:andi, have these clang-trunk builds/tests been useful for anything in the time that they've been running (~3 years now)? I'm wondering if we could turn them off.

I'm not sure they have. I don't even know if perf sheriffs are looking at regressions coming from those jobs, so the question would more be for them. That being said, since bug 1790918 and bug 1791454, we have these perf jobs running on shippable builds (which would be more representative) on the toolchains project branch, so the jobs on central are kind of redundant, although it could be argued they have some level of usefulness (first and foremost, being more regular, except when their upstream tasks are busted).

Comment 11

[Intermittent Failures Robot]

68 failures in 807 pushes (0.084 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 68

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 68
  • opt: 68

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-07-02&endday=2024-07-02&tree=all

Comment 12

[Greg Mierzwinski [:sparky]]

(In reply to Mike Hommey [:glandium] from comment #10)

I'm not sure they have. I don't even know if perf sheriffs are looking at regressions coming from those jobs, so the question would more be for them. That being said, since bug 1790918 and bug 1791454, we have these perf jobs running on shippable builds (which would be more representative) on the toolchains project branch, so the jobs on central are kind of redundant, although it could be argued they have some level of usefulness (first and foremost, being more regular, except when their upstream tasks are busted).

These don't produce alerts that the sheriffs would monitor since they don't run on autoland (where our regression detection is running). Could we disable the tests in either toolchains or m-c (preferably m-c)? I would suggest we disable them in both branches unless there are plans to setup some sort of manual monitoring for them or if this already exists.

Comment 13

[Mike Hommey [:glandium]]

It's also running on beta, which is presumably tracked(?).

Comment 14

[Intermittent Failures Robot]

68 failures in 3199 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 68

Platform and build breakdown:

• linux1804-64-clang-trunk-qr: 68
  • opt: 68

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1905006&startday=2024-07-01&endday=2024-07-07&tree=all

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jmaher, could you have a look please?

For more information, please visit BugBot documentation.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

it seems like these are not monitored and are broken; we should disable these and if they are fixed and running reliable again and there is a desire to make these useful, they can be turned back on.

:sparky, are you up for making the patch to turn these off?

Comment 17

[Mike Hommey [:glandium]]

They're not broken anymore: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=trunk&revision=162448b8b7cdda02d8b081277b29e0b675567876

Comment 18

[Greg Mierzwinski [:sparky]]

Great that they're not broken anymore! I think there's still a question about if we should keep running them regardless of failures.

(In reply to Mike Hommey [:glandium] from comment #13)

It's also running on beta, which is presumably tracked(?).

I don't see them running on mozilla-beta (I checked July+some of June). If they run very infrequently then it will take a long time before any alerts get generated (we need 12 data points before/after a change to trigger an alert).

The load from these tests isn't very much atm with only m-c tests, but it still seems wasteful if we don't use the data for anything. :glandium, if you want to monitor the results from these tests could you setup a dashboard for yourself in redash and post a link to it here? Alternatively, we can disable these.

Comment 19

[Mike Hommey [:glandium]]

I guess disable them, but that will have the side effect of disabling the build-linux64-plain-clang-trunk/opt job too, which is useful.