Closed Bug 1358098 Opened 4 years ago Closed 4 years ago

Increase regression threshold for sccache alerts

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(firefox55 fixed)

RESOLVED FIXED
mozilla55
Tracking Status
firefox55 --- fixed

People

(Reporter: igoldan, Assigned: jmaher)

Details

(Keywords: perf)

Attachments

(1 file)

sccache alerts are triggered too often due to low reggression threshold.

This issue was raised in bug 1357718.

Use 5% as the new threshold for these build_metrics alerts.
:ted, would 5% make sense here?  Also any other build metrics that we should increase from 2% to something higher?
Flags: needinfo?(ted)
For requests_not_cacheable we should have the alert threshold be very high, like 100%. The metric being reported here is the number of compiler calls that sccache isn't able to cache. We run something like 3000 compiles on a normal build, and all of our builds have a small set of existing non-cacheable compiles, which is OK. The purpose of reporting this number is purely to make sure we don't accidentally start using some compiler options that make sccache unable to cache any of our compiles.

I don't know about other build metrics offhand. If you have specific examples we can certainly look at them!
Flags: needinfo?(ted)
100% might be odd, how about 50% for this metric
That's probably fine. Admittedly this metric is not very similar to the Talos metrics that Perfherder was designed around.
rwood, need your review for perfherder specifics.

ted, if you see anything odd in my changes, speak up, maybe cache_write_errors needs to be adjusted as well, etc.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8861096 - Flags: review?(ted)
Attachment #8861096 - Flags: review?(rwood)
Comment on attachment 8861096 [details] [diff] [review]
increase alert threshold

LGTM
Attachment #8861096 - Flags: review?(rwood) → review+
Comment on attachment 8861096 [details] [diff] [review]
increase alert threshold

Review of attachment 8861096 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/mozharness/mozilla/building/buildbase.py
@@ +1910,5 @@
>  
> +        yield {
> +            'name': 'sccache cache_write_errors',
> +            'value': stats['stats']['cache_write_errors'],
> +            'extraOptions': self.perfherder_resource_options(),

We could probably use a similar or even higher threshold here. This is almost always zero, but occasionally jumps up (presumably when we hit intermittent network issues connecting to S3):
https://mzl.la/2owrAdl

It's really only an issue if we start consistently getting a large number here.
Attachment #8861096 - Flags: review?(ted) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/23cfef4874a0
Increase regression threshold for sccache alerts. r=rwood,ted
https://hg.mozilla.org/mozilla-central/rev/23cfef4874a0
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.