Closed Bug 1436819 Opened 2 years ago Closed 3 months ago

generate alerts for performance tests that only run on mozilla-central

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: jmaher, Unassigned)

References

Details

we recently added motionmark to run on try/mozilla-central.  This is not on integration branches as we need to be careful about the CPU hours/push we run on each platform, specifically our hardware limited platforms.  There will be a few other benchmarks in this same scenario.

Right now we do not generate alerts for mozilla-central data because it is duplicate alerts from the integration branches.  I would like the ability to specify in the perfherder_data or some other config that we need to generate alerts on mozilla-central.

Once this is complete, Motionmark will be useful :)
Whiteboard: [PI:February]
branches in treeherder have a boolean attribute that indicates if we can have performance alerts:
"performance_alerts_enabled": true,

https://github.com/mozilla/treeherder/blob/a34546c2f267bc8aedb96b3c510a626c306dc8e5/treeherder/model/fixtures/repository.json#L27


this is not set for mozilla-central- the problem if we set it will be that all data there will generate alerts and the reason we do not alert on mozilla-central is that 99.9% of the alerts there are duplicate of something already found on autoland or mozilla-inbound.

We then use this field along with other metrics:
https://github.com/mozilla/treeherder/blob/ed1ec920516aea3e5e1f6df7946e514b3d08bf92/treeherder/etl/perf.py#L177

            if ((signature.should_alert or (signature.should_alert is None and
                                            suite.get('value') is None)) and
                datum_created and job.repository.performance_alerts_enabled):
                generate_alerts.apply_async(args=[signature.id],
routing_key='generate_perf_alerts')



so in order to enable alerting on mozilla-central for specific alerts we would need to add flags to the signature and adjust the logic in the above referenced perf.py.  if we add a new field, then we need to modify the database.  We can either do that with something like: 'alert_on_central' and make that false by default unless specified by data blobs.  Otherwise I see us doing logic where each harness would need to check the branch and set should_alert=false;

:wlach, could you offer any advice to solving this bug?
Flags: needinfo?(wlachance)
I think we accomplish what you need by changing the behavior so that we always alert if should_alert is explicitly set to True (you will also need to change this line: https://github.com/mozilla/treeherder/blob/a74003f41b38d8d5198db7dfab056fbe19133d75/treeherder/etl/perf.py#L117), and then making sure that property is set with motionmark on mozilla-central (and no where else). 

You may need to modify the gtest benchmarks slightly if you do this, it appears as if they depend on the current behaviour and  making this change will enable alerts on a lot of branches where we're not seeing them currently:

https://searchfox.org/mozilla-central/source/testing/gtest/mozilla/MozGTestBench.cpp#26

You probably want to change the above so you only *disable* alerts if the PERFHERDER_ALERTING_ENABLED environment resolves to false, but don't explicitly enable them otherwise.

Hope that wasn't too confusing.
Flags: needinfo?(wlachance)
Whiteboard: [PI:February] → [PI:March]
Priority: -- → P1
Blocks: 1445952
still high priority and want to look into this in April or no later than May
Whiteboard: [PI:March] → [PI:April]
Whiteboard: [PI:April] → [PI:May]
Whiteboard: [PI:May] → [PI:June]
Blocks: 1476568
No longer blocks: 1445952
circling back on this, we still need alerts for m-c tests (I am going to call these tier-2).  For the purposes of porting AWFY, we will have the jsshell-bench tests running per commit, so the browser tests will need the attention.

One extra motivation for this is when we run on specific hardware (reference win10 laptop) and having a small pool of 15 laptops will not allow us to run on inbound or autoland, just m-c and try.

I think we will pick this up in September.
Depends on: 1490447
Priority: P1 → P3
Whiteboard: [PI:June]

We don't currently have a need for this, and there will be an increase in the workload for perf sheriffs. A mozilla-central alert would need to be matched with any integration branch alerts, and any alerts that are only on mozilla-central will be time consuming to investigate. Let's revisit if a need arises.

Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.