Record mozharness step data in perfherder

NEW
Unassigned

Status

2 years ago
2 years ago

People

(Reporter: gps, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
I would like to see individual mozharness step data (mainly wall execution times) recorded in Perfherder. This will help us see where time is being spent in mozharness (which is now the bulk of most jobs/tasks in automation).

We currently record this data in influxdb. But that requires a login. Perfherder is in the open and will facilitate more/better data analysis.
You can pull it from ActiveData [1]:

>{
>	"select":[
>		"_id",
>		"etl.timestamp",
>		"build.branch",
>		"action.start_time",
>		"builder.step",
>		"harness.step",
>		"harness.duration"
>	],
>	"from":"jobs.action.timings",
>	"where":{"and":[
>		{"exists":"harness.step"},
>		{"gt":{"action.start_time":"{{today-month}}"}},
>		{"lt":{"action.start_time":"{{today}}"}}
>	]},
>	"limit":1000
>}

Also, there is a build times dashboard [2] that uses this (limited to builds).  It needs work, and could certainly be faster.   The detailed view [3] allows you to click on a data point and show the mozharness steps.  Maybe a selector, that only shows the particular step if interest, is a good feature?

[1] http://activedata.allizom.org/tools/query.html#query_id=O98V+shI
[2] http://people.mozilla.org/~klahnakoski/MoBuildbotTimings/Builds-Overview.html
[3] http://people.mozilla.org/~klahnakoski/MoBuildbotTimings/Builds-Detail.html#sampleMax=2016-05-06&sampleMin=2016-03-01&product=Firefox&type=PGO&platform=Windows64
(Reporter)

Comment 2

2 years ago
Oh, I didn't realize you were ingesting this already! How are you getting the data? Log parsing?
Flags: needinfo?(klahnakoski)
(Reporter)

Comment 4

2 years ago
According to wlach in #treeherder, there might be some value in redundantly adding the mozharness step data to perfherder. e.g. we can get alerts when times of certain steps changes significantly.

I guess the only question is whether we should add the data to perfherder via mozharness or whether ekyle's existing data loader could send the data to perfherder.

ekyle: do you have an opinion? (FWIW I don't think it's much effort to record the basic times in mozharness. I already had a patch in progress before you enlightened me about ActiveData.)
Flags: needinfo?(klahnakoski)
If you know what you want to measure, and there are few things to measure, then you may want to use PerfHerder, which will provide you with charts and level-shift detection.  ActiveData is better at a tracking a large number of measures, with many properties, that can be explored and decided at query time, rather than at submission time. 

I do not know if PerfHerder accepts submissions from remote clients, and I doubt it can handle the full job data volume; given the number of steps we are tracking in ActiveData now.
Flags: needinfo?(klahnakoski)
PerfHerder may attempt sending data requests to ActiveData directly, rather than its local database.  This would give it access to more data, without the work of scaling out its database.
(In reply to Kyle Lahnakoski [:ekyle] from comment #6)
> PerfHerder may attempt sending data requests to ActiveData directly, rather
> than its local database.  This would give it access to more data, without
> the work of scaling out its database.

Indeed, it might be interesting at some point to get ActiveData to ingest perfherder's database somehow for queries where SQL just isn't enough (I think this is the premise of Periscope: https://www.periscopedata.com/), but that's definitely beyond the scope of this bug. :)

Just adding a PERFHERDER_DATA line to your scripts is definitely the easiest path forward. I can provide advice on formatting that (but just searching for that string in mozilla-central should give you lots of inspiration).
(Reporter)

Updated

2 years ago
Blocks: 1271162

Comment 8

2 years ago
gps: are you planning on looking into this bug this quarter?
(Reporter)

Comment 9

2 years ago
(Please use needinfo)

No I'm not. I think wlach and ekyle have this under control?
Assignee: gps → nobody
Status: ASSIGNED → NEW
I'm not sure if we want to do this: it might be enough just to record the data in treeherder (bug 1258861). Let's leave this open for now just in case.
You need to log in before you can comment on or make changes to this bug.